Anish Mahto created SPARK-60252:
-----------------------------------
Summary: SCD1 Ignore-null support; Field Reconciliation Data
Classes
Key: SPARK-60252
URL: https://issues.apache.org/jira/browse/SPARK-60252
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
Introduce case classes and helpers that represent user-data fields during SCD1
ignore-null reconciliation.
SCD1 reconciliation combines a key's CDC events into one row holding the key's
latest state. It first collapses the incoming microbatch to at most one row per
key, then reconciles that row against the key's persisted tombstone and, on
merge, against its target row. Without ignore-null, reconciliation is row-wide:
one row wins every field. With ignore-null, an upsert authors a selected leaf
only if it provides a non-null value, so different rows can win different
fields.
`Scd1FieldToReconcile` describes one field that is reconciled as a unit. That
is either a leaf of a column selected for ignore-null, or a whole column
outside the selection, since every upsert authors all of that column's leaves.
Reconciling it produces an `Scd1ReconciledField`, which holds the reconciled
value and the sequence that authored it; the version map is built from those
sequences.
Both reconciliations, within the microbatch and against the target row, use
these same two classes, so they define fields and build the version map the
same way. The transition is one-way, and neither class changes behavior until
follow-up tickets wire them in.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
