Anish Mahto created SPARK-60278:
-----------------------------------
Summary: SCD1 Ignore-null support; Merge to target table before
auxiliary table
Key: SPARK-60278
URL: https://issues.apache.org/jira/browse/SPARK-60278
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
Each microbatch is validated and reconciled into one net row per key, then
merged into both auxiliary and target tables before completion. Today auxiliary
merges first, which is safe for row-level reconciliation because tombstones
only reject older whole-row events. Under leaf-level reconciliation, a
tombstone can null fields authored by some upserts in the microbatch and then
be removed due to an overall newer upsert in the same microbatch, so failure
before target merge makes retry lose that input.
The solution is to reverse the merge order, merging to the target before the
auxiliary table. Row-level reconciliation remains correct with the same
observed behavior, but now leaf-level reconciliation can safely be retried
after a failure between table merges too.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
