Anish Mahto created SPARK-59645:
-----------------------------------
Summary: SCD1 Ignore-null support; Refactor SCD1 microbatch
reconciliation into strategies
Key: SPARK-59645
URL: https://issues.apache.org/jira/browse/SPARK-59645
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
Default SCD1 reconciliation accepts or rejects an event wholesale, so all
columns share one row-level version. Ignore-null permits different events to
author different leaves, requiring fundamentally different deduplication and
tombstone reconciliation.
Refactor `Scd1BatchProcessor` to delegate to a reconciliation strategy and move
existing behavior into the default `Scd1RowLevelReconciliation`. This preserves
current behavior while establishing the boundary for future leaf-level
reconciliation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
