Anish Mahto created SPARK-60347:
-----------------------------------
Summary: SCD1 Ignore-null support; Refactor SCD1 target merge into
strategies
Key: SPARK-60347
URL: https://issues.apache.org/jira/browse/SPARK-60347
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
With ignore-null (leaf-level reconciliation), merging the reconciled microbatch
onto the target table requires additional logic.
With row-level reconciliation, a microbatch row replaces a target row wholesale
when its upsert sequence is at least as new as the target row's. With
leaf-level reconciliation, a microbatch row can update only some of a target
row's fields, because each leaf is versioned individually. The merge must also
write each row's updated version map, whereas row level always writes a null
version map.
In preparation for these differences, we should extract the target merge into
its own strategy class, which Scd1BatchProcessor creates for each merge, so
that a leaf-level strategy can be added beside it. Common expressions and
references move into a shared abstract base class, and each strategy implements
its own merge. This is a pure refactor: the row-level strategy is the existing
merge, moved without logic changes.
Keeping row level as a separate strategy keeps the default path identical to
today's merge and free of any per-leaf work.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
