Anish Mahto created SPARK-60350:
-----------------------------------
Summary: SCD1 Ignore-null support; Target table merge
Key: SPARK-60350
URL: https://issues.apache.org/jira/browse/SPARK-60350
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 4.4.0
Reporter: Anish Mahto
Implement SCD1 target merge for ignore-null flows.
Once per microbatch, the target merge applies the reconciled changes, at most
one per key, to the target table. Today an upsert at least as new replaces the
whole row. Under ignore-null, the incoming change and the target row can each
hold the newest value of different fields, tracked by this stack's per-row
version map of when each leaf field was last written.
Introduce a new leaf-level merge strategy beside the existing whole-row one. It
splits selected columns into leaves, takes each field from whichever side
(microbatch or target) wrote it later, and rebuilds the columns and version
map. A delete removes the row only if newer than its latest upsert, so
later-written fields survive; ties still favor upserts.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
