[
https://issues.apache.org/jira/browse/SPARK-59787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-59787:
-----------------------------------
Labels: pull-request-available (was: )
> SCD1 Ignore-null support; Coalesce incoming microbatch rows per key
> -------------------------------------------------------------------
>
> Key: SPARK-59787
> URL: https://issues.apache.org/jira/browse/SPARK-59787
> Project: Spark
> Issue Type: Sub-task
> Components: Declarative Pipelines
> Affects Versions: 4.4.0
> Reporter: Anish Mahto
> Priority: Major
> Labels: pull-request-available
>
> A microbatch can contain multiple CDC events targeting the same key. With
> SCD1 ignore-null handling, different events may author different leaves, so
> selecting a single winning row is insufficient.
> Instead, collapse all microbatch rows for each key into one synthetic row.
> For every user-data leaf, select the value from the latest event that authors
> that leaf. Upserts determine authorship from their version maps, while
> deletes author null for every leaf.
> The resulting row contains the winning value and authorship sequence for
> every leaf, a reconstructed version map, and the greatest row-wide upsert and
> delete sequences from the microbatch.
> This ticket only reconciles events within one microbatch. It does not
> reconcile against persisted auxiliary-table tombstones or existing
> target-table state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
