AnishMahto opened a new pull request, #59045: URL: https://github.com/apache/spark/pull/59045
This is a stacked PR. See incremental diff here: https://github.com/AnishMahto/spark/compare/scd1-per-leaf-metadata-schema...scd1-populate-version-map-at-ingest See previous PR in stack here: https://github.com/apache/spark/pull/59044 ### What changes were proposed in this pull request? A microbatch can contain multiple CDC events targeting the same key. With SCD1 ignore-null handling, different events may author different leaves, so selecting a single winning row is insufficient. Instead, collapse all microbatch rows for each key into one synthetic row. For every user-data leaf, select the value from the latest event that authors that leaf. Upserts determine authorship from their version maps, while deletes author null for every leaf. The resulting row contains the winning value and authorship sequence for every leaf, a reconstructed version map, and the greatest row-wide upsert and delete sequences from the microbatch. This PR only reconciles events within one microbatch. It does not reconcile against persisted auxiliary-table tombstones or existing target-table state. ### Why are the changes needed? SCD1 ignore-null support. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit tests in `Scd1LeafLevelReconciliationSuite`. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with GPT 5.6 Sol. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
