AnishMahto opened a new pull request, #59603: URL: https://github.com/apache/spark/pull/59603
This is a stacked PR. See incremental diff here: https://github.com/AnishMahto/spark/compare/scd1-target-merge-strategy-refactor...scd1-leaf-level-target-merge. See previous PR in stack here: https://github.com/apache/spark/pull/59601 ### What changes were proposed in this pull request? Implement SCD1 target merge for ignore-null flows. Once per microbatch, the target merge applies the reconciled changes, at most one per key, to the target table. Today an upsert at least as new replaces the whole row. Under ignore-null, the incoming change and the target row can each hold the newest value of different fields, tracked by this stack's per-row version map of when each leaf field was last written. Introduce a new leaf-level merge strategy beside the existing whole-row one. It splits selected columns into leaves, takes each field from whichever side (microbatch or target) wrote it later, and rebuilds the columns and version map. A delete removes the row only if newer than its latest upsert, so later-written fields survive; ties still favor upserts. ### Why are the changes needed? SCD1 ignore null support. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Unit tests addd to `Scd1BatchProcessorMergeSuite`. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Opus 5.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
