AnishMahto opened a new pull request, #56457: URL: https://github.com/apache/spark/pull/56457
Approved AutoCDC SPIP: https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7 ------------ This is a stacked PR. Review incremental diff here: https://github.com/AnishMahto/spark/compare/SPARK-57356-cleanup-delete-encoded-rows-post-reconciliation...SPARK-57378-scd2-merge-reconciled-rows ### What changes were proposed in this pull request? **Preamble:** The SCD type 2 flow is a foreachBatch streaming query on an input change-data-feed, and is responsible for reconciling the incoming change data onto some target table that follows SCD2 replication semantics. SCD2 flows also maintain an "auxiliary" table to keep track of early-arriving out-of-order received events state. Each microbatch will need to reconcile against this auxiliary table as well, and update the auxiliary table's state appropriately for future microbatches. **Merge Reconciled Rows into Aux and Target Tables:** Once we have the full set of rows affected by the incoming microbatch in their post-reconciliation state, with redundant rows dropped, we're ready to persist this set of rows back to the aux/target tables. At a high level this is three step process: 1. Tag whether a row should land in the aux or target table, depending on what type of row it is. No-op upserts and tombstones go to aux, op upserts go to target table. 2. Merge rows headed for the aux into the aux table. Soft-delete rows from the aux if they have been promoted to the target table or made redundant 3. Merge rows headed for the target table. Hard-delete rows from the target table if they have been demoted to the aux table. ### Why are the changes needed? SCD2 core algorithm implementation. ### Does this PR introduce _any_ user-facing change? No, AutoCDC SCD2 is an unreleased feature. ### How was this patch tested? Regular unit tests added in `Scd2BatchProcessorSuite`, unit tests that require catalog support for merge in `Scd2BatchProcessorMergeSuite`. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Claude Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
