AnishMahto opened a new pull request, #56436: URL: https://github.com/apache/spark/pull/56436
Approved AutoCDC SPIP: https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7 ------------ This is a stacked PR. Review incremental diff here: https://github.com/AnishMahto/spark/compare/SPARK-57222-SCD2-decompose-affected-rows...SPARK-57322-reconcile-start-end-at ### What changes were proposed in this pull request? **Preamble**: The SCD type 2 flow is a foreachBatch streaming query on an input change-data-feed, and is responsible for reconciling the incoming change data onto some target table that follows SCD2 replication semantics. SCD2 flows also maintain an "auxiliary" table to keep track of early-arriving out-of-order received events state. Each microbatch will need to reconcile against this auxiliary table as well, and update the auxiliary table's state appropriately for future microbatches. **Reconcile StartAt/EndAt**: Once we have all of the relevant rows for microbatch SCD2 reconciliation decomposed, we can begin setting the start/end-at for incoming microbatch rows, and rewriting existing start/end-at of existing rows in the aux/target tables to account for late arriving events. When iterating on the affected microbatch rows in chronologically sorted order _after decomposition_, we can reconcile a row's end-at by looking at its next neighbor, which naturally succeeds the row. Similarly we may need to rewrite both existing and new upsert-representing rows' start-at depending on whether the upsert row is a now a run-head or not. This can be done by looking backwards to identify the upsert run's head in chronological sorted order. ### Why are the changes needed? SCD2 core implementation. ### Does this PR introduce _any_ user-facing change? No, this is core implementation of an unreleased feature (AutoCDC SCD2). ### How was this patch tested? Unit tested in `Scd2BatchProcessorSuite`. ### Was this patch authored or co-authored using generative AI tooling? Co-authored with Claude Opus 4.7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
