AnishMahto opened a new pull request, #55996: URL: https://github.com/apache/spark/pull/55996
Approved AutoCDC SPIP: https://lists.apache.org/thread/j6sj9wo9odgdpgzlxtvhoy7szs0jplf7 -------- This is a stacked PR. Review incremental diff here: https://github.com/AnishMahto/spark/compare/SPARK-56923-SCD1-merge-microbatch-onto-auxiliary...SPARK-56927-SCD1-merge-microbatch-onto-target -------- Preamble: The SCD type 1 flow is a foreachBatch streaming query on an input change-data-feed, and is responsible for reconciling the incoming change data onto some target table that follows SCD1 replication semantics. SCD1 flows also maintain an "auxiliary" table to keep track of early-arriving out-of-order received events state. Each microbatch will need to reconcile against this auxiliary table as well, and update the auxiliary table's state appropriately for future microbatches. Merge Microbatch onto Target: The final part of an SCD1 reconciliation - merge the reconciled dataframe (deduplicated, target columns and operational metadata projected, and tombstone applied) onto the target table. The target table's contents represent the SCD1 transformation of the change data feed, for the specified target column selection in the `ChangeArgs`. Merging onto the target under SCD1 semantics means deleting existing rows that match against the reconciled microbatch's delete events, and inserting or updating rows as appropriate for the reconciled microbatch's upsert events. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
