anew opened a new pull request, #57584: URL: https://github.com/apache/spark/pull/57584
### What changes were proposed in this pull request? Adds `Scd2MergeStreamingWrite` (mirroring `Scd1MergeStreamingWrite`): it resolves the auxiliary-table identifier, constructs an `Scd2ForeachBatchHandler` over an `Scd2BatchProcessor`, and drives it via Structured Streaming `foreachBatch`. `FlowPlanner` now routes an SCD2 `AutoCdcMergeFlow` to it, replacing the `AUTOCDC_SCD2_NOT_SUPPORTED` throw. ### Why are the changes needed? This removes the **last** remaining SCD2 gate. The flow-schema derivation (SPARK-58319), the auxiliary-table spec (SPARK-58320), the reserved-column / track-history validation (SPARK-57251, SPARK-58313), and the per-microbatch reconciliation handler (SPARK-57395) have all merged; this change makes SCD2 AutoCDC flows runnable end to end. ### Does this PR introduce _any_ user-facing change? Yes: `AUTO CDC ... STORED AS SCD TYPE 2` pipelines are now supported and executable, where previously they failed with `AUTOCDC_SCD2_NOT_SUPPORTED`. ### How was this patch tested? New `AutoCdcScd2SinglePipelineSuite` runs SCD2 flows end to end through the pipeline and asserts SCD2 target semantics (an upsert opens a current record; an update closes the prior record and opens a new one; a delete closes the current record) plus auxiliary-table materialization. Also refreshed an `AutoCdcFlowSuite` test whose name/comment referenced the removed gate. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
