anew opened a new pull request, #57176: URL: https://github.com/apache/spark/pull/57176
### What changes were proposed in this pull request? Hook up the two AUTO CDC SQL constructs introduced in SPARK-56249 to the Spark Declarative Pipelines dataflow graph in `SqlGraphRegistrationContext`: 1. `CREATE STREAMING TABLE <name> FLOW AUTO CDC FROM <source> ...`, parsed into `CreateStreamingTableAutoCdc`, now registers the streaming table and an `AutoCdcFlow` that targets it. 2. `CREATE FLOW <name> AS AUTO CDC INTO <target> FROM <source> ...`, parsed into a `CreateFlowCommand` wrapping an `AutoCdcIntoCommand`, now registers an `AutoCdcFlow` from the named flow into the target dataset. A shared `buildChangeArgs` helper converts the parse-time expressions and unresolved attributes into the `ChangeArgs` consumed by `AutoCdcFlow` (keys, sequencing, delete condition, include/exclude column selection). SQL AUTO CDC only supports SCD Type 1, matching the Connect/proto path. ### Why are the changes needed? Before this change the AUTO CDC SQL syntax parsed successfully but was rejected during graph registration, so it could not be used to define pipeline datasets or flows. ### Does this PR introduce _any_ user-facing change? Yes. AUTO CDC SQL statements now register datasets and flows in a pipeline. ### How was this patch tested? Added registration and end-to-end execution tests to `SqlPipelineSuite` covering both syntaxes, the optional clauses (APPLY AS DELETE WHEN, COLUMNS, COLUMNS * EXCEPT), and the multipart-flow-name error. Full `SqlPipelineSuite` passes (50 tests). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
