anew opened a new pull request, #57434: URL: https://github.com/apache/spark/pull/57434
### What changes were proposed in this pull request? Add SQL syntax for selecting the SCD type and history-tracking columns of an AUTO CDC flow, mirroring the recently added Python/Connect surface. Both SQL AUTO CDC forms -- `CREATE STREAMING TABLE ... FLOW AUTO CDC ...` and `CREATE FLOW ... AS AUTO CDC INTO ...` -- now accept: - `STORED AS SCD TYPE 1 | 2` (defaults to SCD Type 1 when omitted), and - `TRACK HISTORY ON (<cols>)` / `TRACK HISTORY ON * EXCEPT (<cols>)` under SCD Type 2. Specifically: - Grammar: add multi-word lexer tokens `SCD_TYPE_1`/`SCD_TYPE_2` and `HISTORY`/`TRACK` keywords; add `autoCdcStoredAsClause` and `autoCdcTrackHistoryClause`, appended in fixed order after the existing `COLUMNS` clause. `HISTORY`/`TRACK` are registered as non-reserved. - AST/plans: thread `storedAsScdType` (Int, default 1) and the track-history column lists through `AutoCdcParams`, `AutoCdcInto`, and `CreateStreamingTableAutoCdc`. - Registration: `SqlGraphRegistrationContext.buildChangeArgs` maps the SCD type onto `ScdType`, builds `trackHistorySelection`, and rejects TRACK HISTORY under SCD1 as well as specifying both track-history lists. The SCD type and track-history columns feed the existing `ChangeArgs` model (`storedAsScdType` / `trackHistorySelection`), the same contract the engine's SCD2 execution work consumes. ### Why are the changes needed? SQL AUTO CDC previously only supported SCD Type 1 and rejected `STORED AS SCD TYPE 2` / `TRACK HISTORY` at parse time. SCD Type 2 (history tracking) is a common CDC requirement and is already modeled in the engine's `ChangeArgs` / `ScdType`, so the SQL surface was the missing piece. ### Does this PR introduce any user-facing change? Yes. SQL AUTO CDC statements accept `STORED AS SCD TYPE 1|2` and, under SCD Type 2, `TRACK HISTORY ON ...`. Previously these clauses failed to parse. This is a change within the unreleased master branch only. ### How was this patch tested? New parser tests in `AutoCdcParserSuite` (SCD type 1/2, TRACK HISTORY include/except, all clauses combined, clause-ordering and invalid-type negative cases) and registration tests in `SqlPipelineSuite` (SCD2 and track-history map onto `ChangeArgs`; TRACK HISTORY without SCD2 is rejected). Ran `AutoCdcParserSuite`, `SQLKeywordSuite`, and the AUTO CDC `SqlPipelineSuite` cases. ### Was this patch authored or co-authored using generative AI tooling? Co-authored-by: Opus 4.8 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
