lisancao opened a new pull request, #60086: URL: https://github.com/apache/spark/pull/60086
### What changes were proposed in this pull request? This PR documents SCD Type 2 for Auto CDC in the Declarative Pipelines programming guide. It continues @suhuruli's work in #57149, which documented both SCD types before it was narrowed to Type 1 and left Type 2 to SPARK-58570. It starts from the Type 2 text in #57149 at fc03e68011, keeps its structure and the end-to-end comparison, and brings it up to date with master and the merged Type 1 section. It adds an "SCD Type 2: Keeping History" section to the Auto CDC part of the guide. In the existing section, it replaces the SPARK-58570 placeholders with links to the new section, adds the Type 2 options to the parameter table and the SQL clause list, and scopes the delete, reserved-name and explicit-schema notes that only held for Type 1. Changes from the #57149 text, each found by running it: - The sequencing column is tracked by default and changes on every event, so the original coalescing example (excluding only `status`) produced two versions, not one. The section now says so, and the example tracks `name` only. - Changing the tracked columns or the SCD type across runs now fails with `AUTOCDC_INVALID_STATE.TRACK_HISTORY_DRIFT` or `SCD_TYPE_DRIFT` (SPARK-58391) and needs a full refresh. Under default or `* EXCEPT` tracking, adding or removing a target column changes the tracked set too. - Late deletes and late events inside an existing version split it; a delete for a key with no earlier version adds no row. Ignore-null updates are on master and branch-4.x but not branch-4.3, so they are left for a separate docs JIRA. ### Why are the changes needed? SCD Type 2 shipped in Spark 4.3.0 (SPARK-58247, SPARK-58261, SPARK-58321) with no user documentation; the guide links to SPARK-58570 in its place. ### Does this PR introduce _any_ user-facing change? No. Documentation only. ### How was this patch tested? Built the docs with `SKIP_API=1 bundle exec jekyll build` and checked every anchor on the page. Every example and output in the new section was run on the Spark 4.3.0 release and on a master build (11e5a465c51), with `spark-pipelines --remote` against a Connect server whose catalog is `SharedTablesInMemoryRowLevelOperationTableCatalog` from the `spark-catalyst` test jar, since no released lakehouse connector supports Spark 4.3 yet. Both builds gave the same results: | Claim | How it was run | |---|---| | End-to-end example with the Type 2 target, after each run | the page's code as written | | Current-version and as-of queries | the page's SQL | | Late events before and inside the history | the page's Python flow, two runs | | Coalescing with `track_history_column_list=["name"]`; three versions with default tracking | the page's Python and both SQL forms | | Deletes, late deletes, re-insert after a delete, a delete before any version | separate pipelines | | Track history with Type 1 is rejected (Python client side, SQL at registration) | separate pipelines | | Tracking a key or an excluded column fails with `AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA` | separate pipelines | | A Type 2 source with `__START_AT` fails with `AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT` | separate pipeline | | `TRACK_HISTORY_DRIFT`, `SCD_TYPE_DRIFT`, adding a column under default tracking, full refresh | repeated runs of one pipeline | The `create_auto_cdc_flow` signature, the default SCD type when it is omitted (`PipelinesHandler.scala`), the Type 2 target schema (`Flow.scala`) and the `autoCdcTrackHistoryClause` grammar were checked on master and branch-4.3. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (claude-opus-5-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
