suhuruli opened a new pull request, #57149: URL: https://github.com/apache/spark/pull/57149
--- What changes were proposed in this pull request? This PR documents Auto CDC (Change Data Capture) support for Spark Declarative Pipelines (SDP) in the programming guide. It adds a new section, "Change Data Capture (CDC) with Auto CDC," to docs/declarative-pipelines-programming-guide.md, placed after the SQL programming section and before "Writing Data to External Targets with Sinks." The section covers: - Concept - what Auto CDC does, its SCD Type 1 semantics (keep only the current version of each row), and how it handles inserts, updates, deletes, and out-of-order/duplicate events, with a worked before/after example. - Requirements - the target must be a pre-created streaming table, the source must be a streaming source, and keys plus a sequencing expression are required. - Python API reference - the create_auto_cdc_flow function with a full parameter table. - SQL reference - both supported syntax forms (CREATE FLOW ... AS AUTO CDC INTO and CREATE STREAMING TABLE ... FLOW AUTO CDC), including required clause ordering. - End-to-end quickstart - a spark-pipelines init -> run walkthrough. - How-tos - handling deletes, selecting columns, out-of-order events, composite keys, and changing the key set (full refresh). - Limitations/considerations - SCD Type 1 only, streaming-table target only, immutable key set, and table-format requirements. No functional code is changed; this is documentation only. The content is grounded in the existing implementation: create_auto_cdc_flow in python/pyspark/pipelines/api.py, the autoCdcCommand grammar in sql/api/.../SqlBaseParser.g4, and the AutoCdcFlowDetails / SCDType definitions in sql/connect/common/src/main/protobuf/spark/connect/pipelines.proto. Why are the changes needed? Auto CDC was added to SDP but was undocumented in the programming guide, leaving users with no reference for how to apply change data capture to a target table. A new user landing on the SDP docs had no way to discover the feature, its Python or SQL syntax, or its semantics and limitations. This PR closes that gap so users can adopt Auto CDC from the official documentation. Does this PR introduce any user-facing change? Yes, but documentation only. It adds a new section to the Spark Declarative Pipelines programming guide. There is no change to behavior, APIs, or output. How was this patch tested? No tests were added; this is a documentation-only change. The Markdown was verified locally to ensure it is ASCII-only (per the repository convention) and that the internal anchor link (#spark-pipelines-run) resolves to an existing heading. All code snippets and API parameters were cross-checked against the current implementation (create_auto_cdc_flow, the autoCdcCommand SQL grammar, and the pipelines.proto definitions). Was this patch authored or co-authored using generative AI tooling? Yes -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
