szehon-ho commented on code in PR #57149:
URL: https://github.com/apache/spark/pull/57149#discussion_r3761873939
##########
docs/declarative-pipelines-programming-guide.md:
##########
@@ -517,6 +517,273 @@ AS INSERT INTO customers_us
SELECT * FROM STREAM(customers_us_east);
```
+## Change Data Capture (CDC) with Auto CDC
+
+Many source systems emit a stream of *change events* rather than a snapshot of
the current data: each record describes an insert, update, or delete to a row,
identified by a key. Applying these events correctly to a target table by hand
is tricky. You have to match events to existing rows, apply them in the right
order, and handle out-of-order and duplicate events without corrupting the
table.
+
+**Auto CDC** does this for you. You point it at a source of change events and
tell it how to identify and order them, and SDP maintains a target streaming
table that always reflects the latest state for each key.
+
+### What Auto CDC does
+
+Given an ordered stream of change events, Auto CDC keeps the target table in
sync with the source:
+
+- **Inserts and updates** - For each key, the event with the highest sequence
value wins. If no row exists for the key, it's inserted; if one exists, it's
overwritten with the latest values.
+- **Deletes** - Events that match a delete condition you supply remove the
corresponding row from the target.
+- **Out-of-order events** - Events don't have to arrive in order. Auto CDC
uses the sequencing expression to determine the latest state per key, so a
late-arriving event with a lower sequence value doesn't overwrite newer data.
+
+This behavior implements **Slowly Changing Dimensions (SCD) Type 1**: the
target keeps only the current version of each row, with no history of prior
values. SCD Type 1 is the only mode currently supported.
+
+For example, given these change events (ordered by `version`):
+
+| id | name | version | op |
+|----|----------|---------|--------|
+| 1 | alice | 1 | UPSERT |
+| 2 | bob | 1 | UPSERT |
+| 1 | alice_v2 | 2 | UPSERT |
+| 2 | bob | 2 | DELETE |
+| 3 | carol | 1 | UPSERT |
Review Comment:
@anew heads up: the out-of-order event that was added for this thread is no
longer in the concept example. The latest force-push (e9f1e5f) trimmed that
table back to five in-order events and collapsed the by-key walkthrough, so the
first example no longer shows an out-of-order arrival. Line 562 now forwards to
the end-to-end example instead, which does still demonstrate it -- batch 2
carries `id 1`'s two events newest-first, and batch 3 re-delivers the stale one
in a later run.
Nothing is inaccurate as written, so this is a call on where the behavior is
best shown rather than a correctness issue.
If it should be shown here again, it is a cheap change: a stale re-delivery
cannot alter a Type 1 result, since the highest sequence value per key still
wins, so the target table below stays exactly as printed -- and this is the
same example that was already replayed through the merge engine in the earlier
revision. Three edits:
1. Re-add the trailing row to the change-events table:
`| 1 | alice | 1 | UPSERT |`
2. Restore the lead-in on line 545: "For example, take these change events.
They are **not** in `version` order - the last event for `id 1` is a stale
re-delivery of version 1, arriving after version 2:"
3. Add the discard clause to line 562: "...then updated to `alicia` at
version 2, and the stale version 1 event arriving last is discarded because a
higher sequence value has already been applied; ..."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
