anew opened a new pull request, #57625:
URL: https://github.com/apache/spark/pull/57625
### What changes were proposed in this pull request?
Follow-up for AutoCDC SCD2 (SPARK-56249). Adds two configuration-drift
validations for AutoCDC targets, closing gaps left by the existing key-column
and SCD-type drift checks. Both are checked on an incremental run against
values persisted when the auxiliary table was created, and both direct the user
to a full refresh as the remedy.
- **Sequencing-type drift**
(`AUTOCDC_INVALID_STATE.SEQUENCING_TYPE_DRIFT`): the sequencing *expression*
may legitimately change across runs (e.g. a new timestamp parse format), but
its resolved result *type* must not — the target persists the sequencing type
inside
`_cdc_metadata` (and, for SCD2, in the `__START_AT` / `__END_AT` interval
columns), so a changed type would make new events incomparable with the
persisted history. Validated at the target level, before the target-table
merge, for both SCD1 and SCD2.
- **Track-history drift** (`AUTOCDC_INVALID_STATE.TRACK_HISTORY_DRIFT`,
SCD2 only): the resolved SCD2 track-history column set defines what constitutes
a run (a change in any tracked column opens a new historical record), so
changing the set would reinterpret already-reconciled history. The effective
set (an explicit `TRACK HISTORY ON` selection, or the default of every eligible
non-key/non-framework column) is resolved via
`Scd2BatchProcessor.computeTrackedHistoryColumns` — the same source of truth
the reconciler uses — persisted as a new auxiliary-table property
`trackHistoryColumnNames`, and compared order-insensitively on later runs.
Supporting changes:
- New auxiliary-table property `pipelines.autocdc.trackHistoryColumnNames`
(JSON array; SCD2-only, absent for SCD1).
- `AutoCdcAuxiliaryTableSpec` gains `expectedSequencingType` and
`expectedTrackHistoryColumnNames`.
- `DatasetManager` runs the target-level sequencing-type check before
`materializeTable` evolves the target; the aux-level checks run in
`materializeAuxiliaryTable`.
- Reworded `AUXILIARY_TABLE_PROPERTY_MISSING` to speak of "the AutoCDC
configuration" generally rather than "key columns" specifically, since more
properties are now validated.
### Why are the changes needed?
AutoCDC already rejects key-column and SCD-type drift on incremental runs,
because those silently corrupt the persisted state. Sequencing type and (for
SCD2) the track-history column set have the same property: keeping them
constant is an invariant the reconciled on-disk state depends on, but neither
was validated. Without these checks a changed sequencing type surfaces as an
opaque type-mismatch mid-merge, and a changed track-history set silently
reinterprets history. Both now fail fast with an actionable error pointing at
full refresh.
### Does this PR introduce _any_ user-facing change?
Yes. An incremental AutoCDC run whose sequencing result type differs from
the recorded type now fails with `AUTOCDC_INVALID_STATE.SEQUENCING_TYPE_DRIFT`
(SQLSTATE 42000); an SCD2 run that changes its effective track-history column
set now fails with `AUTOCDC_INVALID_STATE.TRACK_HISTORY_DRIFT`. Both messages
name the conflicting vs recorded values and recommend a full refresh.
Reordering the same track-history columns, or changing the sequencing
expression while keeping its type, is unaffected. This is within the unreleased
AutoCDC feature on master.
### How was this patch tested?
New `AutoCdcConfigDriftSuite` (5 tests):
- SCD1 and SCD2 sequencing-type drift each trigger `SEQUENCING_TYPE_DRIFT`.
- Changing the sequencing expression while keeping the same type is allowed.
- Changing the SCD2 `TRACK HISTORY ON` column set triggers
`TRACK_HISTORY_DRIFT`.
- Reordering the same track-history columns does not trigger drift.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
