anew opened a new pull request, #57490:
URL: https://github.com/apache/spark/pull/57490
### What changes were proposed in this pull request?
An SCD2 AutoCDC flow can restrict which columns define a "run" via TRACK
HISTORY ON (...), which populate ChangeArgs.trackHistorySelection. Until now,
that selection was only resolved when the first microbatch ran, inside
Scd2BatchProcessor.computeTrackedHistoryColumns during reconciliation. An
unresolvable or ineligible tracking column — one that is absent from the
source, is a key, is a reserved framework column, or was dropped by the flow's
column_list — therefore surfaced mid-stream rather than at flow construction,
unlike every other AutoCDC misconfiguration (keys, column selection, reserved
names), which fail eagerly.
This PR validates trackHistorySelection at AutoCdcMergeFlow construction
time, mirroring the existing requireKeysPresentInSelectedSchema check:
- The eligibility + resolution logic is extracted from
Scd2BatchProcessor.computeTrackedHistoryColumns into a schema-based companion
helper Scd2BatchProcessor.computeTrackedHistoryColumns(schema, changeArgs,
caseSensitive). Both the per-microbatch runtime path and the new
construction-time validator call it, so the two can never diverge. The refactor
is behavior-preserving.
- AutoCdcMergeFlow gains
requireTrackHistoryColumnsResolvableInSelectedSchema, invoked when deriving the
user-selected schema (right after the key-presence check). It runs before the
flow's schema is forced, so the actionable error surfaces ahead of the
temporary AUTOCDC_SCD2_NOT_SUPPORTED gate and remains correct once SCD2 support
lands.
- No new error condition: an unresolvable selection reuses the existing
AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA (schema name trackHistorySelection).
- The check is a no-op when trackHistorySelection is None, which covers
all SCD1 flows (enforced by ChangeArgs) and SCD2 flows that do not restrict
tracking.
### Why are the changes needed?
Deferring this validation to reconciliation means a simple typo or
misconfiguration (TRACK HISTORY ON (typo), or tracking a key/excluded column)
is not reported at graph analysis time; it only fails once data flows, with an
error raised deep in the SCD2 batch processor. Validating at flow construction
gives a fail-fast, user-actionable error consistent with the rest of the
AutoCDC configuration surface (keys, column_list, reserved names).
### Does this PR introduce any user-facing change?
Yes. An AutoCDC SCD2 flow whose TRACK HISTORY ON (...) references a column
that is not an eligible history-tracking column (absent, a key, a framework
column, or excluded by column_list) now fails at flow construction with
AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA instead of failing when the first
microbatch runs. Valid selections are unaffected, and there is no change for
SCD1 flows. (Note: SCD2 AutoCDC flows are not yet generally supported on master
— still gated by AUTOCDC_SCD2_NOT_SUPPORTED — so no released behavior changes.)
### How was this patch tested?
New unit tests in AutoCdcFlowSuite covering: an SCD2 flow tracking a
non-existent column, a key column (ineligible), and a column dropped by
columnSelection are each rejected at construction with
AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA; a resolvable selection passes the check
(falling through to the SCD2-not-supported gate); and
case-sensitive/insensitive resolution behavior.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
