anew opened a new pull request, #57610:
URL: https://github.com/apache/spark/pull/57610
### What changes were proposed in this pull request?
This is a follow-up refactor for SDP AutoCDC. It reworks
`ColumnSelection.applyToSchema` (and its private helper `lookupFieldIndices`)
to accept a `Resolver` instead of a `caseSensitive: Boolean`:
- `ColumnSelection.applyToSchema` / `lookupFieldIndices` now take a
`Resolver`. `lookupFieldIndices` resolves each requested column via
`schema.fieldNames.indexWhere(resolver(_, name))` rather than switching between
`StructType.getFieldIndex` / `getFieldIndexCaseInsensitive` on a boolean.
- `Scd2BatchProcessor.computeTrackedHistoryColumns` now takes a `Resolver`
directly instead of deriving `caseSensitiveResolution` /
`caseInsensitiveResolution` from a boolean.
- All callers pass `spark.sessionState.conf.resolver` instead of
`conf.caseSensitiveAnalysis`:
`Scd1BatchProcessor.projectTargetColumnsOntoMicrobatch` (2 sites),
`Scd2BatchProcessor.projectTargetColumnsOntoMicrobatch` (2 sites) and its
`computeTrackedHistoryColumns` instance method, and `AutoCdcMergeFlow` (the
`userSelectedSchema` projection and the construction-time track-history
validation).
- The `caseSensitivity` message parameter of
`AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA` is preserved via a new
`CaseSensitivityLabels.of(resolver)` overload that classifies the resolver by
probing it (`!resolver("a", "A")`), so error messages are byte-for-byte
unchanged.
### Why are the changes needed?
The AutoCDC code already resolves identifiers everywhere else through
`conf.resolver` (the canonical Spark abstraction for case-aware identifier
comparison). `applyToSchema` was the odd one out, threading a raw
`caseSensitive` boolean and re-deriving a resolver at each layer. Passing the
`Resolver` directly makes column matching consistent with the rest of the
pipeline, removes the boolean-to-resolver round-trips, and lets
`computeTrackedHistoryColumns` take the resolver it actually needs rather than
reconstructing one.
### Does this PR introduce _any_ user-facing change?
No. This is a pure refactor with no behavior change; the `caseSensitivity`
label in the `AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA` error is preserved.
### How was this patch tested?
Existing tests, updated to pass resolvers instead of booleans:
- `ChangeArgsSuite` and `AutoCdcFlowSuite` (65 tests) — including the
case-sensitive/insensitive selection and missing-column error-message cases.
- `Scd2BatchProcessor*` and `Scd1BatchProcessorMergeSuite` (150 tests) —
exercising the microbatch projection and history-tracking paths under both
case-sensitivity settings.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
