Andreas Neumann created SPARK-58347:
---------------------------------------
Summary: Thread conf.resolver through
ColumnSelection.applyToSchema instead of a caseSensitive boolean
Key: SPARK-58347
URL: https://issues.apache.org/jira/browse/SPARK-58347
Project: Spark
Issue Type: Sub-task
Components: Declarative Pipelines
Affects Versions: 5.0.0
Reporter: Andreas Neumann
Follow-up from review of SPARK-58313 (PR #57490). AutoCDC column-name
resolution is currently driven by a caseSensitive: Boolean that each call site
derives from conf.caseSensitiveAnalysis, then re-derives a resolver (or uses
StructType.getFieldIndex / getFieldIndexCaseInsensitive) internally. A reviewer
suggested threading conf.resolver all the way through for futureproofing, so
resolution semantics live in one place rather than being reconstructed from a
boolean at each layer.
The blocker for doing this within SPARK-58313 was scope:
ColumnSelection.applyToSchema (and its helper lookupFieldIndices) take a
caseSensitive: Boolean and rely on StructType.getFieldIndex /
getFieldIndexCaseInsensitive, which are boolean-driven. Threading a Resolver
requires reworking applyToSchema/lookupFieldIndices to be resolver-based and
updating all callers.
Scope:
- Rework ColumnSelection.applyToSchema and lookupFieldIndices to accept/use a
Resolver (or a small resolution abstraction) instead of a Boolean.
- Update all callers to pass conf.resolver: Scd1BatchProcessor (2 sites),
Scd2BatchProcessor (preprocess projection + computeTrackedHistoryColumns), and
Flow (userSelectedSchema).
- Scd2BatchProcessor.computeTrackedHistoryColumns can then take a Resolver
directly instead of deriving one from a caseSensitive boolean.
This is a pure refactor with no behavior change (case-sensitive and
case-insensitive resolution must remain identical); it also touches the shared
SCD1 code paths, which is why it is intentionally deferred out of SPARK-58313.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
