This is an automated email from the ASF dual-hosted git repository.
szehon-ho pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new cddeec4151e9 [SPARK-58347][SDP] Thread conf.resolver through
ColumnSelection.applyToSchema
cddeec4151e9 is described below
commit cddeec4151e9d3474a83c835919a6f259ddf0ccc
Author: Andreas Neumann <[email protected]>
AuthorDate: Wed Jul 29 11:44:46 2026 -0700
[SPARK-58347][SDP] Thread conf.resolver through
ColumnSelection.applyToSchema
### What changes were proposed in this pull request?
This is a follow-up refactor for SDP AutoCDC. It reworks
`ColumnSelection.applyToSchema` (and its private helper `lookupFieldIndices`)
to accept a `Resolver` instead of a `caseSensitive: Boolean`:
- `ColumnSelection.applyToSchema` / `lookupFieldIndices` now take a
`Resolver`. `lookupFieldIndices` resolves each requested column via
`schema.fieldNames.indexWhere(resolver(_, name))` rather than switching between
`StructType.getFieldIndex` / `getFieldIndexCaseInsensitive` on a boolean.
- `Scd2BatchProcessor.computeTrackedHistoryColumns` now takes a
`Resolver` directly instead of deriving `caseSensitiveResolution` /
`caseInsensitiveResolution` from a boolean.
- All callers pass `spark.sessionState.conf.resolver` instead of
`conf.caseSensitiveAnalysis`:
`Scd1BatchProcessor.projectTargetColumnsOntoMicrobatch` (2 sites),
`Scd2BatchProcessor.projectTargetColumnsOntoMicrobatch` (2 sites) and its
`computeTrackedHistoryColumns` instance method, and `AutoCdcMergeFlow` (the
`userSelectedSchema` projection and the construction-time track-history
validation).
- The `caseSensitivity` message parameter of
`AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA` is preserved via a new
`CaseSensitivityLabels.of(resolver)` overload that classifies the resolver by
probing it (`!resolver("a", "A")`), so error messages are byte-for-byte
unchanged.
### Why are the changes needed?
The AutoCDC code already resolves identifiers everywhere else through
`conf.resolver` (the canonical Spark abstraction for case-aware identifier
comparison). `applyToSchema` was the odd one out, threading a raw
`caseSensitive` boolean and re-deriving a resolver at each layer. Passing the
`Resolver` directly makes column matching consistent with the rest of the
pipeline, removes the boolean-to-resolver round-trips, and lets
`computeTrackedHistoryColumns` take the resolver it actually n [...]
### Does this PR introduce _any_ user-facing change?
No. This is a pure refactor with no behavior change; the `caseSensitivity`
label in the `AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA` error is preserved.
### How was this patch tested?
Existing tests, updated to pass resolvers instead of booleans:
- `ChangeArgsSuite` and `AutoCdcFlowSuite` (65 tests) — including the
case-sensitive/insensitive selection and missing-column error-message cases.
- `Scd2BatchProcessor*` and `Scd1BatchProcessorMergeSuite` (150 tests) —
exercising the microbatch projection and history-tracking paths under both
case-sensitivity settings.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57610 from anew/spark-58347-resolver-columnselection.
Authored-by: Andreas Neumann <[email protected]>
Signed-off-by: Szehon Ho <[email protected]>
(cherry picked from commit 8df89f20fc5c800ef4935c1b49a7525ec9df8921)
Signed-off-by: Szehon Ho <[email protected]>
---
.../spark/sql/pipelines/autocdc/ChangeArgs.scala | 46 +++++++++++++++-------
.../sql/pipelines/autocdc/Scd1BatchProcessor.scala | 7 ++--
.../sql/pipelines/autocdc/Scd2BatchProcessor.scala | 15 ++++---
.../apache/spark/sql/pipelines/graph/Flow.scala | 16 +++-----
.../sql/pipelines/autocdc/ChangeArgsSuite.scala | 27 +++++++------
5 files changed, 60 insertions(+), 51 deletions(-)
diff --git
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgs.scala
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgs.scala
index 15aa535994f9..a8c3485da31c 100644
---
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgs.scala
+++
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgs.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.pipelines.autocdc
import org.apache.spark.SparkException
import org.apache.spark.sql.{AnalysisException, Column}
+import org.apache.spark.sql.catalyst.analysis.{caseInsensitiveResolution,
caseSensitiveResolution, Resolver}
import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
import org.apache.spark.sql.catalyst.util.QuotingUtils
import org.apache.spark.sql.types.StructType
@@ -72,26 +73,26 @@ object ColumnSelection {
* @param schema The schema to filter.
* @param columnSelection The user-provided selection. `None` is a no-op and
returns `schema`
* unchanged.
- * @param caseSensitive Whether to match column names case-sensitively
against the schema.
- * Callers should derive this from the session, e.g.
- * `session.sessionState.conf.caseSensitiveAnalysis`,
so column matching
- * stays consistent with `spark.sql.caseSensitive`.
+ * @param resolver Determines whether two column names are considered
equal. Callers
+ * should pass the session resolver, e.g.
+ * `session.sessionState.conf.resolver`, so column
matching stays
+ * consistent with `spark.sql.caseSensitive`.
*/
def applyToSchema(
schemaName: String,
schema: StructType,
columnSelection: Option[ColumnSelection],
- caseSensitive: Boolean): StructType = columnSelection match {
+ resolver: Resolver): StructType = columnSelection match {
case None =>
// A None column selection is interpreted as a no-op.
schema
case Some(IncludeColumns(cols)) =>
- val keepIndices = lookupFieldIndices(schemaName, schema, cols,
caseSensitive)
+ val keepIndices = lookupFieldIndices(schemaName, schema, cols, resolver)
StructType(schema.fields.zipWithIndex.collect {
case (field, idx) if keepIndices.contains(idx) => field
})
case Some(ExcludeColumns(cols)) =>
- val dropIndices = lookupFieldIndices(schemaName, schema, cols,
caseSensitive)
+ val dropIndices = lookupFieldIndices(schemaName, schema, cols, resolver)
StructType(schema.fields.zipWithIndex.collect {
case (field, idx) if !dropIndices.contains(idx) => field
})
@@ -101,17 +102,20 @@ object ColumnSelection {
schemaName: String,
schema: StructType,
fields: Seq[UnqualifiedColumnName],
- caseSensitive: Boolean): Set[Int] = {
- val caseAwareGetFieldIndex: String => Option[Int] =
- if (caseSensitive) schema.getFieldIndex else
schema.getFieldIndexCaseInsensitive
-
- val fieldIndexResolutions = fields.map(f => f ->
caseAwareGetFieldIndex(f.name))
+ resolver: Resolver): Set[Int] = {
+ def resolveFieldIndex(name: String): Option[Int] =
+ schema.fieldNames.indexWhere(resolver(_, name)) match {
+ case -1 => None
+ case idx => Some(idx)
+ }
+
+ val fieldIndexResolutions = fields.map(f => f -> resolveFieldIndex(f.name))
val missingFieldNames = fieldIndexResolutions.collect { case (f, None) =>
f.name }.distinct
if (missingFieldNames.nonEmpty) {
throw new AnalysisException(
errorClass = "AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA",
messageParameters = Map(
- "caseSensitivity" -> CaseSensitivityLabels.of(caseSensitive),
+ "caseSensitivity" -> CaseSensitivityLabels.of(resolver),
"schemaName" -> schemaName,
"missingColumns" -> missingFieldNames.mkString(", "),
"availableColumns" -> schema.fieldNames.mkString(", ")
@@ -127,8 +131,20 @@ private[pipelines] object CaseSensitivityLabels {
val CaseSensitive: String = "case-sensitive"
val CaseInsensitive: String = "case-insensitive"
- def of(caseSensitive: Boolean): String =
- if (caseSensitive) CaseSensitive else CaseInsensitive
+ /**
+ * Maps a [[Resolver]] to its user-facing label. Classifies by reference
identity against the
+ * two session resolver singletons, matching
`SchemaUtils.isCaseSensitiveAnalysis`.
+ */
+ def of(resolver: Resolver): String = {
+ if (resolver == caseSensitiveResolution) {
+ CaseSensitive
+ } else if (resolver == caseInsensitiveResolution) {
+ CaseInsensitive
+ } else {
+ // this should be unreachable because `conf.resolver` only ever returns
one of these two.
+ throw SparkException.internalError(s"Unknown resolver: $resolver")
+ }
+ }
}
/** The SCD (Slowly Changing Dimension) strategy for a CDC flow. */
diff --git
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1BatchProcessor.scala
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1BatchProcessor.scala
index 35006dc4ee21..397775513703 100644
---
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1BatchProcessor.scala
+++
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1BatchProcessor.scala
@@ -163,8 +163,7 @@ case class Scd1BatchProcessor(
*/
private[autocdc] def projectTargetColumnsOntoMicrobatch(
microbatchWithCdcMetadataDf: DataFrame): DataFrame = {
- val caseSensitiveColumnComparison =
-
microbatchWithCdcMetadataDf.sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val resolver =
microbatchWithCdcMetadataDf.sparkSession.sessionState.conf.resolver
// The user schema is the microbatch schema after dropping the system CDC
metadata column.
// We project out the system column before applying user selection and
project it back in
@@ -178,7 +177,7 @@ case class Scd1BatchProcessor(
Seq(UnqualifiedColumnName(AutoCdcReservedNames.cdcMetadataColName))
)
),
- caseSensitive = caseSensitiveColumnComparison
+ resolver = resolver
)
val userSelectedColumnsInMicrobatchSchema =
@@ -186,7 +185,7 @@ case class Scd1BatchProcessor(
schemaName = "microbatch",
schema = userColumnsInMicrobatchSchema,
columnSelection = changeArgs.columnSelection,
- caseSensitive = caseSensitiveColumnComparison
+ resolver = resolver
)
// In addition to the explicit user-selected columns, re-project the
operational CDC metadata
diff --git
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
index 125f6c895f41..c93460c2ad81 100644
---
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
+++
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
@@ -21,7 +21,7 @@ import org.apache.spark.SparkException
import org.apache.spark.sql.{functions => F}
import org.apache.spark.sql.Column
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.analysis.{caseInsensitiveResolution,
caseSensitiveResolution}
+import org.apache.spark.sql.catalyst.analysis.Resolver
import org.apache.spark.sql.catalyst.expressions.{CreateMap, If, Literal,
RaiseError}
import org.apache.spark.sql.catalyst.util.QuotingUtils
import org.apache.spark.sql.classic.{DataFrame, ExpressionUtils}
@@ -199,7 +199,7 @@ case class Scd2BatchProcessor(
private def projectTargetColumnsOntoMicrobatch(
microbatch: DataFrame
): DataFrame = {
- val caseSensitive =
microbatch.sparkSession.sessionState.conf.caseSensitiveAnalysis
+ val resolver = microbatch.sparkSession.sessionState.conf.resolver
// Strip the framework columns through the same case-aware path as the
user selection, for
// consistency with Scd1BatchProcessor.projectTargetColumnsOntoMicrobatch.
@@ -211,14 +211,14 @@ case class Scd2BatchProcessor(
Scd2BatchProcessor.reservedFrameworkColNames.toSeq.map(UnqualifiedColumnName(_))
)
),
- caseSensitive = caseSensitive
+ resolver = resolver
)
val userSelectedDataSchema =
ColumnSelection.applyToSchema(
schemaName = "microbatch",
schema = dataSchema,
columnSelection = changeArgs.columnSelection,
- caseSensitive = caseSensitive
+ resolver = resolver
)
val finalColumnsToSelect: Seq[Column] =
userSelectedDataSchema.fieldNames.toSeq.map(colName => {
@@ -927,7 +927,7 @@ case class Scd2BatchProcessor(
Scd2BatchProcessor.computeTrackedHistoryColumns(
schema = df.schema,
changeArgs = changeArgs,
- caseSensitive = df.sparkSession.sessionState.conf.caseSensitiveAnalysis
+ resolver = df.sparkSession.sessionState.conf.resolver
)
/**
@@ -1413,8 +1413,7 @@ object Scd2BatchProcessor {
private[pipelines] def computeTrackedHistoryColumns(
schema: StructType,
changeArgs: ChangeArgs,
- caseSensitive: Boolean): Seq[String] = {
- val resolver = if (caseSensitive) caseSensitiveResolution else
caseInsensitiveResolution
+ resolver: Resolver): Seq[String] = {
val keyColNames = changeArgs.keys.map(_.name)
val eligibleSchema = StructType(schema.fields.filterNot { field =>
@@ -1427,7 +1426,7 @@ object Scd2BatchProcessor {
schemaName = "trackHistorySelection",
schema = eligibleSchema,
columnSelection = changeArgs.trackHistorySelection,
- caseSensitive = caseSensitive
+ resolver = resolver
)
.fieldNames
.toImmutableArraySeq
diff --git
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
index bb996d527cdc..54525aab8712 100644
---
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
+++
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
@@ -263,7 +263,7 @@ class AutoCdcMergeFlow(
schemaName = "changeDataFeed",
schema = df.schema,
columnSelection = changeArgs.columnSelection,
- caseSensitive = spark.sessionState.conf.caseSensitiveAnalysis
+ resolver = spark.sessionState.conf.resolver
)
// AutoCDC flows require all key columns to be present in the
user-selected source schema,
// so that they survive into the target table where SCD reconciliation
needs them.
@@ -390,9 +390,7 @@ class AutoCdcMergeFlow(
throw new AnalysisException(
errorClass = "AUTOCDC_RESERVED_COLUMN_NAME_PREFIX_CONFLICT",
messageParameters = Map(
- "caseSensitivity" -> CaseSensitivityLabels.of(
- spark.sessionState.conf.caseSensitiveAnalysis
- ),
+ "caseSensitivity" -> CaseSensitivityLabels.of(resolver),
"columnName" -> conflictingColumnName,
"schemaName" -> "changeDataFeed",
"reservedColumnNamePrefix" -> reservedPrefix
@@ -428,9 +426,7 @@ class AutoCdcMergeFlow(
throw new AnalysisException(
errorClass = "AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT",
messageParameters = Map(
- "caseSensitivity" -> CaseSensitivityLabels.of(
- spark.sessionState.conf.caseSensitiveAnalysis
- ),
+ "caseSensitivity" -> CaseSensitivityLabels.of(resolver),
"columnName" -> conflictingColumnName,
"schemaName" -> "changeDataFeed",
"scdType" -> changeArgs.storedAsScdType.label,
@@ -452,9 +448,7 @@ class AutoCdcMergeFlow(
throw new AnalysisException(
errorClass = "AUTOCDC_KEY_NOT_IN_SELECTED_SCHEMA",
messageParameters = Map(
- "caseSensitivity" -> CaseSensitivityLabels.of(
- spark.sessionState.conf.caseSensitiveAnalysis
- ),
+ "caseSensitivity" -> CaseSensitivityLabels.of(resolver),
"keyColumnName" -> missingKey.name
)
)
@@ -478,7 +472,7 @@ class AutoCdcMergeFlow(
Scd2BatchProcessor.computeTrackedHistoryColumns(
schema = selectedSchema,
changeArgs = changeArgs,
- caseSensitive = spark.sessionState.conf.caseSensitiveAnalysis
+ resolver = spark.sessionState.conf.resolver
)
}
}
diff --git
a/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgsSuite.scala
b/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgsSuite.scala
index bf6a89652353..7deda6fb0d26 100644
---
a/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgsSuite.scala
+++
b/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgsSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.pipelines.autocdc
import org.apache.spark.{SparkException, SparkFunSuite}
import org.apache.spark.sql.{functions => F, AnalysisException, Row}
+import org.apache.spark.sql.catalyst.analysis.{caseInsensitiveResolution,
caseSensitiveResolution}
import org.apache.spark.sql.catalyst.parser.ParseException
import org.apache.spark.sql.test.SharedSparkSession
import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
@@ -36,7 +37,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
schemaName = "test",
schema = sourceSchema,
columnSelection = None,
- caseSensitive = true
+ resolver = caseSensitiveResolution
) == sourceSchema)
}
@@ -48,7 +49,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
schemaName = "test",
schema = sourceSchema,
columnSelection = Some(ColumnSelection.IncludeColumns(Seq.empty)),
- caseSensitive = true
+ resolver = caseSensitiveResolution
) == new StructType())
}
@@ -60,7 +61,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
schemaName = "test",
schema = sourceSchema,
columnSelection = Some(ColumnSelection.ExcludeColumns(Seq.empty)),
- caseSensitive = true
+ resolver = caseSensitiveResolution
) == sourceSchema)
}
@@ -73,7 +74,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
Seq(UnqualifiedColumnName("age"), UnqualifiedColumnName("Name"))
)
),
- caseSensitive = true
+ resolver = caseSensitiveResolution
)
assert(filteredSchema == new StructType()
@@ -88,7 +89,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
columnSelection = Some(
ColumnSelection.ExcludeColumns(Seq(UnqualifiedColumnName("id")))
),
- caseSensitive = true
+ resolver = caseSensitiveResolution
)
assert(filteredSchema == new StructType()
@@ -108,7 +109,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
Seq(UnqualifiedColumnName("name"),
UnqualifiedColumnName("missing"))
)
),
- caseSensitive = true
+ resolver = caseSensitiveResolution
)
},
condition = "AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA",
@@ -134,7 +135,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
Seq(UnqualifiedColumnName("NAME"),
UnqualifiedColumnName("missing"))
)
),
- caseSensitive = true
+ resolver = caseSensitiveResolution
)
},
condition = "AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA",
@@ -159,7 +160,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
Seq(UnqualifiedColumnName("AGE"), UnqualifiedColumnName("NAME"))
)
),
- caseSensitive = false
+ resolver = caseInsensitiveResolution
)
// The retained fields keep their original casing from the schema, not the
user's input.
@@ -180,7 +181,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
Seq(UnqualifiedColumnName("name"), UnqualifiedColumnName("NAME"))
)
),
- caseSensitive = false
+ resolver = caseInsensitiveResolution
)
assert(filteredSchema == new StructType().add("Name", StringType))
@@ -193,7 +194,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
columnSelection = Some(
ColumnSelection.ExcludeColumns(Seq(UnqualifiedColumnName("name")))
),
- caseSensitive = false
+ resolver = caseInsensitiveResolution
)
assert(filteredSchema == new StructType()
@@ -215,7 +216,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
Seq(UnqualifiedColumnName("NAME"),
UnqualifiedColumnName("Missing"))
)
),
- caseSensitive = false
+ resolver = caseInsensitiveResolution
)
},
condition = "AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA",
@@ -301,7 +302,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
columnSelection = Some(
ColumnSelection.IncludeColumns(Seq(UnqualifiedColumnName("`a.b`")))
),
- caseSensitive = true
+ resolver = caseSensitiveResolution
)
assert(filteredSchema == new StructType().add("a.b", IntegerType))
@@ -314,7 +315,7 @@ class ChangeArgsSuite extends SparkFunSuite with
SharedSparkSession {
columnSelection = Some(
ColumnSelection.IncludeColumns(Seq(UnqualifiedColumnName("`Name`")))
),
- caseSensitive = true
+ resolver = caseSensitiveResolution
)
assert(filteredSchema == new StructType().add("Name", StringType))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]