This is an automated email from the ASF dual-hosted git repository.

szehon-ho pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new cddeec4151e9 [SPARK-58347][SDP] Thread conf.resolver through 
ColumnSelection.applyToSchema
cddeec4151e9 is described below

commit cddeec4151e9d3474a83c835919a6f259ddf0ccc
Author: Andreas Neumann <[email protected]>
AuthorDate: Wed Jul 29 11:44:46 2026 -0700

    [SPARK-58347][SDP] Thread conf.resolver through 
ColumnSelection.applyToSchema
    
    ### What changes were proposed in this pull request?
    
    This is a follow-up refactor for SDP AutoCDC. It reworks 
`ColumnSelection.applyToSchema` (and its private helper `lookupFieldIndices`) 
to accept a `Resolver` instead of a `caseSensitive: Boolean`:
      - `ColumnSelection.applyToSchema` / `lookupFieldIndices` now take a 
`Resolver`. `lookupFieldIndices` resolves each requested column via 
`schema.fieldNames.indexWhere(resolver(_, name))` rather than switching between 
`StructType.getFieldIndex` / `getFieldIndexCaseInsensitive` on a boolean.
      - `Scd2BatchProcessor.computeTrackedHistoryColumns` now takes a 
`Resolver` directly instead of deriving `caseSensitiveResolution` / 
`caseInsensitiveResolution` from a boolean.
      - All callers pass `spark.sessionState.conf.resolver` instead of 
`conf.caseSensitiveAnalysis`: 
`Scd1BatchProcessor.projectTargetColumnsOntoMicrobatch` (2 sites), 
`Scd2BatchProcessor.projectTargetColumnsOntoMicrobatch` (2 sites) and its 
`computeTrackedHistoryColumns` instance method, and `AutoCdcMergeFlow` (the 
`userSelectedSchema` projection and the construction-time track-history 
validation).
      - The `caseSensitivity` message parameter of 
`AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA` is preserved via a new 
`CaseSensitivityLabels.of(resolver)` overload that classifies the resolver by 
probing it (`!resolver("a", "A")`), so error messages are byte-for-byte 
unchanged.
    
    ### Why are the changes needed?
    
    The AutoCDC code already resolves identifiers everywhere else through 
`conf.resolver` (the canonical Spark abstraction for case-aware identifier 
comparison). `applyToSchema` was the odd one out, threading a raw 
`caseSensitive` boolean and re-deriving a resolver at each layer. Passing the 
`Resolver` directly makes column matching consistent with the rest of the 
pipeline, removes the boolean-to-resolver round-trips, and lets 
`computeTrackedHistoryColumns` take the resolver it actually n [...]
    
    ### Does this PR introduce _any_ user-facing change?
    No. This is a pure refactor with no behavior change; the `caseSensitivity` 
label in the `AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA` error is preserved.
    
    ### How was this patch tested?
    
    Existing tests, updated to pass resolvers instead of booleans:
      - `ChangeArgsSuite` and `AutoCdcFlowSuite` (65 tests) — including the 
case-sensitive/insensitive selection and missing-column error-message cases.
      - `Scd2BatchProcessor*` and `Scd1BatchProcessorMergeSuite` (150 tests) — 
exercising the microbatch projection and history-tracking paths under both 
case-sensitivity settings.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    Generated-by: Claude Code (Opus 4.8)
    
    Closes #57610 from anew/spark-58347-resolver-columnselection.
    
    Authored-by: Andreas Neumann <[email protected]>
    Signed-off-by: Szehon Ho <[email protected]>
    (cherry picked from commit 8df89f20fc5c800ef4935c1b49a7525ec9df8921)
    Signed-off-by: Szehon Ho <[email protected]>
---
 .../spark/sql/pipelines/autocdc/ChangeArgs.scala   | 46 +++++++++++++++-------
 .../sql/pipelines/autocdc/Scd1BatchProcessor.scala |  7 ++--
 .../sql/pipelines/autocdc/Scd2BatchProcessor.scala | 15 ++++---
 .../apache/spark/sql/pipelines/graph/Flow.scala    | 16 +++-----
 .../sql/pipelines/autocdc/ChangeArgsSuite.scala    | 27 +++++++------
 5 files changed, 60 insertions(+), 51 deletions(-)

diff --git 
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgs.scala
 
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgs.scala
index 15aa535994f9..a8c3485da31c 100644
--- 
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgs.scala
+++ 
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgs.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.pipelines.autocdc
 
 import org.apache.spark.SparkException
 import org.apache.spark.sql.{AnalysisException, Column}
+import org.apache.spark.sql.catalyst.analysis.{caseInsensitiveResolution, 
caseSensitiveResolution, Resolver}
 import org.apache.spark.sql.catalyst.parser.CatalystSqlParser
 import org.apache.spark.sql.catalyst.util.QuotingUtils
 import org.apache.spark.sql.types.StructType
@@ -72,26 +73,26 @@ object ColumnSelection {
    * @param schema          The schema to filter.
    * @param columnSelection The user-provided selection. `None` is a no-op and 
returns `schema`
    *                        unchanged.
-   * @param caseSensitive   Whether to match column names case-sensitively 
against the schema.
-   *                        Callers should derive this from the session, e.g.
-   *                        `session.sessionState.conf.caseSensitiveAnalysis`, 
so column matching
-   *                        stays consistent with `spark.sql.caseSensitive`.
+   * @param resolver        Determines whether two column names are considered 
equal. Callers
+   *                        should pass the session resolver, e.g.
+   *                        `session.sessionState.conf.resolver`, so column 
matching stays
+   *                        consistent with `spark.sql.caseSensitive`.
    */
   def applyToSchema(
       schemaName: String,
       schema: StructType,
       columnSelection: Option[ColumnSelection],
-      caseSensitive: Boolean): StructType = columnSelection match {
+      resolver: Resolver): StructType = columnSelection match {
     case None =>
       // A None column selection is interpreted as a no-op.
       schema
     case Some(IncludeColumns(cols)) =>
-      val keepIndices = lookupFieldIndices(schemaName, schema, cols, 
caseSensitive)
+      val keepIndices = lookupFieldIndices(schemaName, schema, cols, resolver)
       StructType(schema.fields.zipWithIndex.collect {
         case (field, idx) if keepIndices.contains(idx) => field
       })
     case Some(ExcludeColumns(cols)) =>
-      val dropIndices = lookupFieldIndices(schemaName, schema, cols, 
caseSensitive)
+      val dropIndices = lookupFieldIndices(schemaName, schema, cols, resolver)
       StructType(schema.fields.zipWithIndex.collect {
         case (field, idx) if !dropIndices.contains(idx) => field
       })
@@ -101,17 +102,20 @@ object ColumnSelection {
       schemaName: String,
       schema: StructType,
       fields: Seq[UnqualifiedColumnName],
-      caseSensitive: Boolean): Set[Int] = {
-    val caseAwareGetFieldIndex: String => Option[Int] =
-      if (caseSensitive) schema.getFieldIndex else 
schema.getFieldIndexCaseInsensitive
-
-    val fieldIndexResolutions = fields.map(f => f -> 
caseAwareGetFieldIndex(f.name))
+      resolver: Resolver): Set[Int] = {
+    def resolveFieldIndex(name: String): Option[Int] =
+      schema.fieldNames.indexWhere(resolver(_, name)) match {
+        case -1 => None
+        case idx => Some(idx)
+      }
+
+    val fieldIndexResolutions = fields.map(f => f -> resolveFieldIndex(f.name))
     val missingFieldNames = fieldIndexResolutions.collect { case (f, None) => 
f.name }.distinct
     if (missingFieldNames.nonEmpty) {
       throw new AnalysisException(
         errorClass = "AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA",
         messageParameters = Map(
-          "caseSensitivity" -> CaseSensitivityLabels.of(caseSensitive),
+          "caseSensitivity" -> CaseSensitivityLabels.of(resolver),
           "schemaName" -> schemaName,
           "missingColumns" -> missingFieldNames.mkString(", "),
           "availableColumns" -> schema.fieldNames.mkString(", ")
@@ -127,8 +131,20 @@ private[pipelines] object CaseSensitivityLabels {
   val CaseSensitive: String = "case-sensitive"
   val CaseInsensitive: String = "case-insensitive"
 
-  def of(caseSensitive: Boolean): String =
-    if (caseSensitive) CaseSensitive else CaseInsensitive
+  /**
+   * Maps a [[Resolver]] to its user-facing label. Classifies by reference 
identity against the
+   * two session resolver singletons, matching 
`SchemaUtils.isCaseSensitiveAnalysis`.
+   */
+  def of(resolver: Resolver): String = {
+    if (resolver == caseSensitiveResolution) {
+      CaseSensitive
+    } else if (resolver == caseInsensitiveResolution) {
+      CaseInsensitive
+    } else {
+      // this should be unreachable because `conf.resolver` only ever returns 
one of these two.
+      throw SparkException.internalError(s"Unknown resolver: $resolver")
+    }
+  }
 }
 
 /** The SCD (Slowly Changing Dimension) strategy for a CDC flow. */
diff --git 
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1BatchProcessor.scala
 
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1BatchProcessor.scala
index 35006dc4ee21..397775513703 100644
--- 
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1BatchProcessor.scala
+++ 
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd1BatchProcessor.scala
@@ -163,8 +163,7 @@ case class Scd1BatchProcessor(
    */
   private[autocdc] def projectTargetColumnsOntoMicrobatch(
       microbatchWithCdcMetadataDf: DataFrame): DataFrame = {
-    val caseSensitiveColumnComparison =
-      
microbatchWithCdcMetadataDf.sparkSession.sessionState.conf.caseSensitiveAnalysis
+    val resolver = 
microbatchWithCdcMetadataDf.sparkSession.sessionState.conf.resolver
 
     // The user schema is the microbatch schema after dropping the system CDC 
metadata column.
     // We project out the system column before applying user selection and 
project it back in
@@ -178,7 +177,7 @@ case class Scd1BatchProcessor(
           Seq(UnqualifiedColumnName(AutoCdcReservedNames.cdcMetadataColName))
         )
       ),
-      caseSensitive = caseSensitiveColumnComparison
+      resolver = resolver
     )
 
     val userSelectedColumnsInMicrobatchSchema =
@@ -186,7 +185,7 @@ case class Scd1BatchProcessor(
         schemaName = "microbatch",
         schema = userColumnsInMicrobatchSchema,
         columnSelection = changeArgs.columnSelection,
-        caseSensitive = caseSensitiveColumnComparison
+        resolver = resolver
       )
 
     // In addition to the explicit user-selected columns, re-project the 
operational CDC metadata
diff --git 
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
 
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
index 125f6c895f41..c93460c2ad81 100644
--- 
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
+++ 
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/autocdc/Scd2BatchProcessor.scala
@@ -21,7 +21,7 @@ import org.apache.spark.SparkException
 import org.apache.spark.sql.{functions => F}
 import org.apache.spark.sql.Column
 import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.analysis.{caseInsensitiveResolution, 
caseSensitiveResolution}
+import org.apache.spark.sql.catalyst.analysis.Resolver
 import org.apache.spark.sql.catalyst.expressions.{CreateMap, If, Literal, 
RaiseError}
 import org.apache.spark.sql.catalyst.util.QuotingUtils
 import org.apache.spark.sql.classic.{DataFrame, ExpressionUtils}
@@ -199,7 +199,7 @@ case class Scd2BatchProcessor(
   private def projectTargetColumnsOntoMicrobatch(
       microbatch: DataFrame
   ): DataFrame = {
-    val caseSensitive = 
microbatch.sparkSession.sessionState.conf.caseSensitiveAnalysis
+    val resolver = microbatch.sparkSession.sessionState.conf.resolver
 
     // Strip the framework columns through the same case-aware path as the 
user selection, for
     // consistency with Scd1BatchProcessor.projectTargetColumnsOntoMicrobatch.
@@ -211,14 +211,14 @@ case class Scd2BatchProcessor(
           
Scd2BatchProcessor.reservedFrameworkColNames.toSeq.map(UnqualifiedColumnName(_))
         )
       ),
-      caseSensitive = caseSensitive
+      resolver = resolver
     )
     val userSelectedDataSchema =
       ColumnSelection.applyToSchema(
         schemaName = "microbatch",
         schema = dataSchema,
         columnSelection = changeArgs.columnSelection,
-        caseSensitive = caseSensitive
+        resolver = resolver
       )
     val finalColumnsToSelect: Seq[Column] =
       userSelectedDataSchema.fieldNames.toSeq.map(colName => {
@@ -927,7 +927,7 @@ case class Scd2BatchProcessor(
     Scd2BatchProcessor.computeTrackedHistoryColumns(
       schema = df.schema,
       changeArgs = changeArgs,
-      caseSensitive = df.sparkSession.sessionState.conf.caseSensitiveAnalysis
+      resolver = df.sparkSession.sessionState.conf.resolver
     )
 
   /**
@@ -1413,8 +1413,7 @@ object Scd2BatchProcessor {
   private[pipelines] def computeTrackedHistoryColumns(
       schema: StructType,
       changeArgs: ChangeArgs,
-      caseSensitive: Boolean): Seq[String] = {
-    val resolver = if (caseSensitive) caseSensitiveResolution else 
caseInsensitiveResolution
+      resolver: Resolver): Seq[String] = {
     val keyColNames = changeArgs.keys.map(_.name)
 
     val eligibleSchema = StructType(schema.fields.filterNot { field =>
@@ -1427,7 +1426,7 @@ object Scd2BatchProcessor {
         schemaName = "trackHistorySelection",
         schema = eligibleSchema,
         columnSelection = changeArgs.trackHistorySelection,
-        caseSensitive = caseSensitive
+        resolver = resolver
       )
       .fieldNames
       .toImmutableArraySeq
diff --git 
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala 
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
index bb996d527cdc..54525aab8712 100644
--- 
a/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
+++ 
b/sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/Flow.scala
@@ -263,7 +263,7 @@ class AutoCdcMergeFlow(
       schemaName = "changeDataFeed",
       schema = df.schema,
       columnSelection = changeArgs.columnSelection,
-      caseSensitive = spark.sessionState.conf.caseSensitiveAnalysis
+      resolver = spark.sessionState.conf.resolver
     )
     // AutoCDC flows require all key columns to be present in the 
user-selected source schema,
     // so that they survive into the target table where SCD reconciliation 
needs them.
@@ -390,9 +390,7 @@ class AutoCdcMergeFlow(
       throw new AnalysisException(
         errorClass = "AUTOCDC_RESERVED_COLUMN_NAME_PREFIX_CONFLICT",
         messageParameters = Map(
-          "caseSensitivity" -> CaseSensitivityLabels.of(
-            spark.sessionState.conf.caseSensitiveAnalysis
-          ),
+          "caseSensitivity" -> CaseSensitivityLabels.of(resolver),
           "columnName" -> conflictingColumnName,
           "schemaName" -> "changeDataFeed",
           "reservedColumnNamePrefix" -> reservedPrefix
@@ -428,9 +426,7 @@ class AutoCdcMergeFlow(
         throw new AnalysisException(
           errorClass = "AUTOCDC_RESERVED_COLUMN_NAME_CONFLICT",
           messageParameters = Map(
-            "caseSensitivity" -> CaseSensitivityLabels.of(
-              spark.sessionState.conf.caseSensitiveAnalysis
-            ),
+            "caseSensitivity" -> CaseSensitivityLabels.of(resolver),
             "columnName" -> conflictingColumnName,
             "schemaName" -> "changeDataFeed",
             "scdType" -> changeArgs.storedAsScdType.label,
@@ -452,9 +448,7 @@ class AutoCdcMergeFlow(
         throw new AnalysisException(
           errorClass = "AUTOCDC_KEY_NOT_IN_SELECTED_SCHEMA",
           messageParameters = Map(
-            "caseSensitivity" -> CaseSensitivityLabels.of(
-              spark.sessionState.conf.caseSensitiveAnalysis
-            ),
+            "caseSensitivity" -> CaseSensitivityLabels.of(resolver),
             "keyColumnName" -> missingKey.name
           )
         )
@@ -478,7 +472,7 @@ class AutoCdcMergeFlow(
       Scd2BatchProcessor.computeTrackedHistoryColumns(
         schema = selectedSchema,
         changeArgs = changeArgs,
-        caseSensitive = spark.sessionState.conf.caseSensitiveAnalysis
+        resolver = spark.sessionState.conf.resolver
       )
     }
   }
diff --git 
a/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgsSuite.scala
 
b/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgsSuite.scala
index bf6a89652353..7deda6fb0d26 100644
--- 
a/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgsSuite.scala
+++ 
b/sql/pipelines/src/test/scala/org/apache/spark/sql/pipelines/autocdc/ChangeArgsSuite.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.pipelines.autocdc
 
 import org.apache.spark.{SparkException, SparkFunSuite}
 import org.apache.spark.sql.{functions => F, AnalysisException, Row}
+import org.apache.spark.sql.catalyst.analysis.{caseInsensitiveResolution, 
caseSensitiveResolution}
 import org.apache.spark.sql.catalyst.parser.ParseException
 import org.apache.spark.sql.test.SharedSparkSession
 import org.apache.spark.sql.types.{IntegerType, StringType, StructType}
@@ -36,7 +37,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
         schemaName = "test",
         schema = sourceSchema,
         columnSelection = None,
-        caseSensitive = true
+        resolver = caseSensitiveResolution
       ) == sourceSchema)
   }
 
@@ -48,7 +49,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
         schemaName = "test",
         schema = sourceSchema,
         columnSelection = Some(ColumnSelection.IncludeColumns(Seq.empty)),
-        caseSensitive = true
+        resolver = caseSensitiveResolution
       ) == new StructType())
   }
 
@@ -60,7 +61,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
         schemaName = "test",
         schema = sourceSchema,
         columnSelection = Some(ColumnSelection.ExcludeColumns(Seq.empty)),
-        caseSensitive = true
+        resolver = caseSensitiveResolution
       ) == sourceSchema)
   }
 
@@ -73,7 +74,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
           Seq(UnqualifiedColumnName("age"), UnqualifiedColumnName("Name"))
         )
       ),
-      caseSensitive = true
+      resolver = caseSensitiveResolution
     )
 
     assert(filteredSchema == new StructType()
@@ -88,7 +89,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
       columnSelection = Some(
         ColumnSelection.ExcludeColumns(Seq(UnqualifiedColumnName("id")))
       ),
-      caseSensitive = true
+      resolver = caseSensitiveResolution
     )
 
     assert(filteredSchema == new StructType()
@@ -108,7 +109,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
               Seq(UnqualifiedColumnName("name"), 
UnqualifiedColumnName("missing"))
             )
           ),
-          caseSensitive = true
+          resolver = caseSensitiveResolution
         )
       },
       condition = "AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA",
@@ -134,7 +135,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
               Seq(UnqualifiedColumnName("NAME"), 
UnqualifiedColumnName("missing"))
             )
           ),
-          caseSensitive = true
+          resolver = caseSensitiveResolution
         )
       },
       condition = "AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA",
@@ -159,7 +160,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
           Seq(UnqualifiedColumnName("AGE"), UnqualifiedColumnName("NAME"))
         )
       ),
-      caseSensitive = false
+      resolver = caseInsensitiveResolution
     )
 
     // The retained fields keep their original casing from the schema, not the 
user's input.
@@ -180,7 +181,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
           Seq(UnqualifiedColumnName("name"), UnqualifiedColumnName("NAME"))
         )
       ),
-      caseSensitive = false
+      resolver = caseInsensitiveResolution
     )
 
     assert(filteredSchema == new StructType().add("Name", StringType))
@@ -193,7 +194,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
       columnSelection = Some(
         ColumnSelection.ExcludeColumns(Seq(UnqualifiedColumnName("name")))
       ),
-      caseSensitive = false
+      resolver = caseInsensitiveResolution
     )
 
     assert(filteredSchema == new StructType()
@@ -215,7 +216,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
               Seq(UnqualifiedColumnName("NAME"), 
UnqualifiedColumnName("Missing"))
             )
           ),
-          caseSensitive = false
+          resolver = caseInsensitiveResolution
         )
       },
       condition = "AUTOCDC_COLUMNS_NOT_FOUND_IN_SCHEMA",
@@ -301,7 +302,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
       columnSelection = Some(
         ColumnSelection.IncludeColumns(Seq(UnqualifiedColumnName("`a.b`")))
       ),
-      caseSensitive = true
+      resolver = caseSensitiveResolution
     )
 
     assert(filteredSchema == new StructType().add("a.b", IntegerType))
@@ -314,7 +315,7 @@ class ChangeArgsSuite extends SparkFunSuite with 
SharedSparkSession {
       columnSelection = Some(
         ColumnSelection.IncludeColumns(Seq(UnqualifiedColumnName("`Name`")))
       ),
-      caseSensitive = true
+      resolver = caseSensitiveResolution
     )
 
     assert(filteredSchema == new StructType().add("Name", StringType))


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to