This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new ef74401550fc [SPARK-60979][SQL] Fix wrong results by nested schema 
pruning when column names differ only by case
ef74401550fc is described below

commit ef74401550fc5b1ea3fbf887a89e7c65fee2307d
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 08:08:21 2026 -0800

    [SPARK-60979][SQL] Fix wrong results by nested schema pruning when column 
names differ only by case
    
    ### What changes were proposed in this pull request?
    
    This PR fixes a correctness issue in `SchemaPruning.pruneSchema`. When the 
schema has fields whose names differ only by case (for example `salary` and 
`SALARY`), the pruned schema could take the type and nullability of the wrong 
field in a case-insensitive session (the default).
    
    `pruneSchema` now matches by exact name first, and falls back to the 
resolver only when no field of the schema has the exact name of the requested 
field:
    
    - `pruneSchema`: replaced the first resolver match (`mergedSchema.find(m => 
resolver(m.name, d.name))`) with an exact-name lookup, then a resolver lookup 
that skips requested fields whose exact name exists in the schema.
    - `sortLeftFieldsByRight`: the lowercase-keyed `leftStructHashMap` keeps 
only the last field for a folded name. It now looks up the exact name first and 
uses the folded name only when no other field of the right struct has the exact 
name of the matched field.
    
    Case-insensitive matching of differently cased names (SPARK-35096) still 
works when there is no exact match.
    
    ### Why are the changes needed?
    
    This is a correctness issue. For `salary INT NOT NULL, SALARY STRING`, a 
request for `SALARY` produces a pruned schema where `salary` becomes `STRING` 
and nullable:
    
    ```
    before: pk INT NOT NULL, salary STRING, SALARY STRING
    after:  pk INT NOT NULL, salary INT NOT NULL, SALARY STRING
    ```
    
    A DataSource V2 scan (`PushDownUtils.pruneColumns`) is then asked for the 
wrong type and nullability. With a table whose columns differ only by case, 
DSv2 row-level operations were measured to store `0` instead of `NULL`, or to 
fail with `PLAN_VALIDATION_FAILED_RULE_IN_BATCH` or a Janino 
`CompileException`. These results were reproduced with the row-level ordinal 
lookup (`RewriteRowLevelCommand.findColOrdinal`) fixed separately, and 
`spark.sql.optimizer.nestedSchemaPruning.enabled=fal [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes, in the narrow case where a table or schema has columns that differ 
only by case and the session is case-insensitive. The data source is now asked 
for the correct type and nullability of each column instead of those of another 
column. Nothing changes when no two names collide under case folding.
    
    ### How was this patch tested?
    
    Added a unit test to the catalyst `SchemaPruningSuite`. It fails before the 
change and passes after it.
    
    Ran the following suites on this change, all passing:
    
    ```
    build/sbt 'catalyst/testOnly *expressions.SchemaPruningSuite 
*optimizer.NestedColumnAliasingSuite'
    build/sbt 'sql/testOnly *ParquetV2SchemaPruningSuite 
*OrcV2SchemaPruningSuite *ParquetSchemaPruningSuite *OrcV1SchemaPruningSuite 
*FileMetadataStructSuite *FileSourceV2PlanMergingSuite'
    build/sbt 'sql/testOnly *DataSourceV2Suite 
*V2ScanRelationPushDownSubquerySuite'
    ```
    
    Also verified DSv2 UPDATE, DELETE and MERGE on group-based, delta, and 
delta split-update tables with `salary INT, SALARY STRING` and NULL values in 
nullable and `NOT NULL` colliding columns, together with the separate ordinal 
lookup fix, and all results were correct.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60156 from dongjoon-hyun/SPARK-60979.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit a1dc1d317aaaac28424880c2740d5fea3c0521e3)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../sql/catalyst/expressions/SchemaPruning.scala   | 29 ++++++++++++++++++----
 .../spark/sql/catalyst/optimizer/objects.scala     | 24 ++++++++++++------
 .../catalyst/expressions/SchemaPruningSuite.scala  | 16 ++++++++++++
 .../optimizer/ObjectSerializerPruningSuite.scala   | 11 ++++++++
 .../spark/sql/DatasetOptimizationSuite.scala       | 25 ++++++++++++++++++-
 .../spark/sql/connector/UpdateTableSuiteBase.scala | 19 ++++++++++++++
 6 files changed, 111 insertions(+), 13 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
index 59ff86071647..c4957f2329e0 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
@@ -44,13 +44,25 @@ object SchemaPruning extends SQLConfHelper {
     val mergedSchema = requestedRootFields
       .map { root: RootField => StructType(Array(root.field)) }
       .reduceLeft(_ merge _)
-    val mergedDataSchema =
-      StructType(schema.map(d => mergedSchema.find(m => resolver(m.name, 
d.name)).getOrElse(d)))
+    // Match by the exact name first. Fall back to the resolver only for a 
requested field that
+    // no field of the schema has the exact name of, as the schema may have 
fields whose names
+    // differ only by case.
+    val mergedByName = firstByName(mergedSchema)
+    val schemaNames = schema.fieldNames.toSet
+    val mergedDataSchema = StructType(schema.map { d =>
+      mergedByName.get(d.name).orElse {
+        mergedSchema.find(m => resolver(m.name, d.name) && 
!schemaNames.contains(m.name))
+      }.getOrElse(d)
+    })
     // Sort the fields of mergedDataSchema according to their order in 
dataSchema,
     // recursively. This makes mergedDataSchema a pruned schema of dataSchema
     sortLeftFieldsByRight(mergedDataSchema, schema).asInstanceOf[StructType]
   }
 
+  // Maps each exact field name to the first field with that name.
+  private def firstByName(struct: StructType): Map[String, StructField] =
+    struct.fields.reverseIterator.map(f => f.name -> f).toMap
+
   /**
    * Sorts the fields and descendant fields of structs in left according to 
their order in
    * right. This function assumes that the fields of left are a subset of the 
fields of
@@ -76,10 +88,17 @@ object SchemaPruning extends SQLConfHelper {
 
         val leftStructHashMap =
           HashMap(leftStruct.map(f => 
formatFieldName(f.name)).zip(leftStruct): _*)
+        val leftStructByName = firstByName(leftStruct)
+        val rightFieldNames = rightStruct.fieldNames.toSet
         val sortedLeftFields = rightStruct.fieldNames.flatMap { fieldName =>
-          val formattedFieldName = formatFieldName(fieldName)
-          if (leftStructHashMap.contains(formattedFieldName)) {
-            val resolvedLeftStruct = leftStructHashMap(formattedFieldName)
+          // Prefer the exact name. A field of left whose name is the exact 
name of another field
+          // of right is not a match for this one.
+          val leftFieldOpt = leftStructByName.get(fieldName).orElse {
+            leftStructHashMap.get(formatFieldName(fieldName))
+              .filterNot(f => rightFieldNames.contains(f.name))
+          }
+          if (leftFieldOpt.isDefined) {
+            val resolvedLeftStruct = leftFieldOpt.get
             val leftFieldType = resolvedLeftStruct.dataType
             val rightFieldType = rightStruct(fieldName).dataType
             val sortedLeftFieldType = sortLeftFieldsByRight(leftFieldType, 
rightFieldType)
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
index 163b6d9ff782..a6fe22424624 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
@@ -28,6 +28,7 @@ import org.apache.spark.sql.catalyst.rules._
 import org.apache.spark.sql.catalyst.trees.TreePattern._
 import org.apache.spark.sql.catalyst.types.DataTypeUtils
 import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType, 
UserDefinedType}
+import org.apache.spark.sql.util.SchemaUtils
 
 /*
  * This file defines optimization rules related to object manipulation (for 
the Dataset API).
@@ -151,11 +152,11 @@ object ObjectSerializerPruning extends Rule[LogicalPlan] {
    * and a pruned `StructType`.
    */
   private def pruneNamedStruct(struct: CreateNamedStruct, prunedType: 
StructType) = {
-    // Filters out the pruned fields.
-    val resolver = conf.resolver
+    // Filters out the pruned fields. The pruned type names its fields after 
the fields of the
+    // serializer, so match by the exact name as the names may differ only by 
case.
+    val prunedFieldNames = prunedType.fieldNames.toSet
     val prunedFields = struct.nameExprs.zip(struct.valExprs).filter { case 
(nameExpr, _) =>
-      val name = nameExpr.eval(EmptyRow).toString
-      prunedType.fieldNames.exists(resolver(_, name))
+      prunedFieldNames.contains(nameExpr.eval(EmptyRow).toString)
     }.flatMap(pair => Seq(pair._1, pair._2))
 
     CreateNamedStruct(prunedFields)
@@ -221,7 +222,14 @@ object ObjectSerializerPruning extends Rule[LogicalPlan] {
       val usedRefs = p.references
       val prunedSerializer = s.serializer.filter(usedRefs.contains)
 
-      val rootFields = SchemaPruning.identifyRootFields(p.projectList, 
Seq.empty)
+      // Name the references after the serializer output. The analyzer keeps 
the case that the
+      // query used, and pruneSchema and ProjectionOverSchema match names 
exactly.
+      val nameById = s.output.map(a => a.exprId -> a.name).toMap
+      val normalizedProjects = p.projectList.map(_.transform {
+        case a: AttributeReference if nameById.contains(a.exprId) =>
+          a.withName(nameById(a.exprId))
+      }.asInstanceOf[NamedExpression])
+      val rootFields = SchemaPruning.identifyRootFields(normalizedProjects, 
Seq.empty)
 
       if (conf.serializerNestedSchemaPruningEnabled && rootFields.nonEmpty) {
         // Prunes nested fields in serializers.
@@ -233,10 +241,12 @@ object ObjectSerializerPruning extends Rule[LogicalPlan] {
 
         // Builds new projection.
         val projectionOverSchema = ProjectionOverSchema(prunedSchema, 
AttributeSet(s.output))
-        val newProjects = p.projectList.map(_.transformDown {
+        val newProjects = normalizedProjects.map(_.transformDown {
           case projectionOverSchema(expr) => expr
         }).map { case expr: NamedExpression => expr }
-        p.copy(projectList = newProjects,
+        p.copy(
+          projectList =
+            SchemaUtils.restoreOriginalOutputNames(newProjects, 
p.projectList.map(_.name)),
           child = SerializeFromObject(nestedPrunedSerializer, s.child))
       } else {
         p.copy(child = SerializeFromObject(prunedSerializer, s.child))
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruningSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruningSuite.scala
index 8fae15316fc9..2c50d31a07bb 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruningSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruningSuite.scala
@@ -134,6 +134,22 @@ class SchemaPruningSuite extends SparkFunSuite with 
SQLHelper {
     }
   }
 
+  test("SPARK-60979: prefer the exact name when field names differ only by 
case") {
+    withSQLConf(CASE_SENSITIVE.key -> "false") {
+      // top-level fields
+      testPrunedSchema(
+        StructType.fromDDL("pk INT NOT NULL, salary INT NOT NULL, SALARY 
STRING"),
+        Seq(StructField("SALARY", StringType)),
+        StructType.fromDDL("pk INT NOT NULL, salary INT NOT NULL, SALARY 
STRING"))
+
+      // nested fields
+      testPrunedSchema(
+        StructType.fromDDL("s struct<a: INT NOT NULL, A: STRING, b: INT>"),
+        Seq(StructField("s", StructType.fromDDL("A STRING"))),
+        StructType.fromDDL("s struct<A: STRING>"))
+    }
+  }
+
   test("SPARK-35232: getRootFields/pruneDataSchema should retain attribute 
metadata") {
     val metadata = new MetadataBuilder().putString("foo", "bar").build()
     val attr = AttributeReference("my_attr", IntegerType, metadata = 
metadata)()
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala
index 8c4daacd1b2f..75095e283ef5 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala
@@ -74,6 +74,17 @@ class ObjectSerializerPruningSuite extends PlanTest {
     }
   }
 
+  test("SPARK-60979: Prune a serializer by the exact field names") {
+    withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+      val rowEncoder = ExpressionEncoder(
+        new StructType().add("s", StructType.fromDDL("a string, A string, b 
int")))
+      val prunedType = StructType.fromDDL("A string")
+      val pruned = ObjectSerializerPruning.pruneSerializer(
+        rowEncoder.namedExpressions.head, prunedType)
+      assert(pruned.dataType === prunedType)
+    }
+  }
+
   test("SPARK-26619: Prune the unused serializers from SerializeFromObject") {
     val testRelation = LocalRelation($"_1".int, $"_2".int)
     val serializerObject = CatalystSerde.serialize[(Int, Int)](
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/DatasetOptimizationSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DatasetOptimizationSuite.scala
index edf93badd64d..8dcb9bd8ce6b 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/DatasetOptimizationSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/DatasetOptimizationSuite.scala
@@ -19,8 +19,9 @@ package org.apache.spark.sql
 
 import org.apache.spark.metrics.source.CodegenMetrics
 import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
-import org.apache.spark.sql.catalyst.expressions.{CreateNamedStruct, 
Expression}
+import org.apache.spark.sql.catalyst.expressions.{CreateNamedStruct, 
Expression, GetStructField}
 import org.apache.spark.sql.catalyst.plans.logical.SerializeFromObject
+import org.apache.spark.sql.classic.ExpressionUtils.column
 import org.apache.spark.sql.functions._
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSparkSession
@@ -208,4 +209,26 @@ class DatasetOptimizationSuite extends SharedSparkSession {
     testSerializer(query, Seq(Seq("i")))
     checkAnswer(query, Seq(Row("a"), Row("b"), Row("c")))
   }
+
+  test("SPARK-60979: Pruned nested serializers: field names differ only by 
case") {
+    withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+      val encoder = ExpressionEncoder(
+        new StructType().add("s", StructType.fromDDL("a string, A string, b 
int")))
+      val ds = spark.range(1).map(_ => Row(Row("x", "y", 1)))(encoder)
+      // `A` by its ordinal as the name is ambiguous in a case-insensitive 
session
+      val query = ds.select(column(GetStructField(col("s").expr, 1)))
+      testSerializer(query, Seq(Seq("A")))
+      checkAnswer(query, Row("y"))
+    }
+  }
+
+  test("SPARK-60979: Pruned nested serializers: references differ in case from 
the schema") {
+    withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+      val encoder = ExpressionEncoder(StructType.fromDDL("s struct<x: INT, y: 
INT>"))
+      val ds = spark.range(1).map(_ => Row(Row(1, 2)))(encoder)
+      checkAnswer(ds.select($"S", $"s.x"), Row(Row(1, 2), 1))
+      checkAnswer(ds.select($"S.x", $"s.y"), Row(1, 2))
+      checkAnswer(ds.select($"S.y"), Row(2))
+    }
+  }
 }
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
index 04b608a4a24b..9e9b2a7dfbe9 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
@@ -264,6 +264,25 @@ abstract class UpdateTableSuiteBase extends 
RowLevelOperationSuiteBase {
     }
   }
 
+  test("SPARK-60979: update keeps columns whose names differ only by case and 
type") {
+    // such a table can only be created and read in a case-sensitive session
+    withSQLConf(SQLConf.CASE_SENSITIVE.key -> "true") {
+      createAndInitTable("pk INT NOT NULL, salary INT NOT NULL, SALARY STRING, 
dep STRING",
+        """{ "pk": 1, "salary": 100, "SALARY": null, "dep": "hr" }
+          |{ "pk": 2, "salary": 300, "SALARY": "x", "dep": "hr" }
+          |""".stripMargin)
+    }
+
+    // the statement does not name the colliding columns so it is not ambiguous
+    sql(s"UPDATE $tableNameAsString SET pk = 3 WHERE pk = 1")
+
+    withSQLConf(SQLConf.CASE_SENSITIVE.key -> "true") {
+      checkAnswer(
+        sql(s"SELECT * FROM $tableNameAsString"),
+        Row(3, 100, null, "hr") :: Row(2, 300, "x", "hr") :: Nil)
+    }
+  }
+
   test("update non-existing records") {
     createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
       """{ "pk": 1, "salary": 100, "dep": "hr" }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to