This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new a1dc1d317aaa [SPARK-60979][SQL] Fix wrong results by nested schema
pruning when column names differ only by case
a1dc1d317aaa is described below
commit a1dc1d317aaaac28424880c2740d5fea3c0521e3
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 08:08:21 2026 -0800
[SPARK-60979][SQL] Fix wrong results by nested schema pruning when column
names differ only by case
### What changes were proposed in this pull request?
This PR fixes a correctness issue in `SchemaPruning.pruneSchema`. When the
schema has fields whose names differ only by case (for example `salary` and
`SALARY`), the pruned schema could take the type and nullability of the wrong
field in a case-insensitive session (the default).
`pruneSchema` now matches by exact name first, and falls back to the
resolver only when no field of the schema has the exact name of the requested
field:
- `pruneSchema`: replaced the first resolver match (`mergedSchema.find(m =>
resolver(m.name, d.name))`) with an exact-name lookup, then a resolver lookup
that skips requested fields whose exact name exists in the schema.
- `sortLeftFieldsByRight`: the lowercase-keyed `leftStructHashMap` keeps
only the last field for a folded name. It now looks up the exact name first and
uses the folded name only when no other field of the right struct has the exact
name of the matched field.
Case-insensitive matching of differently cased names (SPARK-35096) still
works when there is no exact match.
### Why are the changes needed?
This is a correctness issue. For `salary INT NOT NULL, SALARY STRING`, a
request for `SALARY` produces a pruned schema where `salary` becomes `STRING`
and nullable:
```
before: pk INT NOT NULL, salary STRING, SALARY STRING
after: pk INT NOT NULL, salary INT NOT NULL, SALARY STRING
```
A DataSource V2 scan (`PushDownUtils.pruneColumns`) is then asked for the
wrong type and nullability. With a table whose columns differ only by case,
DSv2 row-level operations were measured to store `0` instead of `NULL`, or to
fail with `PLAN_VALIDATION_FAILED_RULE_IN_BATCH` or a Janino
`CompileException`. These results were reproduced with the row-level ordinal
lookup (`RewriteRowLevelCommand.findColOrdinal`) fixed separately, and
`spark.sql.optimizer.nestedSchemaPruning.enabled=fal [...]
### Does this PR introduce _any_ user-facing change?
Yes, in the narrow case where a table or schema has columns that differ
only by case and the session is case-insensitive. The data source is now asked
for the correct type and nullability of each column instead of those of another
column. Nothing changes when no two names collide under case folding.
### How was this patch tested?
Added a unit test to the catalyst `SchemaPruningSuite`. It fails before the
change and passes after it.
Ran the following suites on this change, all passing:
```
build/sbt 'catalyst/testOnly *expressions.SchemaPruningSuite
*optimizer.NestedColumnAliasingSuite'
build/sbt 'sql/testOnly *ParquetV2SchemaPruningSuite
*OrcV2SchemaPruningSuite *ParquetSchemaPruningSuite *OrcV1SchemaPruningSuite
*FileMetadataStructSuite *FileSourceV2PlanMergingSuite'
build/sbt 'sql/testOnly *DataSourceV2Suite
*V2ScanRelationPushDownSubquerySuite'
```
Also verified DSv2 UPDATE, DELETE and MERGE on group-based, delta, and
delta split-update tables with `salary INT, SALARY STRING` and NULL values in
nullable and `NOT NULL` colliding columns, together with the separate ordinal
lookup fix, and all results were correct.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60156 from dongjoon-hyun/SPARK-60979.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../sql/catalyst/expressions/SchemaPruning.scala | 29 ++++++++++++++++++----
.../spark/sql/catalyst/optimizer/objects.scala | 24 ++++++++++++------
.../catalyst/expressions/SchemaPruningSuite.scala | 16 ++++++++++++
.../optimizer/ObjectSerializerPruningSuite.scala | 11 ++++++++
.../spark/sql/DatasetOptimizationSuite.scala | 25 ++++++++++++++++++-
.../spark/sql/connector/UpdateTableSuiteBase.scala | 19 ++++++++++++++
6 files changed, 111 insertions(+), 13 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
index a0f1bec46410..59055fcefaf7 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
@@ -44,13 +44,25 @@ object SchemaPruning extends SQLConfHelper {
val mergedSchema = requestedRootFields
.map { root: RootField => StructType(Array(root.field)) }
.reduceLeft(_ merge _)
- val mergedDataSchema =
- StructType(schema.map(d => mergedSchema.find(m => resolver(m.name,
d.name)).getOrElse(d)))
+ // Match by the exact name first. Fall back to the resolver only for a
requested field that
+ // no field of the schema has the exact name of, as the schema may have
fields whose names
+ // differ only by case.
+ val mergedByName = firstByName(mergedSchema)
+ val schemaNames = schema.fieldNames.toSet
+ val mergedDataSchema = StructType(schema.map { d =>
+ mergedByName.get(d.name).orElse {
+ mergedSchema.find(m => resolver(m.name, d.name) &&
!schemaNames.contains(m.name))
+ }.getOrElse(d)
+ })
// Sort the fields of mergedDataSchema according to their order in
dataSchema,
// recursively. This makes mergedDataSchema a pruned schema of dataSchema
sortLeftFieldsByRight(mergedDataSchema, schema).asInstanceOf[StructType]
}
+ // Maps each exact field name to the first field with that name.
+ private def firstByName(struct: StructType): Map[String, StructField] =
+ struct.fields.reverseIterator.map(f => f.name -> f).toMap
+
/**
* Sorts the fields and descendant fields of structs in left according to
their order in
* right. This function assumes that the fields of left are a subset of the
fields of
@@ -76,10 +88,17 @@ object SchemaPruning extends SQLConfHelper {
val leftStructHashMap =
HashMap(leftStruct.map(f =>
formatFieldName(f.name)).zip(leftStruct): _*)
+ val leftStructByName = firstByName(leftStruct)
+ val rightFieldNames = rightStruct.fieldNames.toSet
val sortedLeftFields = rightStruct.fieldNames.flatMap { fieldName =>
- val formattedFieldName = formatFieldName(fieldName)
- if (leftStructHashMap.contains(formattedFieldName)) {
- val resolvedLeftStruct = leftStructHashMap(formattedFieldName)
+ // Prefer the exact name. A field of left whose name is the exact
name of another field
+ // of right is not a match for this one.
+ val leftFieldOpt = leftStructByName.get(fieldName).orElse {
+ leftStructHashMap.get(formatFieldName(fieldName))
+ .filterNot(f => rightFieldNames.contains(f.name))
+ }
+ if (leftFieldOpt.isDefined) {
+ val resolvedLeftStruct = leftFieldOpt.get
val leftFieldType = resolvedLeftStruct.dataType
val rightFieldType = rightStruct(fieldName).dataType
val sortedLeftFieldType = sortLeftFieldsByRight(leftFieldType,
rightFieldType)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
index 163b6d9ff782..a6fe22424624 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala
@@ -28,6 +28,7 @@ import org.apache.spark.sql.catalyst.rules._
import org.apache.spark.sql.catalyst.trees.TreePattern._
import org.apache.spark.sql.catalyst.types.DataTypeUtils
import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType,
UserDefinedType}
+import org.apache.spark.sql.util.SchemaUtils
/*
* This file defines optimization rules related to object manipulation (for
the Dataset API).
@@ -151,11 +152,11 @@ object ObjectSerializerPruning extends Rule[LogicalPlan] {
* and a pruned `StructType`.
*/
private def pruneNamedStruct(struct: CreateNamedStruct, prunedType:
StructType) = {
- // Filters out the pruned fields.
- val resolver = conf.resolver
+ // Filters out the pruned fields. The pruned type names its fields after
the fields of the
+ // serializer, so match by the exact name as the names may differ only by
case.
+ val prunedFieldNames = prunedType.fieldNames.toSet
val prunedFields = struct.nameExprs.zip(struct.valExprs).filter { case
(nameExpr, _) =>
- val name = nameExpr.eval(EmptyRow).toString
- prunedType.fieldNames.exists(resolver(_, name))
+ prunedFieldNames.contains(nameExpr.eval(EmptyRow).toString)
}.flatMap(pair => Seq(pair._1, pair._2))
CreateNamedStruct(prunedFields)
@@ -221,7 +222,14 @@ object ObjectSerializerPruning extends Rule[LogicalPlan] {
val usedRefs = p.references
val prunedSerializer = s.serializer.filter(usedRefs.contains)
- val rootFields = SchemaPruning.identifyRootFields(p.projectList,
Seq.empty)
+ // Name the references after the serializer output. The analyzer keeps
the case that the
+ // query used, and pruneSchema and ProjectionOverSchema match names
exactly.
+ val nameById = s.output.map(a => a.exprId -> a.name).toMap
+ val normalizedProjects = p.projectList.map(_.transform {
+ case a: AttributeReference if nameById.contains(a.exprId) =>
+ a.withName(nameById(a.exprId))
+ }.asInstanceOf[NamedExpression])
+ val rootFields = SchemaPruning.identifyRootFields(normalizedProjects,
Seq.empty)
if (conf.serializerNestedSchemaPruningEnabled && rootFields.nonEmpty) {
// Prunes nested fields in serializers.
@@ -233,10 +241,12 @@ object ObjectSerializerPruning extends Rule[LogicalPlan] {
// Builds new projection.
val projectionOverSchema = ProjectionOverSchema(prunedSchema,
AttributeSet(s.output))
- val newProjects = p.projectList.map(_.transformDown {
+ val newProjects = normalizedProjects.map(_.transformDown {
case projectionOverSchema(expr) => expr
}).map { case expr: NamedExpression => expr }
- p.copy(projectList = newProjects,
+ p.copy(
+ projectList =
+ SchemaUtils.restoreOriginalOutputNames(newProjects,
p.projectList.map(_.name)),
child = SerializeFromObject(nestedPrunedSerializer, s.child))
} else {
p.copy(child = SerializeFromObject(prunedSerializer, s.child))
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruningSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruningSuite.scala
index ae0257522727..e3a5fcf58c8b 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruningSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruningSuite.scala
@@ -134,6 +134,22 @@ class SchemaPruningSuite extends SparkFunSuite with
SQLHelper {
}
}
+ test("SPARK-60979: prefer the exact name when field names differ only by
case") {
+ withSQLConf(CASE_SENSITIVE.key -> "false") {
+ // top-level fields
+ testPrunedSchema(
+ StructType.fromDDL("pk INT NOT NULL, salary INT NOT NULL, SALARY
STRING"),
+ Seq(StructField("SALARY", StringType)),
+ StructType.fromDDL("pk INT NOT NULL, salary INT NOT NULL, SALARY
STRING"))
+
+ // nested fields
+ testPrunedSchema(
+ StructType.fromDDL("s struct<a: INT NOT NULL, A: STRING, b: INT>"),
+ Seq(StructField("s", StructType.fromDDL("A STRING"))),
+ StructType.fromDDL("s struct<A: STRING>"))
+ }
+ }
+
test("SPARK-35232: getRootFields/pruneDataSchema should retain attribute
metadata") {
val metadata = new MetadataBuilder().putString("foo", "bar").build()
val attr = AttributeReference("my_attr", IntegerType, metadata =
metadata)()
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala
index 8c4daacd1b2f..75095e283ef5 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ObjectSerializerPruningSuite.scala
@@ -74,6 +74,17 @@ class ObjectSerializerPruningSuite extends PlanTest {
}
}
+ test("SPARK-60979: Prune a serializer by the exact field names") {
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+ val rowEncoder = ExpressionEncoder(
+ new StructType().add("s", StructType.fromDDL("a string, A string, b
int")))
+ val prunedType = StructType.fromDDL("A string")
+ val pruned = ObjectSerializerPruning.pruneSerializer(
+ rowEncoder.namedExpressions.head, prunedType)
+ assert(pruned.dataType === prunedType)
+ }
+ }
+
test("SPARK-26619: Prune the unused serializers from SerializeFromObject") {
val testRelation = LocalRelation($"_1".int, $"_2".int)
val serializerObject = CatalystSerde.serialize[(Int, Int)](
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DatasetOptimizationSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DatasetOptimizationSuite.scala
index edf93badd64d..8dcb9bd8ce6b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/DatasetOptimizationSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/DatasetOptimizationSuite.scala
@@ -19,8 +19,9 @@ package org.apache.spark.sql
import org.apache.spark.metrics.source.CodegenMetrics
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
-import org.apache.spark.sql.catalyst.expressions.{CreateNamedStruct,
Expression}
+import org.apache.spark.sql.catalyst.expressions.{CreateNamedStruct,
Expression, GetStructField}
import org.apache.spark.sql.catalyst.plans.logical.SerializeFromObject
+import org.apache.spark.sql.classic.ExpressionUtils.column
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
@@ -208,4 +209,26 @@ class DatasetOptimizationSuite extends SharedSparkSession {
testSerializer(query, Seq(Seq("i")))
checkAnswer(query, Seq(Row("a"), Row("b"), Row("c")))
}
+
+ test("SPARK-60979: Pruned nested serializers: field names differ only by
case") {
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+ val encoder = ExpressionEncoder(
+ new StructType().add("s", StructType.fromDDL("a string, A string, b
int")))
+ val ds = spark.range(1).map(_ => Row(Row("x", "y", 1)))(encoder)
+ // `A` by its ordinal as the name is ambiguous in a case-insensitive
session
+ val query = ds.select(column(GetStructField(col("s").expr, 1)))
+ testSerializer(query, Seq(Seq("A")))
+ checkAnswer(query, Row("y"))
+ }
+ }
+
+ test("SPARK-60979: Pruned nested serializers: references differ in case from
the schema") {
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "false") {
+ val encoder = ExpressionEncoder(StructType.fromDDL("s struct<x: INT, y:
INT>"))
+ val ds = spark.range(1).map(_ => Row(Row(1, 2)))(encoder)
+ checkAnswer(ds.select($"S", $"s.x"), Row(Row(1, 2), 1))
+ checkAnswer(ds.select($"S.x", $"s.y"), Row(1, 2))
+ checkAnswer(ds.select($"S.y"), Row(2))
+ }
+ }
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
index 04b608a4a24b..9e9b2a7dfbe9 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/connector/UpdateTableSuiteBase.scala
@@ -264,6 +264,25 @@ abstract class UpdateTableSuiteBase extends
RowLevelOperationSuiteBase {
}
}
+ test("SPARK-60979: update keeps columns whose names differ only by case and
type") {
+ // such a table can only be created and read in a case-sensitive session
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "true") {
+ createAndInitTable("pk INT NOT NULL, salary INT NOT NULL, SALARY STRING,
dep STRING",
+ """{ "pk": 1, "salary": 100, "SALARY": null, "dep": "hr" }
+ |{ "pk": 2, "salary": 300, "SALARY": "x", "dep": "hr" }
+ |""".stripMargin)
+ }
+
+ // the statement does not name the colliding columns so it is not ambiguous
+ sql(s"UPDATE $tableNameAsString SET pk = 3 WHERE pk = 1")
+
+ withSQLConf(SQLConf.CASE_SENSITIVE.key -> "true") {
+ checkAnswer(
+ sql(s"SELECT * FROM $tableNameAsString"),
+ Row(3, 100, null, "hr") :: Row(2, 300, "x", "hr") :: Nil)
+ }
+ }
+
test("update non-existing records") {
createAndInitTable("pk INT NOT NULL, salary INT, dep STRING",
"""{ "pk": 1, "salary": 100, "dep": "hr" }
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]