This is an automated email from the ASF dual-hosted git repository.
gengliangwang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new eb786b7df449 [SPARK-60718][SQL] Check whether an expression accepts an
indeterminate collation before typing its children
eb786b7df449 is described below
commit eb786b7df4491fb48d4dabee1b790532e6c5a196
Author: Gengliang Wang <[email protected]>
AuthorDate: Fri Oct 30 11:55:57 2026 -0700
[SPARK-60718][SQL] Check whether an expression accepts an indeterminate
collation before typing its children
### What changes were proposed in this pull request?
`CollationTypeCoercion` checks every expression it visits for an input with
indeterminate collation. The check computed the data type of every child first,
and only then asked whether the expression accepts such an input:
```scala
expression.children.exists(child =>
shouldFailWithIndeterminateCollation(expression, getDataTypeSafe(child)))
```
`shouldFailWithIndeterminateCollation(expression, dataType)` is
`!canContainIndeterminateCollation(expression) &&
hasIndeterminateCollation(dataType)`, and almost every expression can contain
indeterminate collation, so the child types were thrown away.
This PR checks the expression first and computes the child types only when
the expression does not accept an indeterminate collation:
```scala
!canContainIndeterminateCollation(expression) &&
expression.children.exists(child =>
hasIndeterminateCollation(getDataTypeSafe(child)))
```
The first operand does not depend on the child, so the verdict is the same
for every expression.
### Why are the changes needed?
Some of those child types are expensive to ask for. The `WithField` and
`DropField` operations of `UpdateFields` have no data type:
`StructFieldsOperation.dataType` throws `SparkException.internalError`, and
`getDataTypeSafe` catches it and returns `NullType`. So analyzing a plan that
updates struct fields, for example `col("s").withField("c",
lit(1)).dropFields("b")`, built and discarded one exception, stack trace
included, for each such operation every time type coercion visited the [...]
### Does this PR introduce _any_ user-facing change?
No. Which expressions fail with `INDETERMINATE_COLLATION_IN_EXPRESSION` is
unchanged.
### How was this patch tested?
New `CollationTypeCoercionSuite` (catalyst) with four tests:
- `CollationTypeCoercion`, applied to every node of an `UpdateFields` as
`CollationTypeCasts` does, does not ask its field operations for a data type.
The `UpdateFields` has a `WithField`, a field operation and an
`UpdateNestedField` with a nested field operation; both field operations count
the requests.
- An expression that does not accept an indeterminate collation input,
`upper(x)` over an `IndeterminateStringType` attribute, still fails with
`INDETERMINATE_COLLATION_IN_EXPRESSION`.
- So does `'a' = x`, where only the second child has an indeterminate
collation.
- An expression that accepts one, `length(x)`, is returned unchanged.
Without the change to `CollationTypeCoercion`, the first test fails and the
other three pass. With it, all four pass, and so do the 12 tests of the
existing `IndeterminateCollationTestSuite`:
```
build/sbt "catalyst/testOnly
org.apache.spark.sql.catalyst.analysis.CollationTypeCoercionSuite"
build/sbt "sql/testOnly
org.apache.spark.sql.collation.IndeterminateCollationTestSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5.5)
This pull request and its description were written by Isaac.
Closes #59926 from gengliangwang/SPARK-60718.
Authored-by: Gengliang Wang <[email protected]>
Signed-off-by: Gengliang Wang <[email protected]>
(cherry picked from commit 1dc0839ca2918ae78eae39635a551648c4e0789f)
Signed-off-by: Gengliang Wang <[email protected]>
---
.../catalyst/analysis/CollationTypeCoercion.scala | 8 ++-
.../analysis/CollationTypeCoercionSuite.scala | 84 ++++++++++++++++++++++
2 files changed, 90 insertions(+), 2 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
index 902d88f75938..907cdfc7d16b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
@@ -507,8 +507,12 @@ object CollationTypeCoercion extends SQLConfHelper {
case _: Throwable => NullType
}
- expression.children.exists(child =>
- shouldFailWithIndeterminateCollation(expression, getDataTypeSafe(child)))
+ // Most expressions can contain indeterminate collation, so check the
expression before
+ // typing its children. Some children have no data type and throw an
internal error when
+ // asked for one, such as the WithField, DropField and UpdateNestedField
operations of an
+ // UpdateFields.
+ !canContainIndeterminateCollation(expression) &&
+ expression.children.exists(child =>
hasIndeterminateCollation(getDataTypeSafe(child)))
}
/**
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercionSuite.scala
new file mode 100644
index 000000000000..3aaa48562cae
--- /dev/null
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercionSuite.scala
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, EqualTo,
Expression, Length, Literal, StructFieldsOperation, UpdateFields,
UpdateNestedField, Upper, WithField}
+import org.apache.spark.sql.catalyst.trees.LeafLike
+import org.apache.spark.sql.types.{DataType, IndeterminateStringType,
IntegerType, StringType, StructType}
+
+class CollationTypeCoercionSuite extends SparkFunSuite {
+
+ private val struct = AttributeReference(
+ "s",
+ new StructType()
+ .add("a", IntegerType)
+ .add("b", StringType)
+ .add("n", new StructType().add("y", StringType)))()
+
+ private val indeterminate = AttributeReference("x",
IndeterminateStringType)()
+
+ test("does not ask the field operations of UpdateFields for a data type") {
+ val op = CountingFieldOperation("b")
+ val nestedOp = CountingFieldOperation("y")
+ val update = UpdateFields(struct,
+ Seq(WithField("c", Literal(1)), op, UpdateNestedField("n",
Seq(nestedOp))))
+ // Visit every node, as CollationTypeCasts does.
+ assert(update.transformUp { case e => CollationTypeCoercion(e) } eq update)
+ assert(op.dataTypeRequests == 0)
+ assert(nestedOp.dataTypeRequests == 0)
+ }
+
+ test("fails an expression that does not accept an indeterminate collation
input") {
+ checkError(
+ exception = intercept[AnalysisException] {
+ CollationTypeCoercion(Upper(indeterminate))
+ },
+ condition = "INDETERMINATE_COLLATION_IN_EXPRESSION",
+ parameters = Map("expr" -> "\"upper(x)\""))
+ }
+
+ test("fails when only a later child has an indeterminate collation") {
+ checkError(
+ exception = intercept[AnalysisException] {
+ CollationTypeCoercion(EqualTo(Literal("a"), indeterminate))
+ },
+ condition = "INDETERMINATE_COLLATION_IN_EXPRESSION",
+ parameters = Map("expr" -> "\"(a = x)\""))
+ }
+
+ test("keeps an expression that accepts an indeterminate collation input") {
+ val length = Length(indeterminate)
+ assert(CollationTypeCoercion(length) eq length)
+ }
+
+ /**
+ * A field operation that counts the requests for its data type, which, like
every
+ * [[StructFieldsOperation]], it does not have.
+ */
+ private case class CountingFieldOperation(name: String)
+ extends StructFieldsOperation with LeafLike[Expression] {
+ var dataTypeRequests = 0
+
+ override def dataType: DataType = {
+ dataTypeRequests += 1
+ super.dataType
+ }
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]