This is an automated email from the ASF dual-hosted git repository.

gengliangwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 1dc0839ca291 [SPARK-60718][SQL] Check whether an expression accepts an 
indeterminate collation before typing its children
1dc0839ca291 is described below

commit 1dc0839ca2918ae78eae39635a551648c4e0789f
Author: Gengliang Wang <[email protected]>
AuthorDate: Fri Oct 30 11:55:57 2026 -0700

    [SPARK-60718][SQL] Check whether an expression accepts an indeterminate 
collation before typing its children
    
    ### What changes were proposed in this pull request?
    
    `CollationTypeCoercion` checks every expression it visits for an input with 
indeterminate collation. The check computed the data type of every child first, 
and only then asked whether the expression accepts such an input:
    
    ```scala
    expression.children.exists(child =>
      shouldFailWithIndeterminateCollation(expression, getDataTypeSafe(child)))
    ```
    
    `shouldFailWithIndeterminateCollation(expression, dataType)` is 
`!canContainIndeterminateCollation(expression) && 
hasIndeterminateCollation(dataType)`, and almost every expression can contain 
indeterminate collation, so the child types were thrown away.
    
    This PR checks the expression first and computes the child types only when 
the expression does not accept an indeterminate collation:
    
    ```scala
    !canContainIndeterminateCollation(expression) &&
      expression.children.exists(child => 
hasIndeterminateCollation(getDataTypeSafe(child)))
    ```
    
    The first operand does not depend on the child, so the verdict is the same 
for every expression.
    
    ### Why are the changes needed?
    
    Some of those child types are expensive to ask for. The `WithField` and 
`DropField` operations of `UpdateFields` have no data type: 
`StructFieldsOperation.dataType` throws `SparkException.internalError`, and 
`getDataTypeSafe` catches it and returns `NullType`. So analyzing a plan that 
updates struct fields, for example `col("s").withField("c", 
lit(1)).dropFields("b")`, built and discarded one exception, stack trace 
included, for each such operation every time type coercion visited the [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    No. Which expressions fail with `INDETERMINATE_COLLATION_IN_EXPRESSION` is 
unchanged.
    
    ### How was this patch tested?
    
    New `CollationTypeCoercionSuite` (catalyst) with four tests:
    
    - `CollationTypeCoercion`, applied to every node of an `UpdateFields` as 
`CollationTypeCasts` does, does not ask its field operations for a data type. 
The `UpdateFields` has a `WithField`, a field operation and an 
`UpdateNestedField` with a nested field operation; both field operations count 
the requests.
    - An expression that does not accept an indeterminate collation input, 
`upper(x)` over an `IndeterminateStringType` attribute, still fails with 
`INDETERMINATE_COLLATION_IN_EXPRESSION`.
    - So does `'a' = x`, where only the second child has an indeterminate 
collation.
    - An expression that accepts one, `length(x)`, is returned unchanged.
    
    Without the change to `CollationTypeCoercion`, the first test fails and the 
other three pass. With it, all four pass, and so do the 12 tests of the 
existing `IndeterminateCollationTestSuite`:
    
    ```
    build/sbt "catalyst/testOnly 
org.apache.spark.sql.catalyst.analysis.CollationTypeCoercionSuite"
    build/sbt "sql/testOnly 
org.apache.spark.sql.collation.IndeterminateCollationTestSuite"
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #59926 from gengliangwang/SPARK-60718.
    
    Authored-by: Gengliang Wang <[email protected]>
    Signed-off-by: Gengliang Wang <[email protected]>
---
 .../catalyst/analysis/CollationTypeCoercion.scala  |  8 ++-
 .../analysis/CollationTypeCoercionSuite.scala      | 84 ++++++++++++++++++++++
 2 files changed, 90 insertions(+), 2 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
index 902d88f75938..907cdfc7d16b 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercion.scala
@@ -507,8 +507,12 @@ object CollationTypeCoercion extends SQLConfHelper {
       case _: Throwable => NullType
     }
 
-    expression.children.exists(child =>
-      shouldFailWithIndeterminateCollation(expression, getDataTypeSafe(child)))
+    // Most expressions can contain indeterminate collation, so check the 
expression before
+    // typing its children. Some children have no data type and throw an 
internal error when
+    // asked for one, such as the WithField, DropField and UpdateNestedField 
operations of an
+    // UpdateFields.
+    !canContainIndeterminateCollation(expression) &&
+      expression.children.exists(child => 
hasIndeterminateCollation(getDataTypeSafe(child)))
   }
 
   /**
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercionSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercionSuite.scala
new file mode 100644
index 000000000000..3aaa48562cae
--- /dev/null
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/CollationTypeCoercionSuite.scala
@@ -0,0 +1,84 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.analysis
+
+import org.apache.spark.SparkFunSuite
+import org.apache.spark.sql.AnalysisException
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, EqualTo, 
Expression, Length, Literal, StructFieldsOperation, UpdateFields, 
UpdateNestedField, Upper, WithField}
+import org.apache.spark.sql.catalyst.trees.LeafLike
+import org.apache.spark.sql.types.{DataType, IndeterminateStringType, 
IntegerType, StringType, StructType}
+
+class CollationTypeCoercionSuite extends SparkFunSuite {
+
+  private val struct = AttributeReference(
+    "s",
+    new StructType()
+      .add("a", IntegerType)
+      .add("b", StringType)
+      .add("n", new StructType().add("y", StringType)))()
+
+  private val indeterminate = AttributeReference("x", 
IndeterminateStringType)()
+
+  test("does not ask the field operations of UpdateFields for a data type") {
+    val op = CountingFieldOperation("b")
+    val nestedOp = CountingFieldOperation("y")
+    val update = UpdateFields(struct,
+      Seq(WithField("c", Literal(1)), op, UpdateNestedField("n", 
Seq(nestedOp))))
+    // Visit every node, as CollationTypeCasts does.
+    assert(update.transformUp { case e => CollationTypeCoercion(e) } eq update)
+    assert(op.dataTypeRequests == 0)
+    assert(nestedOp.dataTypeRequests == 0)
+  }
+
+  test("fails an expression that does not accept an indeterminate collation 
input") {
+    checkError(
+      exception = intercept[AnalysisException] {
+        CollationTypeCoercion(Upper(indeterminate))
+      },
+      condition = "INDETERMINATE_COLLATION_IN_EXPRESSION",
+      parameters = Map("expr" -> "\"upper(x)\""))
+  }
+
+  test("fails when only a later child has an indeterminate collation") {
+    checkError(
+      exception = intercept[AnalysisException] {
+        CollationTypeCoercion(EqualTo(Literal("a"), indeterminate))
+      },
+      condition = "INDETERMINATE_COLLATION_IN_EXPRESSION",
+      parameters = Map("expr" -> "\"(a = x)\""))
+  }
+
+  test("keeps an expression that accepts an indeterminate collation input") {
+    val length = Length(indeterminate)
+    assert(CollationTypeCoercion(length) eq length)
+  }
+
+  /**
+   * A field operation that counts the requests for its data type, which, like 
every
+   * [[StructFieldsOperation]], it does not have.
+   */
+  private case class CountingFieldOperation(name: String)
+    extends StructFieldsOperation with LeafLike[Expression] {
+    var dataTypeRequests = 0
+
+    override def dataType: DataType = {
+      dataTypeRequests += 1
+      super.dataType
+    }
+  }
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to