This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new 4452bd79f47f [SPARK-60755][SQL] Fix codegen compilation failure of 
`element_at` and array index when the result is not nullable
4452bd79f47f is described below

commit 4452bd79f47f1be8106ba87b50416ee1b558fbaa
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 31 01:12:58 2026 -0700

    [SPARK-60755][SQL] Fix codegen compilation failure of `element_at` and 
array index when the result is not nullable
    
    This PR aims to fix the generated code of `ElementAt` and `GetArrayItem` to 
assign `isNull` only when the expression is nullable.
    
    `BinaryExpression.nullSafeCodeGen` declares `ev.isNull` only for a nullable 
expression. When the child is `CreateArray`, or `GetArrayStructFields` over it, 
and the index is a literal, `ElementAt` and `GetArrayItem` can be non-nullable 
because:
    
    - the index selects an element that cannot be NULL,
    - the index is out of bounds under ANSI mode, which raises an error, or
    - the index of `element_at` is 0, which raises an error.
    
    However, `doGenCode` generates `isNull = true` regardless of `nullable` 
when the array can contain NULL, and in the non-ANSI out-of-bounds branch. So, 
the generated Java code assigns to an undeclared variable and fails to compile.
    
    ```java
    UTF8String filter_value_1 = null;
    
    int filter_elementAtIndex_0 = 
org.apache.spark.sql.catalyst.expressions.ArrayExpressionUtils.resolveArrayIndex(
      filter_arrayData_0.numElements(), (int) 5, 
((org.apache.spark.sql.catalyst.trees.SQLQueryContext) references[1] /* errCtx 
*/));
    
    if (filter_arrayData_0.isNullAt(filter_elementAtIndex_0)) {
      filter_isNull_1 = true; // This variable is not declared.
    } else {
      filter_value_1 = 
filter_arrayData_0.getUTF8String(filter_elementAtIndex_0);
    }
    ```
    
    The nullability is correct and this PR does not change it. When the 
expression is not nullable, the removed assignments are unreachable.
    
    The code generation of `element_at` has failed since Apache Spark 
[3.0.0](https://github.com/apache/spark/releases/tag/v3.0.0) (2020-06-06) for a 
selected element that cannot be NULL (SPARK-26965), and since Apache Spark 
[3.1.1](https://github.com/apache/spark/releases/tag/v3.1.1) (2021-02-22) for 
an out-of-bounds index under ANSI mode (SPARK-33386).
    
    The array index has the same defect, but `SimplifyExtractValueOps` removes 
`GetArrayItem` over `CreateArray` for most `INT` literal indexes. So, it fails 
with a literal index of the other integral types such as `0L`, and since Apache 
Spark [4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27) 
with an out-of-bounds `INT` literal index under ANSI mode (SPARK-58431).
    
    Most operators fall back to the interpreted evaluation, which disables 
whole-stage codegen and logs a compilation error per task. The ordering of 
`TakeOrderedAndProjectExec` and the predicates of `MergeRowsExec` have no such 
fallback, so `ORDER BY ... LIMIT` and `MERGE INTO` queries fail.
    
    Yes. Queries that failed with `CompileException` now succeed. For example:
    
    ```sql
    SELECT id FROM range(3) ORDER BY element_at(array(id, nullif(id, 0)), 1) 
DESC LIMIT 1;
    ```
    
    **BEFORE** (Apache Spark 
[4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27))
    
    ```
    org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 
51, Column 1: Expression "isNull_0" is not an rvalue
    ```
    
    **AFTER**
    
    ```
    2
    ```
    
    Queries that used the fallback return the same results, without the 
compilation errors in the logs.
    
    Pass the CIs with the newly added test cases. Each of them fails without 
this fix.
    
    ```
    $ build/sbt 'catalyst/testOnly *CollectionExpressionsSuite 
*ComplexTypeSuite -- -z "generates code when the result is not nullable"'
    $ build/sbt 'sql/testOnly *DataFrameFunctionsSuite -- -z "compile when the 
result is not nullable"'
    ```
    
    Generated-by: Claude Opus 5.5
    
    Closes #59963 from dongjoon-hyun/SPARK-60755.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 075749d08a3bf0c3f407600d0b5d5ab790321306)
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit e8bea5e8de16fa9709c0c029c51058a65629f357)
---
 .../expressions/collectionOperations.scala         | 10 +++++--
 .../expressions/complexTypeExtractors.scala        | 12 ++++++--
 .../expressions/CollectionExpressionsSuite.scala   | 30 ++++++++++++++++++++
 .../catalyst/expressions/ComplexTypeSuite.scala    | 21 ++++++++++++++
 .../apache/spark/sql/DataFrameFunctionsSuite.scala | 33 +++++++++++++++++++++-
 5 files changed, 100 insertions(+), 6 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index fedea7cda76d..9bbf8130bda5 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -2983,7 +2983,9 @@ case class ElementAt(
           val errorContext = getContextOrNullCode(ctx)
           val utils = classOf[ArrayExpressionUtils].getName
           val assignment = s"${ev.value} = ${CodeGenerator.getValue(eval1, 
dataType, index)};"
-          val body = if (arrayElementNullable) {
+          // `ev.isNull` is declared only for a nullable expression. Otherwise 
the index raises
+          // an error or selects an element that cannot be NULL.
+          val body = if (nullable && arrayElementNullable) {
             s"""
                |if ($eval1.isNullAt($index)) {
                |  ${ev.isNull} = true;
@@ -3004,7 +3006,10 @@ case class ElementAt(
         nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
           val index = ctx.freshName("elementAtIndex")
           val intIndex = ctx.freshName("elementAtIntIndex")
-          val nullCheck = if (arrayElementNullable) {
+          // `ev.isNull` is declared only for a nullable expression. Otherwise 
the index is 0, or
+          // selects an element that cannot be NULL, so neither this branch 
nor the out-of-bounds
+          // branch below is taken.
+          val nullCheck = if (nullable && arrayElementNullable) {
             s"""
                |if ($eval1.isNullAt($intIndex)) {
                |  ${ev.isNull} = true;
@@ -3015,6 +3020,7 @@ case class ElementAt(
           }
           val errorContext = getContextOrNullCode(ctx)
           val indexOutOfBoundBranch = defaultValueOutOfBound match {
+            case _ if !nullable => ""
             case Some(value) =>
               val defaultValueEval = value.genCode(ctx)
               s"""
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
index 2f9c1e2dd704..5cba61088da6 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
@@ -404,7 +404,9 @@ case class GetArrayItem(
         val utils = classOf[ArrayExpressionUtils].getName
         val childArrayElementNullable = 
child.dataType.asInstanceOf[ArrayType].containsNull
         val assignment = s"${ev.value} = ${CodeGenerator.getValue(eval1, 
dataType, index)};"
-        val body = if (childArrayElementNullable) {
+        // `ev.isNull` is declared only for a nullable expression. Otherwise 
the index raises an
+        // error or selects an element that cannot be NULL.
+        val body = if (nullable && childArrayElementNullable) {
           s"""
              |if ($eval1.isNullAt($index)) {
              |  ${ev.isNull} = true;
@@ -424,7 +426,10 @@ case class GetArrayItem(
       nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
         val index = ctx.freshName("index")
         val childArrayElementNullable = 
child.dataType.asInstanceOf[ArrayType].containsNull
-        val nullCheck = if (childArrayElementNullable) {
+        // `ev.isNull` is declared only for a nullable expression. Otherwise 
the index selects an
+        // element that cannot be NULL, so neither this branch nor the 
out-of-bounds branch below
+        // is taken.
+        val nullCheck = if (nullable && childArrayElementNullable) {
           s"""else if ($eval1.isNullAt($index)) {
                  ${ev.isNull} = true;
               }
@@ -432,10 +437,11 @@ case class GetArrayItem(
         } else {
           ""
         }
+        val outOfBound = if (nullable) s"${ev.isNull} = true;" else ""
         s"""
           final int $index = (int) $eval2;
           if ($index >= $eval1.numElements() || $index < 0) {
-            ${ev.isNull} = true;
+            $outOfBound
           } $nullCheck else {
             ${ev.value} = ${CodeGenerator.getValue(eval1, dataType, index)};
           }
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
index 44bbff9214be..d98060cd98c9 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
@@ -2342,6 +2342,36 @@ class CollectionExpressionsSuite
     }
   }
 
+  test("SPARK-60755: ElementAt on CreateArray generates code when the result 
is not nullable") {
+    val a = BoundReference(0, StringType, nullable = false)
+    val b = BoundReference(1, StringType, nullable = true)
+    val input = create_row("x", null)
+    val field = StructField("f", StringType, nullable = false)
+    val c = BoundReference(0, StructType(Seq(field)), nullable = false)
+    val d = BoundReference(1, StructType(Seq(field)), nullable = true)
+    val inputWithStruct = create_row(create_row("x"), null)
+    Seq(true, false).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        // The array can contain NULL, but the selected element cannot.
+        checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(1)), "x", 
input)
+        checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(-2)), "x", 
input)
+        val fields = GetArrayStructFields(CreateArray(Seq(c, d)), field, 0, 1, 
containsNull = true)
+        checkEvaluation(ElementAt(fields, Literal(1)), "x", inputWithStruct)
+        // No element can be NULL.
+        checkEvaluation(ElementAt(CreateArray(Seq(a)), Literal(1)), "x", input)
+        // The index is invalid in every row.
+        checkExceptionInExpression[Exception](
+          ElementAt(CreateArray(Seq(b)), Literal(0)), input, "The index 0 is 
invalid")
+        if (ansiEnabled) {
+          checkExceptionInExpression[Exception](
+            ElementAt(CreateArray(Seq(b)), Literal(5)), input, "out of bounds")
+        } else {
+          checkEvaluation(ElementAt(CreateArray(Seq(b)), Literal(5)), null, 
input)
+        }
+      }
+    }
+  }
+
   test("Concat") {
     // Primitive-type elements
     val ai0 = Literal.create(Seq(1, 2, 3), ArrayType(IntegerType, containsNull 
= false))
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
index abc28ff203f8..cc91478a0fbf 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
@@ -129,6 +129,27 @@ class ComplexTypeSuite extends SparkFunSuite with 
ExpressionEvalHelper {
     assert(GetArrayItem(stArray4, Literal(1)).nullable)
   }
 
+  test("SPARK-60755: GetArrayItem on CreateArray generates code when the 
result is not nullable") {
+    val a = BoundReference(0, StringType, nullable = false)
+    val b = BoundReference(1, StringType, nullable = true)
+    val input = create_row("x", null)
+    Seq(true, false).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        // The array can contain NULL, but the selected element cannot.
+        checkEvaluation(GetArrayItem(CreateArray(Seq(a, b)), Literal(0)), "x", 
input)
+        // No element can be NULL.
+        checkEvaluation(GetArrayItem(CreateArray(Seq(a)), Literal(0)), "x", 
input)
+        // The index is invalid in every row.
+        if (ansiEnabled) {
+          checkExceptionInExpression[Exception](
+            GetArrayItem(CreateArray(Seq(b)), Literal(5)), input, "out of 
bounds")
+        } else {
+          checkEvaluation(GetArrayItem(CreateArray(Seq(b)), Literal(5)), null, 
input)
+        }
+      }
+    }
+  }
+
   Seq((Int.MaxValue, "Linear Lookup"), (0, "Hash Lookup")).foreach { case 
(threshold, name) =>
     test(s"GetMapValue - $name") {
       withSQLConf(SQLConf.MAP_LOOKUP_HASH_THRESHOLD.key -> threshold.toString) 
{
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
index d46aebeba8c7..f5ce6a716a6a 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
@@ -23,7 +23,7 @@ import java.sql.{Date, Timestamp}
 import scala.reflect.runtime.universe.runtimeMirror
 import scala.util.Random
 
-import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT, SparkException, 
SparkRuntimeException}
+import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT, 
SparkArrayIndexOutOfBoundsException, SparkException, SparkRuntimeException}
 import org.apache.spark.sql.catalyst.{ExtendedAnalysisException, InternalRow}
 import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
 import org.apache.spark.sql.catalyst.expressions.{ArrayExists, Expression, 
Literal, UnaryExpression}
@@ -2515,6 +2515,37 @@ class DataFrameFunctionsSuite extends SharedSparkSession 
{
     )
   }
 
+  test("SPARK-60755: element_at and array index compile when the result is not 
nullable") {
+    // The first element cannot be NULL, but the array can contain NULL.
+    val df = spark.range(2).selectExpr("id", "if(id = 0, 'a', NULL) AS s")
+    val notNullable = "element_at(array(coalesce(s, 'z'), s), 1)"
+    Seq(true, false).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        checkAnswer(df.selectExpr(notNullable), Seq(Row("a"), Row("z")))
+        checkAnswer(df.where(s"$notNullable = 'z'").select("id"), Seq(Row(1)))
+        // The ordering of `TakeOrderedAndProjectExec` has no interpreted 
fallback.
+        checkAnswer(df.orderBy(expr(notNullable).desc).limit(1).select("id"), 
Seq(Row(1)))
+      }
+    }
+    // Under ANSI, an out-of-bounds index raises an error, so the result is 
not nullable.
+    withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+      checkError(
+        exception = intercept[SparkArrayIndexOutOfBoundsException] {
+          df.where("element_at(array(s), 5) = 'x'").collect()
+        },
+        condition = "INVALID_ARRAY_INDEX_IN_ELEMENT_AT",
+        parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+        context = ExpectedContext(fragment = "element_at(array(s), 5)", start 
= 0, stop = 22))
+      checkError(
+        exception = intercept[SparkArrayIndexOutOfBoundsException] {
+          df.selectExpr("array(s)[5]").collect()
+        },
+        condition = "INVALID_ARRAY_INDEX",
+        parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+        context = ExpectedContext(fragment = "array(s)[5]", start = 0, stop = 
10))
+    }
+  }
+
   test("element_at function") {
     val df = Seq(
       (Seq[String]("1", "2", "3"), 1),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to