This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 075749d08a3b [SPARK-60755][SQL] Fix codegen compilation failure of 
`element_at` and array index when the result is not nullable
075749d08a3b is described below

commit 075749d08a3bf0c3f407600d0b5d5ab790321306
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 31 01:12:58 2026 -0700

    [SPARK-60755][SQL] Fix codegen compilation failure of `element_at` and 
array index when the result is not nullable
    
    ### What changes were proposed in this pull request?
    
    This PR aims to fix the generated code of `ElementAt` and `GetArrayItem` to 
assign `isNull` only when the expression is nullable.
    
    `BinaryExpression.nullSafeCodeGen` declares `ev.isNull` only for a nullable 
expression. When the child is `CreateArray`, or `GetArrayStructFields` over it, 
and the index is a literal, `ElementAt` and `GetArrayItem` can be non-nullable 
because:
    
    - the index selects an element that cannot be NULL,
    - the index is out of bounds under ANSI mode, which raises an error, or
    - the index of `element_at` is 0, which raises an error.
    
    However, `doGenCode` generates `isNull = true` regardless of `nullable` 
when the array can contain NULL, and in the non-ANSI out-of-bounds branch. So, 
the generated Java code assigns to an undeclared variable and fails to compile.
    
    ```java
    UTF8String filter_value_1 = null;
    
    int filter_elementAtIndex_0 = 
org.apache.spark.sql.catalyst.expressions.ArrayExpressionUtils.resolveArrayIndex(
      filter_arrayData_0.numElements(), (int) 5, 
((org.apache.spark.sql.catalyst.trees.SQLQueryContext) references[1] /* errCtx 
*/));
    
    if (filter_arrayData_0.isNullAt(filter_elementAtIndex_0)) {
      filter_isNull_1 = true; // This variable is not declared.
    } else {
      filter_value_1 = 
filter_arrayData_0.getUTF8String(filter_elementAtIndex_0);
    }
    ```
    
    The nullability is correct and this PR does not change it. When the 
expression is not nullable, the removed assignments are unreachable.
    
    ### Why are the changes needed?
    
    The code generation of `element_at` has failed since Apache Spark 
[3.0.0](https://github.com/apache/spark/releases/tag/v3.0.0) (2020-06-06) for a 
selected element that cannot be NULL (SPARK-26965), and since Apache Spark 
[3.1.1](https://github.com/apache/spark/releases/tag/v3.1.1) (2021-02-22) for 
an out-of-bounds index under ANSI mode (SPARK-33386).
    
    The array index has the same defect, but `SimplifyExtractValueOps` removes 
`GetArrayItem` over `CreateArray` for most `INT` literal indexes. So, it fails 
with a literal index of the other integral types such as `0L`, and since Apache 
Spark [4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27) 
with an out-of-bounds `INT` literal index under ANSI mode (SPARK-58431).
    
    Most operators fall back to the interpreted evaluation, which disables 
whole-stage codegen and logs a compilation error per task. The ordering of 
`TakeOrderedAndProjectExec` and the predicates of `MergeRowsExec` have no such 
fallback, so `ORDER BY ... LIMIT` and `MERGE INTO` queries fail.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Queries that failed with `CompileException` now succeed. For example:
    
    ```sql
    SELECT id FROM range(3) ORDER BY element_at(array(id, nullif(id, 0)), 1) 
DESC LIMIT 1;
    ```
    
    **BEFORE** (Apache Spark 
[4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27))
    
    ```
    org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 
51, Column 1: Expression "isNull_0" is not an rvalue
    ```
    
    **AFTER**
    
    ```
    2
    ```
    
    Queries that used the fallback return the same results, without the 
compilation errors in the logs.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases. Each of them fails without 
this fix.
    
    ```
    $ build/sbt 'catalyst/testOnly *CollectionExpressionsSuite 
*ComplexTypeSuite -- -z "generates code when the result is not nullable"'
    $ build/sbt 'sql/testOnly *DataFrameFunctionsSuite -- -z "compile when the 
result is not nullable"'
    ```
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #59963 from dongjoon-hyun/SPARK-60755.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../expressions/collectionOperations.scala         | 10 +++++--
 .../expressions/complexTypeExtractors.scala        | 12 ++++++--
 .../expressions/CollectionExpressionsSuite.scala   | 30 ++++++++++++++++++++
 .../catalyst/expressions/ComplexTypeSuite.scala    | 21 ++++++++++++++
 .../apache/spark/sql/DataFrameFunctionsSuite.scala | 33 +++++++++++++++++++++-
 5 files changed, 100 insertions(+), 6 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index 23cb7119d073..71d9401d63ee 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -3072,7 +3072,9 @@ case class ElementAt(
           val errorContext = getContextOrNullCode(ctx)
           val utils = classOf[ArrayExpressionUtils].getName
           val assignment = s"${ev.value} = ${CodeGenerator.getValue(eval1, 
dataType, index)};"
-          val body = if (arrayElementNullable) {
+          // `ev.isNull` is declared only for a nullable expression. Otherwise 
the index raises
+          // an error or selects an element that cannot be NULL.
+          val body = if (nullable && arrayElementNullable) {
             s"""
                |if ($eval1.isNullAt($index)) {
                |  ${ev.isNull} = true;
@@ -3093,7 +3095,10 @@ case class ElementAt(
         nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
           val index = ctx.freshName("elementAtIndex")
           val intIndex = ctx.freshName("elementAtIntIndex")
-          val nullCheck = if (arrayElementNullable) {
+          // `ev.isNull` is declared only for a nullable expression. Otherwise 
the index is 0, or
+          // selects an element that cannot be NULL, so neither this branch 
nor the out-of-bounds
+          // branch below is taken.
+          val nullCheck = if (nullable && arrayElementNullable) {
             s"""
                |if ($eval1.isNullAt($intIndex)) {
                |  ${ev.isNull} = true;
@@ -3104,6 +3109,7 @@ case class ElementAt(
           }
           val errorContext = getContextOrNullCode(ctx)
           val indexOutOfBoundBranch = defaultValueOutOfBound match {
+            case _ if !nullable => ""
             case Some(value) =>
               val defaultValueEval = value.genCode(ctx)
               s"""
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
index 7b824ea3c69d..074bcc63ce87 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
@@ -404,7 +404,9 @@ case class GetArrayItem(
         val utils = classOf[ArrayExpressionUtils].getName
         val childArrayElementNullable = 
child.dataType.asInstanceOf[ArrayType].containsNull
         val assignment = s"${ev.value} = ${CodeGenerator.getValue(eval1, 
dataType, index)};"
-        val body = if (childArrayElementNullable) {
+        // `ev.isNull` is declared only for a nullable expression. Otherwise 
the index raises an
+        // error or selects an element that cannot be NULL.
+        val body = if (nullable && childArrayElementNullable) {
           s"""
              |if ($eval1.isNullAt($index)) {
              |  ${ev.isNull} = true;
@@ -424,7 +426,10 @@ case class GetArrayItem(
       nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
         val index = ctx.freshName("index")
         val childArrayElementNullable = 
child.dataType.asInstanceOf[ArrayType].containsNull
-        val nullCheck = if (childArrayElementNullable) {
+        // `ev.isNull` is declared only for a nullable expression. Otherwise 
the index selects an
+        // element that cannot be NULL, so neither this branch nor the 
out-of-bounds branch below
+        // is taken.
+        val nullCheck = if (nullable && childArrayElementNullable) {
           s"""else if ($eval1.isNullAt($index)) {
                  ${ev.isNull} = true;
               }
@@ -432,10 +437,11 @@ case class GetArrayItem(
         } else {
           ""
         }
+        val outOfBound = if (nullable) s"${ev.isNull} = true;" else ""
         s"""
           final int $index = (int) $eval2;
           if ($index >= $eval1.numElements() || $index < 0) {
-            ${ev.isNull} = true;
+            $outOfBound
           } $nullCheck else {
             ${ev.value} = ${CodeGenerator.getValue(eval1, dataType, index)};
           }
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
index 211a47282cb9..c6940ea4363f 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
@@ -2528,6 +2528,36 @@ class CollectionExpressionsSuite
     }
   }
 
+  test("SPARK-60755: ElementAt on CreateArray generates code when the result 
is not nullable") {
+    val a = BoundReference(0, StringType, nullable = false)
+    val b = BoundReference(1, StringType, nullable = true)
+    val input = create_row("x", null)
+    val field = StructField("f", StringType, nullable = false)
+    val c = BoundReference(0, StructType(Seq(field)), nullable = false)
+    val d = BoundReference(1, StructType(Seq(field)), nullable = true)
+    val inputWithStruct = create_row(create_row("x"), null)
+    Seq(true, false).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        // The array can contain NULL, but the selected element cannot.
+        checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(1)), "x", 
input)
+        checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(-2)), "x", 
input)
+        val fields = GetArrayStructFields(CreateArray(Seq(c, d)), field, 0, 1, 
containsNull = true)
+        checkEvaluation(ElementAt(fields, Literal(1)), "x", inputWithStruct)
+        // No element can be NULL.
+        checkEvaluation(ElementAt(CreateArray(Seq(a)), Literal(1)), "x", input)
+        // The index is invalid in every row.
+        checkExceptionInExpression[Exception](
+          ElementAt(CreateArray(Seq(b)), Literal(0)), input, "The index 0 is 
invalid")
+        if (ansiEnabled) {
+          checkExceptionInExpression[Exception](
+            ElementAt(CreateArray(Seq(b)), Literal(5)), input, "out of bounds")
+        } else {
+          checkEvaluation(ElementAt(CreateArray(Seq(b)), Literal(5)), null, 
input)
+        }
+      }
+    }
+  }
+
   test("Concat") {
     // Primitive-type elements
     val ai0 = Literal.create(Seq(1, 2, 3), ArrayType(IntegerType, containsNull 
= false))
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
index deed423b71c5..66c3be203f5d 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
@@ -137,6 +137,27 @@ class ComplexTypeSuite extends SparkFunSuite with 
ExpressionEvalHelper {
     assert(GetArrayItem(stArray4, Literal(1)).nullable)
   }
 
+  test("SPARK-60755: GetArrayItem on CreateArray generates code when the 
result is not nullable") {
+    val a = BoundReference(0, StringType, nullable = false)
+    val b = BoundReference(1, StringType, nullable = true)
+    val input = create_row("x", null)
+    Seq(true, false).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        // The array can contain NULL, but the selected element cannot.
+        checkEvaluation(GetArrayItem(CreateArray(Seq(a, b)), Literal(0)), "x", 
input)
+        // No element can be NULL.
+        checkEvaluation(GetArrayItem(CreateArray(Seq(a)), Literal(0)), "x", 
input)
+        // The index is invalid in every row.
+        if (ansiEnabled) {
+          checkExceptionInExpression[Exception](
+            GetArrayItem(CreateArray(Seq(b)), Literal(5)), input, "out of 
bounds")
+        } else {
+          checkEvaluation(GetArrayItem(CreateArray(Seq(b)), Literal(5)), null, 
input)
+        }
+      }
+    }
+  }
+
   Seq((Int.MaxValue, "Linear Lookup"), (0, "Hash Lookup")).foreach { case 
(threshold, name) =>
     test(s"GetMapValue - $name") {
       withSQLConf(SQLConf.MAP_LOOKUP_HASH_THRESHOLD.key -> threshold.toString) 
{
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
index 67ee0b98d27a..b688bf205ef9 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
@@ -23,7 +23,7 @@ import java.sql.{Date, Timestamp}
 import scala.reflect.runtime.universe.runtimeMirror
 import scala.util.Random
 
-import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT, SparkException, 
SparkRuntimeException}
+import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT, 
SparkArrayIndexOutOfBoundsException, SparkException, SparkRuntimeException}
 import org.apache.spark.sql.catalyst.{ExtendedAnalysisException, InternalRow}
 import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
 import org.apache.spark.sql.catalyst.expressions.{ArrayExists, Expression, 
Literal, UnaryExpression}
@@ -2547,6 +2547,37 @@ class DataFrameFunctionsSuite extends SharedSparkSession 
{
     }
   }
 
+  test("SPARK-60755: element_at and array index compile when the result is not 
nullable") {
+    // The first element cannot be NULL, but the array can contain NULL.
+    val df = spark.range(2).selectExpr("id", "if(id = 0, 'a', NULL) AS s")
+    val notNullable = "element_at(array(coalesce(s, 'z'), s), 1)"
+    Seq(true, false).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        checkAnswer(df.selectExpr(notNullable), Seq(Row("a"), Row("z")))
+        checkAnswer(df.where(s"$notNullable = 'z'").select("id"), Seq(Row(1)))
+        // The ordering of `TakeOrderedAndProjectExec` has no interpreted 
fallback.
+        checkAnswer(df.orderBy(expr(notNullable).desc).limit(1).select("id"), 
Seq(Row(1)))
+      }
+    }
+    // Under ANSI, an out-of-bounds index raises an error, so the result is 
not nullable.
+    withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+      checkError(
+        exception = intercept[SparkArrayIndexOutOfBoundsException] {
+          df.where("element_at(array(s), 5) = 'x'").collect()
+        },
+        condition = "INVALID_ARRAY_INDEX_IN_ELEMENT_AT",
+        parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+        context = ExpectedContext(fragment = "element_at(array(s), 5)", start 
= 0, stop = 22))
+      checkError(
+        exception = intercept[SparkArrayIndexOutOfBoundsException] {
+          df.selectExpr("array(s)[5]").collect()
+        },
+        condition = "INVALID_ARRAY_INDEX",
+        parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+        context = ExpectedContext(fragment = "array(s)[5]", start = 0, stop = 
10))
+    }
+  }
+
   test("element_at function") {
     val df = Seq(
       (Seq[String]("1", "2", "3"), 1),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to