This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new 099a1707091c [SPARK-60755][SQL] Fix codegen compilation failure of 
`element_at` and array index when the result is not nullable
099a1707091c is described below

commit 099a1707091c56f6ad1a262fac4f49cc08afa804
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 31 01:12:58 2026 -0700

    [SPARK-60755][SQL] Fix codegen compilation failure of `element_at` and 
array index when the result is not nullable
    
    This PR aims to fix the generated code of `ElementAt` and `GetArrayItem` to 
assign `isNull` only when the expression is nullable.
    
    `BinaryExpression.nullSafeCodeGen` declares `ev.isNull` only for a nullable 
expression. When the child is `CreateArray`, or `GetArrayStructFields` over it, 
and the index is a literal, `ElementAt` and `GetArrayItem` can be non-nullable 
because:
    
    - the index selects an element that cannot be NULL,
    - the index is out of bounds under ANSI mode, which raises an error, or
    - the index of `element_at` is 0, which raises an error.
    
    However, `doGenCode` generates `isNull = true` regardless of `nullable` 
when the array can contain NULL, and in the non-ANSI out-of-bounds branch. So, 
the generated Java code assigns to an undeclared variable and fails to compile.
    
    ```java
    UTF8String filter_value_1 = null;
    
    int filter_elementAtIndex_0 = 
org.apache.spark.sql.catalyst.expressions.ArrayExpressionUtils.resolveArrayIndex(
      filter_arrayData_0.numElements(), (int) 5, 
((org.apache.spark.sql.catalyst.trees.SQLQueryContext) references[1] /* errCtx 
*/));
    
    if (filter_arrayData_0.isNullAt(filter_elementAtIndex_0)) {
      filter_isNull_1 = true; // This variable is not declared.
    } else {
      filter_value_1 = 
filter_arrayData_0.getUTF8String(filter_elementAtIndex_0);
    }
    ```
    
    The nullability is correct and this PR does not change it. When the 
expression is not nullable, the removed assignments are unreachable.
    
    The code generation of `element_at` has failed since Apache Spark 
[3.0.0](https://github.com/apache/spark/releases/tag/v3.0.0) (2020-06-06) for a 
selected element that cannot be NULL (SPARK-26965), and since Apache Spark 
[3.1.1](https://github.com/apache/spark/releases/tag/v3.1.1) (2021-02-22) for 
an out-of-bounds index under ANSI mode (SPARK-33386).
    
    The array index has the same defect, but `SimplifyExtractValueOps` removes 
`GetArrayItem` over `CreateArray` for most `INT` literal indexes. So, it fails 
with a literal index of the other integral types such as `0L`, and since Apache 
Spark [4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27) 
with an out-of-bounds `INT` literal index under ANSI mode (SPARK-58431).
    
    Most operators fall back to the interpreted evaluation, which disables 
whole-stage codegen and logs a compilation error per task. The ordering of 
`TakeOrderedAndProjectExec` and the predicates of `MergeRowsExec` have no such 
fallback, so `ORDER BY ... LIMIT` and `MERGE INTO` queries fail.
    
    Yes. Queries that failed with `CompileException` now succeed. For example:
    
    ```sql
    SELECT id FROM range(3) ORDER BY element_at(array(id, nullif(id, 0)), 1) 
DESC LIMIT 1;
    ```
    
    **BEFORE** (Apache Spark 
[4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27))
    
    ```
    org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 
51, Column 1: Expression "isNull_0" is not an rvalue
    ```
    
    **AFTER**
    
    ```
    2
    ```
    
    Queries that used the fallback return the same results, without the 
compilation errors in the logs.
    
    Pass the CIs with the newly added test cases. Each of them fails without 
this fix.
    
    ```
    $ build/sbt 'catalyst/testOnly *CollectionExpressionsSuite 
*ComplexTypeSuite -- -z "generates code when the result is not nullable"'
    $ build/sbt 'sql/testOnly *DataFrameFunctionsSuite -- -z "compile when the 
result is not nullable"'
    ```
    
    Generated-by: Claude Opus 5.5
    
    Closes #59963 from dongjoon-hyun/SPARK-60755.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 075749d08a3bf0c3f407600d0b5d5ab790321306)
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit e8bea5e8de16fa9709c0c029c51058a65629f357)
    (cherry picked from commit 4452bd79f47f1be8106ba87b50416ee1b558fbaa)
---
 .../expressions/collectionOperations.scala         |  6 ++++-
 .../expressions/complexTypeExtractors.scala        |  9 +++++--
 .../expressions/CollectionExpressionsSuite.scala   | 30 ++++++++++++++++++++++
 .../catalyst/expressions/ComplexTypeSuite.scala    | 21 +++++++++++++++
 .../apache/spark/sql/DataFrameFunctionsSuite.scala | 26 ++++++++++++++++++-
 5 files changed, 88 insertions(+), 4 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index c9a7b9423f70..9b2c974a6b7c 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -2837,7 +2837,10 @@ case class ElementAt(
       case _: ArrayType =>
         nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
           val index = ctx.freshName("elementAtIndex")
-          val nullCheck = if (arrayElementNullable) {
+          // `ev.isNull` is declared only for a nullable expression. Otherwise 
the index is 0, or
+          // selects an element that cannot be NULL, so neither this branch 
nor the out-of-bounds
+          // branch below is taken.
+          val nullCheck = if (nullable && arrayElementNullable) {
             s"""
                |if ($eval1.isNullAt($index)) {
                |  ${ev.isNull} = true;
@@ -2853,6 +2856,7 @@ case class ElementAt(
             // scalastyle:on line.size.limit
           } else {
             defaultValueOutOfBound match {
+              case _ if !nullable => ""
               case Some(value) =>
                 val defaultValueEval = value.genCode(ctx)
                 s"""
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
index ed72e4dc2101..5b066677ad3d 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
@@ -401,7 +401,10 @@ case class GetArrayItem(
     nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
       val index = ctx.freshName("index")
       val childArrayElementNullable = 
child.dataType.asInstanceOf[ArrayType].containsNull
-      val nullCheck = if (childArrayElementNullable) {
+      // `ev.isNull` is declared only for a nullable expression. Otherwise the 
index selects an
+      // element that cannot be NULL, so neither this branch nor the 
out-of-bounds branch below
+      // is taken.
+      val nullCheck = if (nullable && childArrayElementNullable) {
         s"""else if ($eval1.isNullAt($index)) {
                ${ev.isNull} = true;
             }
@@ -415,8 +418,10 @@ case class GetArrayItem(
         // scalastyle:off line.size.limit
         s"throw QueryExecutionErrors.invalidArrayIndexError($index, 
$eval1.numElements(), $errorContext);"
         // scalastyle:on line.size.limit
-      } else {
+      } else if (nullable) {
         s"${ev.isNull} = true;"
+      } else {
+        ""
       }
 
       s"""
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
index be97214fd952..e2a14d9af432 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
@@ -2282,6 +2282,36 @@ class CollectionExpressionsSuite
     }
   }
 
+  test("SPARK-60755: ElementAt on CreateArray generates code when the result 
is not nullable") {
+    val a = BoundReference(0, StringType, nullable = false)
+    val b = BoundReference(1, StringType, nullable = true)
+    val input = create_row("x", null)
+    val field = StructField("f", StringType, nullable = false)
+    val c = BoundReference(0, StructType(Seq(field)), nullable = false)
+    val d = BoundReference(1, StructType(Seq(field)), nullable = true)
+    val inputWithStruct = create_row(create_row("x"), null)
+    Seq(true, false).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        // The array can contain NULL, but the selected element cannot.
+        checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(1)), "x", 
input)
+        checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(-2)), "x", 
input)
+        val fields = GetArrayStructFields(CreateArray(Seq(c, d)), field, 0, 1, 
containsNull = true)
+        checkEvaluation(ElementAt(fields, Literal(1)), "x", inputWithStruct)
+        // No element can be NULL.
+        checkEvaluation(ElementAt(CreateArray(Seq(a)), Literal(1)), "x", input)
+        // The index is invalid in every row.
+        checkExceptionInExpression[Exception](
+          ElementAt(CreateArray(Seq(b)), Literal(0)), input, "The index 0 is 
invalid")
+        if (ansiEnabled) {
+          checkExceptionInExpression[Exception](
+            ElementAt(CreateArray(Seq(b)), Literal(5)), input, "out of bounds")
+        } else {
+          checkEvaluation(ElementAt(CreateArray(Seq(b)), Literal(5)), null, 
input)
+        }
+      }
+    }
+  }
+
   test("Concat") {
     // Primitive-type elements
     val ai0 = Literal.create(Seq(1, 2, 3), ArrayType(IntegerType, containsNull 
= false))
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
index f2a8e5b64d21..dc2926df818c 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
@@ -129,6 +129,27 @@ class ComplexTypeSuite extends SparkFunSuite with 
ExpressionEvalHelper {
     assert(GetArrayItem(stArray4, Literal(1)).nullable)
   }
 
+  test("SPARK-60755: GetArrayItem on CreateArray generates code when the 
result is not nullable") {
+    val a = BoundReference(0, StringType, nullable = false)
+    val b = BoundReference(1, StringType, nullable = true)
+    val input = create_row("x", null)
+    Seq(true, false).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        // The array can contain NULL, but the selected element cannot.
+        checkEvaluation(GetArrayItem(CreateArray(Seq(a, b)), Literal(0)), "x", 
input)
+        // No element can be NULL.
+        checkEvaluation(GetArrayItem(CreateArray(Seq(a)), Literal(0)), "x", 
input)
+        // The index is invalid in every row.
+        if (ansiEnabled) {
+          checkExceptionInExpression[Exception](
+            GetArrayItem(CreateArray(Seq(b)), Literal(5)), input, "out of 
bounds")
+        } else {
+          checkEvaluation(GetArrayItem(CreateArray(Seq(b)), Literal(5)), null, 
input)
+        }
+      }
+    }
+  }
+
   Seq((Int.MaxValue, "Linear Lookup"), (0, "Hash Lookup")).foreach { case 
(threshold, name) =>
     test(s"GetMapValue - $name") {
       withSQLConf(SQLConf.MAP_LOOKUP_HASH_THRESHOLD.key -> threshold.toString) 
{
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
index e2d0c900f58a..deddc56d9e5e 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
@@ -23,7 +23,7 @@ import java.sql.{Date, Timestamp}
 import scala.reflect.runtime.universe.runtimeMirror
 import scala.util.Random
 
-import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT, SparkException, 
SparkRuntimeException}
+import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT, 
SparkArrayIndexOutOfBoundsException, SparkException, SparkRuntimeException}
 import org.apache.spark.sql.catalyst.{ExtendedAnalysisException, InternalRow}
 import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
 import org.apache.spark.sql.catalyst.expressions.{ArrayExists, Expression, 
Literal, UnaryExpression}
@@ -2467,6 +2467,30 @@ class DataFrameFunctionsSuite extends SharedSparkSession 
{
     )
   }
 
+  test("SPARK-60755: element_at and array index compile when the result is not 
nullable") {
+    // The first element cannot be NULL, but the array can contain NULL.
+    val df = spark.range(2).selectExpr("id", "if(id = 0, 'a', NULL) AS s")
+    val notNullable = "element_at(array(coalesce(s, 'z'), s), 1)"
+    Seq(true, false).foreach { ansiEnabled =>
+      withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+        checkAnswer(df.selectExpr(notNullable), Seq(Row("a"), Row("z")))
+        checkAnswer(df.where(s"$notNullable = 'z'").select("id"), Seq(Row(1)))
+        // The ordering of `TakeOrderedAndProjectExec` has no interpreted 
fallback.
+        checkAnswer(df.orderBy(expr(notNullable).desc).limit(1).select("id"), 
Seq(Row(1)))
+      }
+    }
+    // Under ANSI, an out-of-bounds index raises an error, so the result is 
not nullable.
+    withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+      checkError(
+        exception = intercept[SparkArrayIndexOutOfBoundsException] {
+          df.where("element_at(array(s), 5) = 'x'").collect()
+        },
+        condition = "INVALID_ARRAY_INDEX_IN_ELEMENT_AT",
+        parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+        context = ExpectedContext(fragment = "element_at(array(s), 5)", start 
= 0, stop = 22))
+    }
+  }
+
   test("element_at function") {
     val df = Seq(
       (Seq[String]("1", "2", "3"), 1),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to