This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 099a1707091c [SPARK-60755][SQL] Fix codegen compilation failure of
`element_at` and array index when the result is not nullable
099a1707091c is described below
commit 099a1707091c56f6ad1a262fac4f49cc08afa804
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 31 01:12:58 2026 -0700
[SPARK-60755][SQL] Fix codegen compilation failure of `element_at` and
array index when the result is not nullable
This PR aims to fix the generated code of `ElementAt` and `GetArrayItem` to
assign `isNull` only when the expression is nullable.
`BinaryExpression.nullSafeCodeGen` declares `ev.isNull` only for a nullable
expression. When the child is `CreateArray`, or `GetArrayStructFields` over it,
and the index is a literal, `ElementAt` and `GetArrayItem` can be non-nullable
because:
- the index selects an element that cannot be NULL,
- the index is out of bounds under ANSI mode, which raises an error, or
- the index of `element_at` is 0, which raises an error.
However, `doGenCode` generates `isNull = true` regardless of `nullable`
when the array can contain NULL, and in the non-ANSI out-of-bounds branch. So,
the generated Java code assigns to an undeclared variable and fails to compile.
```java
UTF8String filter_value_1 = null;
int filter_elementAtIndex_0 =
org.apache.spark.sql.catalyst.expressions.ArrayExpressionUtils.resolveArrayIndex(
filter_arrayData_0.numElements(), (int) 5,
((org.apache.spark.sql.catalyst.trees.SQLQueryContext) references[1] /* errCtx
*/));
if (filter_arrayData_0.isNullAt(filter_elementAtIndex_0)) {
filter_isNull_1 = true; // This variable is not declared.
} else {
filter_value_1 =
filter_arrayData_0.getUTF8String(filter_elementAtIndex_0);
}
```
The nullability is correct and this PR does not change it. When the
expression is not nullable, the removed assignments are unreachable.
The code generation of `element_at` has failed since Apache Spark
[3.0.0](https://github.com/apache/spark/releases/tag/v3.0.0) (2020-06-06) for a
selected element that cannot be NULL (SPARK-26965), and since Apache Spark
[3.1.1](https://github.com/apache/spark/releases/tag/v3.1.1) (2021-02-22) for
an out-of-bounds index under ANSI mode (SPARK-33386).
The array index has the same defect, but `SimplifyExtractValueOps` removes
`GetArrayItem` over `CreateArray` for most `INT` literal indexes. So, it fails
with a literal index of the other integral types such as `0L`, and since Apache
Spark [4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27)
with an out-of-bounds `INT` literal index under ANSI mode (SPARK-58431).
Most operators fall back to the interpreted evaluation, which disables
whole-stage codegen and logs a compilation error per task. The ordering of
`TakeOrderedAndProjectExec` and the predicates of `MergeRowsExec` have no such
fallback, so `ORDER BY ... LIMIT` and `MERGE INTO` queries fail.
Yes. Queries that failed with `CompileException` now succeed. For example:
```sql
SELECT id FROM range(3) ORDER BY element_at(array(id, nullif(id, 0)), 1)
DESC LIMIT 1;
```
**BEFORE** (Apache Spark
[4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27))
```
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
51, Column 1: Expression "isNull_0" is not an rvalue
```
**AFTER**
```
2
```
Queries that used the fallback return the same results, without the
compilation errors in the logs.
Pass the CIs with the newly added test cases. Each of them fails without
this fix.
```
$ build/sbt 'catalyst/testOnly *CollectionExpressionsSuite
*ComplexTypeSuite -- -z "generates code when the result is not nullable"'
$ build/sbt 'sql/testOnly *DataFrameFunctionsSuite -- -z "compile when the
result is not nullable"'
```
Generated-by: Claude Opus 5.5
Closes #59963 from dongjoon-hyun/SPARK-60755.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 075749d08a3bf0c3f407600d0b5d5ab790321306)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit e8bea5e8de16fa9709c0c029c51058a65629f357)
(cherry picked from commit 4452bd79f47f1be8106ba87b50416ee1b558fbaa)
---
.../expressions/collectionOperations.scala | 6 ++++-
.../expressions/complexTypeExtractors.scala | 9 +++++--
.../expressions/CollectionExpressionsSuite.scala | 30 ++++++++++++++++++++++
.../catalyst/expressions/ComplexTypeSuite.scala | 21 +++++++++++++++
.../apache/spark/sql/DataFrameFunctionsSuite.scala | 26 ++++++++++++++++++-
5 files changed, 88 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index c9a7b9423f70..9b2c974a6b7c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -2837,7 +2837,10 @@ case class ElementAt(
case _: ArrayType =>
nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
val index = ctx.freshName("elementAtIndex")
- val nullCheck = if (arrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise
the index is 0, or
+ // selects an element that cannot be NULL, so neither this branch
nor the out-of-bounds
+ // branch below is taken.
+ val nullCheck = if (nullable && arrayElementNullable) {
s"""
|if ($eval1.isNullAt($index)) {
| ${ev.isNull} = true;
@@ -2853,6 +2856,7 @@ case class ElementAt(
// scalastyle:on line.size.limit
} else {
defaultValueOutOfBound match {
+ case _ if !nullable => ""
case Some(value) =>
val defaultValueEval = value.genCode(ctx)
s"""
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
index ed72e4dc2101..5b066677ad3d 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
@@ -401,7 +401,10 @@ case class GetArrayItem(
nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
val index = ctx.freshName("index")
val childArrayElementNullable =
child.dataType.asInstanceOf[ArrayType].containsNull
- val nullCheck = if (childArrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise the
index selects an
+ // element that cannot be NULL, so neither this branch nor the
out-of-bounds branch below
+ // is taken.
+ val nullCheck = if (nullable && childArrayElementNullable) {
s"""else if ($eval1.isNullAt($index)) {
${ev.isNull} = true;
}
@@ -415,8 +418,10 @@ case class GetArrayItem(
// scalastyle:off line.size.limit
s"throw QueryExecutionErrors.invalidArrayIndexError($index,
$eval1.numElements(), $errorContext);"
// scalastyle:on line.size.limit
- } else {
+ } else if (nullable) {
s"${ev.isNull} = true;"
+ } else {
+ ""
}
s"""
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
index be97214fd952..e2a14d9af432 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
@@ -2282,6 +2282,36 @@ class CollectionExpressionsSuite
}
}
+ test("SPARK-60755: ElementAt on CreateArray generates code when the result
is not nullable") {
+ val a = BoundReference(0, StringType, nullable = false)
+ val b = BoundReference(1, StringType, nullable = true)
+ val input = create_row("x", null)
+ val field = StructField("f", StringType, nullable = false)
+ val c = BoundReference(0, StructType(Seq(field)), nullable = false)
+ val d = BoundReference(1, StructType(Seq(field)), nullable = true)
+ val inputWithStruct = create_row(create_row("x"), null)
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ // The array can contain NULL, but the selected element cannot.
+ checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(1)), "x",
input)
+ checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(-2)), "x",
input)
+ val fields = GetArrayStructFields(CreateArray(Seq(c, d)), field, 0, 1,
containsNull = true)
+ checkEvaluation(ElementAt(fields, Literal(1)), "x", inputWithStruct)
+ // No element can be NULL.
+ checkEvaluation(ElementAt(CreateArray(Seq(a)), Literal(1)), "x", input)
+ // The index is invalid in every row.
+ checkExceptionInExpression[Exception](
+ ElementAt(CreateArray(Seq(b)), Literal(0)), input, "The index 0 is
invalid")
+ if (ansiEnabled) {
+ checkExceptionInExpression[Exception](
+ ElementAt(CreateArray(Seq(b)), Literal(5)), input, "out of bounds")
+ } else {
+ checkEvaluation(ElementAt(CreateArray(Seq(b)), Literal(5)), null,
input)
+ }
+ }
+ }
+ }
+
test("Concat") {
// Primitive-type elements
val ai0 = Literal.create(Seq(1, 2, 3), ArrayType(IntegerType, containsNull
= false))
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
index f2a8e5b64d21..dc2926df818c 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
@@ -129,6 +129,27 @@ class ComplexTypeSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(GetArrayItem(stArray4, Literal(1)).nullable)
}
+ test("SPARK-60755: GetArrayItem on CreateArray generates code when the
result is not nullable") {
+ val a = BoundReference(0, StringType, nullable = false)
+ val b = BoundReference(1, StringType, nullable = true)
+ val input = create_row("x", null)
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ // The array can contain NULL, but the selected element cannot.
+ checkEvaluation(GetArrayItem(CreateArray(Seq(a, b)), Literal(0)), "x",
input)
+ // No element can be NULL.
+ checkEvaluation(GetArrayItem(CreateArray(Seq(a)), Literal(0)), "x",
input)
+ // The index is invalid in every row.
+ if (ansiEnabled) {
+ checkExceptionInExpression[Exception](
+ GetArrayItem(CreateArray(Seq(b)), Literal(5)), input, "out of
bounds")
+ } else {
+ checkEvaluation(GetArrayItem(CreateArray(Seq(b)), Literal(5)), null,
input)
+ }
+ }
+ }
+ }
+
Seq((Int.MaxValue, "Linear Lookup"), (0, "Hash Lookup")).foreach { case
(threshold, name) =>
test(s"GetMapValue - $name") {
withSQLConf(SQLConf.MAP_LOOKUP_HASH_THRESHOLD.key -> threshold.toString)
{
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
index e2d0c900f58a..deddc56d9e5e 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
@@ -23,7 +23,7 @@ import java.sql.{Date, Timestamp}
import scala.reflect.runtime.universe.runtimeMirror
import scala.util.Random
-import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT, SparkException,
SparkRuntimeException}
+import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT,
SparkArrayIndexOutOfBoundsException, SparkException, SparkRuntimeException}
import org.apache.spark.sql.catalyst.{ExtendedAnalysisException, InternalRow}
import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
import org.apache.spark.sql.catalyst.expressions.{ArrayExists, Expression,
Literal, UnaryExpression}
@@ -2467,6 +2467,30 @@ class DataFrameFunctionsSuite extends SharedSparkSession
{
)
}
+ test("SPARK-60755: element_at and array index compile when the result is not
nullable") {
+ // The first element cannot be NULL, but the array can contain NULL.
+ val df = spark.range(2).selectExpr("id", "if(id = 0, 'a', NULL) AS s")
+ val notNullable = "element_at(array(coalesce(s, 'z'), s), 1)"
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ checkAnswer(df.selectExpr(notNullable), Seq(Row("a"), Row("z")))
+ checkAnswer(df.where(s"$notNullable = 'z'").select("id"), Seq(Row(1)))
+ // The ordering of `TakeOrderedAndProjectExec` has no interpreted
fallback.
+ checkAnswer(df.orderBy(expr(notNullable).desc).limit(1).select("id"),
Seq(Row(1)))
+ }
+ }
+ // Under ANSI, an out-of-bounds index raises an error, so the result is
not nullable.
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ checkError(
+ exception = intercept[SparkArrayIndexOutOfBoundsException] {
+ df.where("element_at(array(s), 5) = 'x'").collect()
+ },
+ condition = "INVALID_ARRAY_INDEX_IN_ELEMENT_AT",
+ parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+ context = ExpectedContext(fragment = "element_at(array(s), 5)", start
= 0, stop = 22))
+ }
+ }
+
test("element_at function") {
val df = Seq(
(Seq[String]("1", "2", "3"), 1),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]