This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 075749d08a3b [SPARK-60755][SQL] Fix codegen compilation failure of
`element_at` and array index when the result is not nullable
075749d08a3b is described below
commit 075749d08a3bf0c3f407600d0b5d5ab790321306
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 31 01:12:58 2026 -0700
[SPARK-60755][SQL] Fix codegen compilation failure of `element_at` and
array index when the result is not nullable
### What changes were proposed in this pull request?
This PR aims to fix the generated code of `ElementAt` and `GetArrayItem` to
assign `isNull` only when the expression is nullable.
`BinaryExpression.nullSafeCodeGen` declares `ev.isNull` only for a nullable
expression. When the child is `CreateArray`, or `GetArrayStructFields` over it,
and the index is a literal, `ElementAt` and `GetArrayItem` can be non-nullable
because:
- the index selects an element that cannot be NULL,
- the index is out of bounds under ANSI mode, which raises an error, or
- the index of `element_at` is 0, which raises an error.
However, `doGenCode` generates `isNull = true` regardless of `nullable`
when the array can contain NULL, and in the non-ANSI out-of-bounds branch. So,
the generated Java code assigns to an undeclared variable and fails to compile.
```java
UTF8String filter_value_1 = null;
int filter_elementAtIndex_0 =
org.apache.spark.sql.catalyst.expressions.ArrayExpressionUtils.resolveArrayIndex(
filter_arrayData_0.numElements(), (int) 5,
((org.apache.spark.sql.catalyst.trees.SQLQueryContext) references[1] /* errCtx
*/));
if (filter_arrayData_0.isNullAt(filter_elementAtIndex_0)) {
filter_isNull_1 = true; // This variable is not declared.
} else {
filter_value_1 =
filter_arrayData_0.getUTF8String(filter_elementAtIndex_0);
}
```
The nullability is correct and this PR does not change it. When the
expression is not nullable, the removed assignments are unreachable.
### Why are the changes needed?
The code generation of `element_at` has failed since Apache Spark
[3.0.0](https://github.com/apache/spark/releases/tag/v3.0.0) (2020-06-06) for a
selected element that cannot be NULL (SPARK-26965), and since Apache Spark
[3.1.1](https://github.com/apache/spark/releases/tag/v3.1.1) (2021-02-22) for
an out-of-bounds index under ANSI mode (SPARK-33386).
The array index has the same defect, but `SimplifyExtractValueOps` removes
`GetArrayItem` over `CreateArray` for most `INT` literal indexes. So, it fails
with a literal index of the other integral types such as `0L`, and since Apache
Spark [4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27)
with an out-of-bounds `INT` literal index under ANSI mode (SPARK-58431).
Most operators fall back to the interpreted evaluation, which disables
whole-stage codegen and logs a compilation error per task. The ordering of
`TakeOrderedAndProjectExec` and the predicates of `MergeRowsExec` have no such
fallback, so `ORDER BY ... LIMIT` and `MERGE INTO` queries fail.
### Does this PR introduce _any_ user-facing change?
Yes. Queries that failed with `CompileException` now succeed. For example:
```sql
SELECT id FROM range(3) ORDER BY element_at(array(id, nullif(id, 0)), 1)
DESC LIMIT 1;
```
**BEFORE** (Apache Spark
[4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27))
```
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
51, Column 1: Expression "isNull_0" is not an rvalue
```
**AFTER**
```
2
```
Queries that used the fallback return the same results, without the
compilation errors in the logs.
### How was this patch tested?
Pass the CIs with the newly added test cases. Each of them fails without
this fix.
```
$ build/sbt 'catalyst/testOnly *CollectionExpressionsSuite
*ComplexTypeSuite -- -z "generates code when the result is not nullable"'
$ build/sbt 'sql/testOnly *DataFrameFunctionsSuite -- -z "compile when the
result is not nullable"'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #59963 from dongjoon-hyun/SPARK-60755.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../expressions/collectionOperations.scala | 10 +++++--
.../expressions/complexTypeExtractors.scala | 12 ++++++--
.../expressions/CollectionExpressionsSuite.scala | 30 ++++++++++++++++++++
.../catalyst/expressions/ComplexTypeSuite.scala | 21 ++++++++++++++
.../apache/spark/sql/DataFrameFunctionsSuite.scala | 33 +++++++++++++++++++++-
5 files changed, 100 insertions(+), 6 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index 23cb7119d073..71d9401d63ee 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -3072,7 +3072,9 @@ case class ElementAt(
val errorContext = getContextOrNullCode(ctx)
val utils = classOf[ArrayExpressionUtils].getName
val assignment = s"${ev.value} = ${CodeGenerator.getValue(eval1,
dataType, index)};"
- val body = if (arrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise
the index raises
+ // an error or selects an element that cannot be NULL.
+ val body = if (nullable && arrayElementNullable) {
s"""
|if ($eval1.isNullAt($index)) {
| ${ev.isNull} = true;
@@ -3093,7 +3095,10 @@ case class ElementAt(
nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
val index = ctx.freshName("elementAtIndex")
val intIndex = ctx.freshName("elementAtIntIndex")
- val nullCheck = if (arrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise
the index is 0, or
+ // selects an element that cannot be NULL, so neither this branch
nor the out-of-bounds
+ // branch below is taken.
+ val nullCheck = if (nullable && arrayElementNullable) {
s"""
|if ($eval1.isNullAt($intIndex)) {
| ${ev.isNull} = true;
@@ -3104,6 +3109,7 @@ case class ElementAt(
}
val errorContext = getContextOrNullCode(ctx)
val indexOutOfBoundBranch = defaultValueOutOfBound match {
+ case _ if !nullable => ""
case Some(value) =>
val defaultValueEval = value.genCode(ctx)
s"""
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
index 7b824ea3c69d..074bcc63ce87 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
@@ -404,7 +404,9 @@ case class GetArrayItem(
val utils = classOf[ArrayExpressionUtils].getName
val childArrayElementNullable =
child.dataType.asInstanceOf[ArrayType].containsNull
val assignment = s"${ev.value} = ${CodeGenerator.getValue(eval1,
dataType, index)};"
- val body = if (childArrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise
the index raises an
+ // error or selects an element that cannot be NULL.
+ val body = if (nullable && childArrayElementNullable) {
s"""
|if ($eval1.isNullAt($index)) {
| ${ev.isNull} = true;
@@ -424,7 +426,10 @@ case class GetArrayItem(
nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
val index = ctx.freshName("index")
val childArrayElementNullable =
child.dataType.asInstanceOf[ArrayType].containsNull
- val nullCheck = if (childArrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise
the index selects an
+ // element that cannot be NULL, so neither this branch nor the
out-of-bounds branch below
+ // is taken.
+ val nullCheck = if (nullable && childArrayElementNullable) {
s"""else if ($eval1.isNullAt($index)) {
${ev.isNull} = true;
}
@@ -432,10 +437,11 @@ case class GetArrayItem(
} else {
""
}
+ val outOfBound = if (nullable) s"${ev.isNull} = true;" else ""
s"""
final int $index = (int) $eval2;
if ($index >= $eval1.numElements() || $index < 0) {
- ${ev.isNull} = true;
+ $outOfBound
} $nullCheck else {
${ev.value} = ${CodeGenerator.getValue(eval1, dataType, index)};
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
index 211a47282cb9..c6940ea4363f 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
@@ -2528,6 +2528,36 @@ class CollectionExpressionsSuite
}
}
+ test("SPARK-60755: ElementAt on CreateArray generates code when the result
is not nullable") {
+ val a = BoundReference(0, StringType, nullable = false)
+ val b = BoundReference(1, StringType, nullable = true)
+ val input = create_row("x", null)
+ val field = StructField("f", StringType, nullable = false)
+ val c = BoundReference(0, StructType(Seq(field)), nullable = false)
+ val d = BoundReference(1, StructType(Seq(field)), nullable = true)
+ val inputWithStruct = create_row(create_row("x"), null)
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ // The array can contain NULL, but the selected element cannot.
+ checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(1)), "x",
input)
+ checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(-2)), "x",
input)
+ val fields = GetArrayStructFields(CreateArray(Seq(c, d)), field, 0, 1,
containsNull = true)
+ checkEvaluation(ElementAt(fields, Literal(1)), "x", inputWithStruct)
+ // No element can be NULL.
+ checkEvaluation(ElementAt(CreateArray(Seq(a)), Literal(1)), "x", input)
+ // The index is invalid in every row.
+ checkExceptionInExpression[Exception](
+ ElementAt(CreateArray(Seq(b)), Literal(0)), input, "The index 0 is
invalid")
+ if (ansiEnabled) {
+ checkExceptionInExpression[Exception](
+ ElementAt(CreateArray(Seq(b)), Literal(5)), input, "out of bounds")
+ } else {
+ checkEvaluation(ElementAt(CreateArray(Seq(b)), Literal(5)), null,
input)
+ }
+ }
+ }
+ }
+
test("Concat") {
// Primitive-type elements
val ai0 = Literal.create(Seq(1, 2, 3), ArrayType(IntegerType, containsNull
= false))
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
index deed423b71c5..66c3be203f5d 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
@@ -137,6 +137,27 @@ class ComplexTypeSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(GetArrayItem(stArray4, Literal(1)).nullable)
}
+ test("SPARK-60755: GetArrayItem on CreateArray generates code when the
result is not nullable") {
+ val a = BoundReference(0, StringType, nullable = false)
+ val b = BoundReference(1, StringType, nullable = true)
+ val input = create_row("x", null)
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ // The array can contain NULL, but the selected element cannot.
+ checkEvaluation(GetArrayItem(CreateArray(Seq(a, b)), Literal(0)), "x",
input)
+ // No element can be NULL.
+ checkEvaluation(GetArrayItem(CreateArray(Seq(a)), Literal(0)), "x",
input)
+ // The index is invalid in every row.
+ if (ansiEnabled) {
+ checkExceptionInExpression[Exception](
+ GetArrayItem(CreateArray(Seq(b)), Literal(5)), input, "out of
bounds")
+ } else {
+ checkEvaluation(GetArrayItem(CreateArray(Seq(b)), Literal(5)), null,
input)
+ }
+ }
+ }
+ }
+
Seq((Int.MaxValue, "Linear Lookup"), (0, "Hash Lookup")).foreach { case
(threshold, name) =>
test(s"GetMapValue - $name") {
withSQLConf(SQLConf.MAP_LOOKUP_HASH_THRESHOLD.key -> threshold.toString)
{
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
index 67ee0b98d27a..b688bf205ef9 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
@@ -23,7 +23,7 @@ import java.sql.{Date, Timestamp}
import scala.reflect.runtime.universe.runtimeMirror
import scala.util.Random
-import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT, SparkException,
SparkRuntimeException}
+import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT,
SparkArrayIndexOutOfBoundsException, SparkException, SparkRuntimeException}
import org.apache.spark.sql.catalyst.{ExtendedAnalysisException, InternalRow}
import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
import org.apache.spark.sql.catalyst.expressions.{ArrayExists, Expression,
Literal, UnaryExpression}
@@ -2547,6 +2547,37 @@ class DataFrameFunctionsSuite extends SharedSparkSession
{
}
}
+ test("SPARK-60755: element_at and array index compile when the result is not
nullable") {
+ // The first element cannot be NULL, but the array can contain NULL.
+ val df = spark.range(2).selectExpr("id", "if(id = 0, 'a', NULL) AS s")
+ val notNullable = "element_at(array(coalesce(s, 'z'), s), 1)"
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ checkAnswer(df.selectExpr(notNullable), Seq(Row("a"), Row("z")))
+ checkAnswer(df.where(s"$notNullable = 'z'").select("id"), Seq(Row(1)))
+ // The ordering of `TakeOrderedAndProjectExec` has no interpreted
fallback.
+ checkAnswer(df.orderBy(expr(notNullable).desc).limit(1).select("id"),
Seq(Row(1)))
+ }
+ }
+ // Under ANSI, an out-of-bounds index raises an error, so the result is
not nullable.
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ checkError(
+ exception = intercept[SparkArrayIndexOutOfBoundsException] {
+ df.where("element_at(array(s), 5) = 'x'").collect()
+ },
+ condition = "INVALID_ARRAY_INDEX_IN_ELEMENT_AT",
+ parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+ context = ExpectedContext(fragment = "element_at(array(s), 5)", start
= 0, stop = 22))
+ checkError(
+ exception = intercept[SparkArrayIndexOutOfBoundsException] {
+ df.selectExpr("array(s)[5]").collect()
+ },
+ condition = "INVALID_ARRAY_INDEX",
+ parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+ context = ExpectedContext(fragment = "array(s)[5]", start = 0, stop =
10))
+ }
+ }
+
test("element_at function") {
val df = Seq(
(Seq[String]("1", "2", "3"), 1),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]