This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 4452bd79f47f [SPARK-60755][SQL] Fix codegen compilation failure of
`element_at` and array index when the result is not nullable
4452bd79f47f is described below
commit 4452bd79f47f1be8106ba87b50416ee1b558fbaa
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Oct 31 01:12:58 2026 -0700
[SPARK-60755][SQL] Fix codegen compilation failure of `element_at` and
array index when the result is not nullable
This PR aims to fix the generated code of `ElementAt` and `GetArrayItem` to
assign `isNull` only when the expression is nullable.
`BinaryExpression.nullSafeCodeGen` declares `ev.isNull` only for a nullable
expression. When the child is `CreateArray`, or `GetArrayStructFields` over it,
and the index is a literal, `ElementAt` and `GetArrayItem` can be non-nullable
because:
- the index selects an element that cannot be NULL,
- the index is out of bounds under ANSI mode, which raises an error, or
- the index of `element_at` is 0, which raises an error.
However, `doGenCode` generates `isNull = true` regardless of `nullable`
when the array can contain NULL, and in the non-ANSI out-of-bounds branch. So,
the generated Java code assigns to an undeclared variable and fails to compile.
```java
UTF8String filter_value_1 = null;
int filter_elementAtIndex_0 =
org.apache.spark.sql.catalyst.expressions.ArrayExpressionUtils.resolveArrayIndex(
filter_arrayData_0.numElements(), (int) 5,
((org.apache.spark.sql.catalyst.trees.SQLQueryContext) references[1] /* errCtx
*/));
if (filter_arrayData_0.isNullAt(filter_elementAtIndex_0)) {
filter_isNull_1 = true; // This variable is not declared.
} else {
filter_value_1 =
filter_arrayData_0.getUTF8String(filter_elementAtIndex_0);
}
```
The nullability is correct and this PR does not change it. When the
expression is not nullable, the removed assignments are unreachable.
The code generation of `element_at` has failed since Apache Spark
[3.0.0](https://github.com/apache/spark/releases/tag/v3.0.0) (2020-06-06) for a
selected element that cannot be NULL (SPARK-26965), and since Apache Spark
[3.1.1](https://github.com/apache/spark/releases/tag/v3.1.1) (2021-02-22) for
an out-of-bounds index under ANSI mode (SPARK-33386).
The array index has the same defect, but `SimplifyExtractValueOps` removes
`GetArrayItem` over `CreateArray` for most `INT` literal indexes. So, it fails
with a literal index of the other integral types such as `0L`, and since Apache
Spark [4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27)
with an out-of-bounds `INT` literal index under ANSI mode (SPARK-58431).
Most operators fall back to the interpreted evaluation, which disables
whole-stage codegen and logs a compilation error per task. The ordering of
`TakeOrderedAndProjectExec` and the predicates of `MergeRowsExec` have no such
fallback, so `ORDER BY ... LIMIT` and `MERGE INTO` queries fail.
Yes. Queries that failed with `CompileException` now succeed. For example:
```sql
SELECT id FROM range(3) ORDER BY element_at(array(id, nullif(id, 0)), 1)
DESC LIMIT 1;
```
**BEFORE** (Apache Spark
[4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27))
```
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
51, Column 1: Expression "isNull_0" is not an rvalue
```
**AFTER**
```
2
```
Queries that used the fallback return the same results, without the
compilation errors in the logs.
Pass the CIs with the newly added test cases. Each of them fails without
this fix.
```
$ build/sbt 'catalyst/testOnly *CollectionExpressionsSuite
*ComplexTypeSuite -- -z "generates code when the result is not nullable"'
$ build/sbt 'sql/testOnly *DataFrameFunctionsSuite -- -z "compile when the
result is not nullable"'
```
Generated-by: Claude Opus 5.5
Closes #59963 from dongjoon-hyun/SPARK-60755.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 075749d08a3bf0c3f407600d0b5d5ab790321306)
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit e8bea5e8de16fa9709c0c029c51058a65629f357)
---
.../expressions/collectionOperations.scala | 10 +++++--
.../expressions/complexTypeExtractors.scala | 12 ++++++--
.../expressions/CollectionExpressionsSuite.scala | 30 ++++++++++++++++++++
.../catalyst/expressions/ComplexTypeSuite.scala | 21 ++++++++++++++
.../apache/spark/sql/DataFrameFunctionsSuite.scala | 33 +++++++++++++++++++++-
5 files changed, 100 insertions(+), 6 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
index fedea7cda76d..9bbf8130bda5 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala
@@ -2983,7 +2983,9 @@ case class ElementAt(
val errorContext = getContextOrNullCode(ctx)
val utils = classOf[ArrayExpressionUtils].getName
val assignment = s"${ev.value} = ${CodeGenerator.getValue(eval1,
dataType, index)};"
- val body = if (arrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise
the index raises
+ // an error or selects an element that cannot be NULL.
+ val body = if (nullable && arrayElementNullable) {
s"""
|if ($eval1.isNullAt($index)) {
| ${ev.isNull} = true;
@@ -3004,7 +3006,10 @@ case class ElementAt(
nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
val index = ctx.freshName("elementAtIndex")
val intIndex = ctx.freshName("elementAtIntIndex")
- val nullCheck = if (arrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise
the index is 0, or
+ // selects an element that cannot be NULL, so neither this branch
nor the out-of-bounds
+ // branch below is taken.
+ val nullCheck = if (nullable && arrayElementNullable) {
s"""
|if ($eval1.isNullAt($intIndex)) {
| ${ev.isNull} = true;
@@ -3015,6 +3020,7 @@ case class ElementAt(
}
val errorContext = getContextOrNullCode(ctx)
val indexOutOfBoundBranch = defaultValueOutOfBound match {
+ case _ if !nullable => ""
case Some(value) =>
val defaultValueEval = value.genCode(ctx)
s"""
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
index 2f9c1e2dd704..5cba61088da6 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/complexTypeExtractors.scala
@@ -404,7 +404,9 @@ case class GetArrayItem(
val utils = classOf[ArrayExpressionUtils].getName
val childArrayElementNullable =
child.dataType.asInstanceOf[ArrayType].containsNull
val assignment = s"${ev.value} = ${CodeGenerator.getValue(eval1,
dataType, index)};"
- val body = if (childArrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise
the index raises an
+ // error or selects an element that cannot be NULL.
+ val body = if (nullable && childArrayElementNullable) {
s"""
|if ($eval1.isNullAt($index)) {
| ${ev.isNull} = true;
@@ -424,7 +426,10 @@ case class GetArrayItem(
nullSafeCodeGen(ctx, ev, (eval1, eval2) => {
val index = ctx.freshName("index")
val childArrayElementNullable =
child.dataType.asInstanceOf[ArrayType].containsNull
- val nullCheck = if (childArrayElementNullable) {
+ // `ev.isNull` is declared only for a nullable expression. Otherwise
the index selects an
+ // element that cannot be NULL, so neither this branch nor the
out-of-bounds branch below
+ // is taken.
+ val nullCheck = if (nullable && childArrayElementNullable) {
s"""else if ($eval1.isNullAt($index)) {
${ev.isNull} = true;
}
@@ -432,10 +437,11 @@ case class GetArrayItem(
} else {
""
}
+ val outOfBound = if (nullable) s"${ev.isNull} = true;" else ""
s"""
final int $index = (int) $eval2;
if ($index >= $eval1.numElements() || $index < 0) {
- ${ev.isNull} = true;
+ $outOfBound
} $nullCheck else {
${ev.value} = ${CodeGenerator.getValue(eval1, dataType, index)};
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
index 44bbff9214be..d98060cd98c9 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CollectionExpressionsSuite.scala
@@ -2342,6 +2342,36 @@ class CollectionExpressionsSuite
}
}
+ test("SPARK-60755: ElementAt on CreateArray generates code when the result
is not nullable") {
+ val a = BoundReference(0, StringType, nullable = false)
+ val b = BoundReference(1, StringType, nullable = true)
+ val input = create_row("x", null)
+ val field = StructField("f", StringType, nullable = false)
+ val c = BoundReference(0, StructType(Seq(field)), nullable = false)
+ val d = BoundReference(1, StructType(Seq(field)), nullable = true)
+ val inputWithStruct = create_row(create_row("x"), null)
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ // The array can contain NULL, but the selected element cannot.
+ checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(1)), "x",
input)
+ checkEvaluation(ElementAt(CreateArray(Seq(a, b)), Literal(-2)), "x",
input)
+ val fields = GetArrayStructFields(CreateArray(Seq(c, d)), field, 0, 1,
containsNull = true)
+ checkEvaluation(ElementAt(fields, Literal(1)), "x", inputWithStruct)
+ // No element can be NULL.
+ checkEvaluation(ElementAt(CreateArray(Seq(a)), Literal(1)), "x", input)
+ // The index is invalid in every row.
+ checkExceptionInExpression[Exception](
+ ElementAt(CreateArray(Seq(b)), Literal(0)), input, "The index 0 is
invalid")
+ if (ansiEnabled) {
+ checkExceptionInExpression[Exception](
+ ElementAt(CreateArray(Seq(b)), Literal(5)), input, "out of bounds")
+ } else {
+ checkEvaluation(ElementAt(CreateArray(Seq(b)), Literal(5)), null,
input)
+ }
+ }
+ }
+ }
+
test("Concat") {
// Primitive-type elements
val ai0 = Literal.create(Seq(1, 2, 3), ArrayType(IntegerType, containsNull
= false))
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
index abc28ff203f8..cc91478a0fbf 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ComplexTypeSuite.scala
@@ -129,6 +129,27 @@ class ComplexTypeSuite extends SparkFunSuite with
ExpressionEvalHelper {
assert(GetArrayItem(stArray4, Literal(1)).nullable)
}
+ test("SPARK-60755: GetArrayItem on CreateArray generates code when the
result is not nullable") {
+ val a = BoundReference(0, StringType, nullable = false)
+ val b = BoundReference(1, StringType, nullable = true)
+ val input = create_row("x", null)
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ // The array can contain NULL, but the selected element cannot.
+ checkEvaluation(GetArrayItem(CreateArray(Seq(a, b)), Literal(0)), "x",
input)
+ // No element can be NULL.
+ checkEvaluation(GetArrayItem(CreateArray(Seq(a)), Literal(0)), "x",
input)
+ // The index is invalid in every row.
+ if (ansiEnabled) {
+ checkExceptionInExpression[Exception](
+ GetArrayItem(CreateArray(Seq(b)), Literal(5)), input, "out of
bounds")
+ } else {
+ checkEvaluation(GetArrayItem(CreateArray(Seq(b)), Literal(5)), null,
input)
+ }
+ }
+ }
+ }
+
Seq((Int.MaxValue, "Linear Lookup"), (0, "Hash Lookup")).foreach { case
(threshold, name) =>
test(s"GetMapValue - $name") {
withSQLConf(SQLConf.MAP_LOOKUP_HASH_THRESHOLD.key -> threshold.toString)
{
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
index d46aebeba8c7..f5ce6a716a6a 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala
@@ -23,7 +23,7 @@ import java.sql.{Date, Timestamp}
import scala.reflect.runtime.universe.runtimeMirror
import scala.util.Random
-import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT, SparkException,
SparkRuntimeException}
+import org.apache.spark.{QueryContextType, SPARK_DOC_ROOT,
SparkArrayIndexOutOfBoundsException, SparkException, SparkRuntimeException}
import org.apache.spark.sql.catalyst.{ExtendedAnalysisException, InternalRow}
import org.apache.spark.sql.catalyst.analysis.FunctionRegistry
import org.apache.spark.sql.catalyst.expressions.{ArrayExists, Expression,
Literal, UnaryExpression}
@@ -2515,6 +2515,37 @@ class DataFrameFunctionsSuite extends SharedSparkSession
{
)
}
+ test("SPARK-60755: element_at and array index compile when the result is not
nullable") {
+ // The first element cannot be NULL, but the array can contain NULL.
+ val df = spark.range(2).selectExpr("id", "if(id = 0, 'a', NULL) AS s")
+ val notNullable = "element_at(array(coalesce(s, 'z'), s), 1)"
+ Seq(true, false).foreach { ansiEnabled =>
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> ansiEnabled.toString) {
+ checkAnswer(df.selectExpr(notNullable), Seq(Row("a"), Row("z")))
+ checkAnswer(df.where(s"$notNullable = 'z'").select("id"), Seq(Row(1)))
+ // The ordering of `TakeOrderedAndProjectExec` has no interpreted
fallback.
+ checkAnswer(df.orderBy(expr(notNullable).desc).limit(1).select("id"),
Seq(Row(1)))
+ }
+ }
+ // Under ANSI, an out-of-bounds index raises an error, so the result is
not nullable.
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ checkError(
+ exception = intercept[SparkArrayIndexOutOfBoundsException] {
+ df.where("element_at(array(s), 5) = 'x'").collect()
+ },
+ condition = "INVALID_ARRAY_INDEX_IN_ELEMENT_AT",
+ parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+ context = ExpectedContext(fragment = "element_at(array(s), 5)", start
= 0, stop = 22))
+ checkError(
+ exception = intercept[SparkArrayIndexOutOfBoundsException] {
+ df.selectExpr("array(s)[5]").collect()
+ },
+ condition = "INVALID_ARRAY_INDEX",
+ parameters = Map("indexValue" -> "5", "arraySize" -> "1"),
+ context = ExpectedContext(fragment = "array(s)[5]", start = 0, stop =
10))
+ }
+ }
+
test("element_at function") {
val df = Seq(
(Seq[String]("1", "2", "3"), 1),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]