This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new ddec9b81b090 [SPARK-59589][4.1][SQL] Hand Expand's varying output
columns to the consumer as globals
ddec9b81b090 is described below
commit ddec9b81b0904940acf2b0bfe5cef45b7a00fd1f
Author: YangJie <[email protected]>
AuthorDate: Fri Nov 6 20:57:39 2026 +0800
[SPARK-59589][4.1][SQL] Hand Expand's varying output columns to the
consumer as globals
### What changes were proposed in this pull request?
This is a backport of #58881 to `branch-4.1`, the same change as #60121 for
`branch-4.3`.
`ExpandExec.doConsume` hands each output column that varies across its
projections to the consumer as `JavaCode.global` / `JavaCode.isNullGlobal`
instead of `JavaCode.variable` / `JavaCode.isNullVariable`. The value lives in
mutable state, a field or a slot of a compacted array such as
`expand_mutableStateArray_0[1]`, not in a local variable. See #58881 for the
full description.
Differences from the `master` commit:
- `branch-4.1` does not have SPARK-33301 or SPARK-59295, so there is no
`CompactedSlot` workaround to remove, no
`spark.sql.codegen.wholeStage.splitExpressions` conf, and no `collectInputs`.
This backport only changes `ExpandExec`.
- The testing check on method parameters (`assertMethodParameters`,
`checkParameterNames`) and its tests in `CollectInputsSuite` and
`WithExpressionEvalSuite` are not backported.
### Why are the changes needed?
A consumer that moves code into a method passes the local variables the
code reads as parameters. Taken for a local variable, the Expand's slot becomes
a parameter named `expand_mutableStateArray_0[1]`, which does not compile:
```sql
CREATE OR REPLACE TEMP VIEW t AS
SELECT concat('a', CAST(id AS STRING)) AS a, concat('b', CAST(id AS
STRING)) AS b FROM range(10);
SELECT c,
concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v),
initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')) AS x,
concat(concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v),
rtrim(v), initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')), 'z')
AS y
FROM t UNPIVOT (v FOR c IN (a, b));
```
Janino fails with `')' expected instead of '['`. The stage falls back to
running without whole-stage codegen, or the query fails with
`spark.sql.codegen.fallback=false`.
### Does this PR introduce _any_ user-facing change?
Yes. Such a stage now compiles under whole-stage codegen.
### How was this patch tested?
The `WholeStageCodegenSuite` tests from #58881, without the loop over
`spark.sql.codegen.wholeStage.splitExpressions` and without the case that
builds the methods from local variables. For the subexpression method (the
query above), the switch cases of the Expand that `RewriteDistinctAggregates`
adds, and the split aggregate functions, the method calls `lpad` on the slot,
the code compiles, and the rows match those computed without whole-stage
codegen. Without the change to `ExpandExe [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Closes #60159 from LuciferYang/SPARK-59589-4.1.
Lead-authored-by: YangJie <[email protected]>
Co-authored-by: reviewer <reviewer@local>
Signed-off-by: yangjie01 <[email protected]>
---
.../apache/spark/sql/execution/ExpandExec.scala | 10 +++-
.../sql/execution/WholeStageCodegenSuite.scala | 70 +++++++++++++++++++++-
2 files changed, 77 insertions(+), 3 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
index 1b9432047d9d..bf4670635848 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
@@ -162,9 +162,15 @@ case class ExpandExec(
"resultValue",
v => s"$v = ${CodeGenerator.defaultValue(firstExpr.dataType)};")
+ // `addMutableState` hands back a field: a name of its own, or a slot
in a compacted array
+ // (`mutableStateArray_0[3]`) for a state not inlined. Either is
reachable from every
+ // method of the generated class and is not a local variable, so a
consumer collecting
+ // local variables to pass into a split function must not take it for
one -- a slot
+ // expression is not a legal parameter name. Whole-stage subexpression
elimination wraps
+ // its own mutable state the same way.
ExprCode(
- JavaCode.isNullVariable(isNull),
- JavaCode.variable(value, firstExpr.dataType))
+ JavaCode.isNullGlobal(isNull),
+ JavaCode.global(value, firstExpr.dataType))
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
index a3cfdc5a240a..4266db979585 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
@@ -19,7 +19,7 @@ package org.apache.spark.sql.execution
import org.apache.spark.SparkException
import org.apache.spark.rdd.MapPartitionsWithEvaluatorRDD
-import org.apache.spark.sql.{Dataset, QueryTest, Row, SaveMode}
+import org.apache.spark.sql.{DataFrame, Dataset, QueryTest, Row, SaveMode}
import org.apache.spark.sql.catalyst.expressions.CodegenObjectFactoryMode
import org.apache.spark.sql.catalyst.expressions.codegen.{ByteCodeStats,
CodeAndComment, CodeGenerator}
import org.apache.spark.sql.execution.adaptive.DisableAdaptiveExecutionSuite
@@ -944,4 +944,72 @@ class WholeStageCodegenSuite extends QueryTest with
SharedSparkSession
}
}
}
+
+ test("SPARK-59589: a subexpression method reads an Expand's varying output
as a field") {
+ // `ExpandExec` holds an UNPIVOT's string output in a slot of a compacted
mutable state array,
+ // such as `expand_mutableStateArray_0[1]`. A large common subexpression
reading it is computed
+ // in a method of its own, which takes the local variables it reads as
parameters. Taken for a
+ // local variable, the slot became a parameter `UTF8String
expand_mutableStateArray_0[1]` and
+ // the stage did not compile. Handed out as a field, it is read as one.
+ val f = "concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v),
rtrim(v), initcap(v), " +
+ "repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y'))"
+ val stringParameter = "private void \\w*subExpr_\\d+\\([^)]*UTF8String ".r
+ checkExpandSlotRead("subExpr", unpivot => s"SELECT c, $f AS x, concat($f,
'z') AS y $unpivot",
+ code => assert(code.forall(c =>
stringParameter.findFirstIn(c.body).isEmpty)))
+ }
+
+ test("SPARK-59589: split switch cases and aggregate functions read an
Expand's output as a " +
+ "field") {
+ // With the consume functions not split by operator, the consumer reads
the UNPIVOT's output
+ // directly, and a tiny threshold moves its code into methods: the switch
cases of the Expand
+ // that `RewriteDistinctAggregates` adds, and the aggregate functions.
+ withSQLConf(
+ SQLConf.WHOLESTAGE_SPLIT_CONSUME_FUNC_BY_OPERATOR.key -> "false",
+ SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1") {
+ checkExpandSlotRead("switchCaseCode", unpivot =>
+ s"SELECT count(DISTINCT lpad(v, 10, 'x')), count(DISTINCT rpad(v, 10,
'y')) $unpivot")
+ checkExpandSlotRead("doAggregate_sum", unpivot =>
+ s"SELECT c, sum(length(lpad(v, 10, 'x'))), sum(length(rpad(v, 10,
'y'))) $unpivot " +
+ "GROUP BY c")
+ }
+ }
+
+ /**
+ * Checks the query `query` builds from a FROM clause that UNPIVOTs two
string columns into
+ * `v`: a method named like `method` calls `lpad` on the slot of the
compacted mutable state
+ * array that `ExpandExec` holds `v` in, `check` holds of the code, the code
compiles, and the
+ * rows match those computed without whole stage codegen.
+ */
+ private def checkExpandSlotRead(
+ method: String,
+ query: String => String,
+ check: Seq[CodeAndComment] => Unit = _ => ()): Unit = {
+ withTempView("t") {
+ spark.range(10).selectExpr("concat('a', CAST(id AS STRING)) AS a",
+ "concat('b', CAST(id AS STRING)) AS b").createOrReplaceTempView("t")
+ def df: DataFrame = sql(query("FROM t UNPIVOT (v FOR c IN (a, b))"))
+ val code = genCode(df)
+ // A subexpression method or a switch case also writes into a slot of
the same array,
+ // so the array's name alone does not show a read.
+ assert(code.exists(c => methodsHolding(c,
"].lpad(").exists(_.contains(method))), method)
+ withClue(s"$method: ")(check(code))
+ code.foreach(CodeGenerator.compile)
+ val expected = withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key ->
"false") {
+ df.collect().toSeq
+ }
+ checkAnswer(df, expected)
+ }
+ }
+
+ /**
+ * The methods of the generated class `code` whose code holds `text`, by
name, whose prefix
+ * says which operator they were generated for, such as
`expand_switchCaseCode_0`.
+ */
+ private def methodsHolding(code: CodeAndComment, text: String): Set[String]
= {
+ val header =
"(?m)^\\s*(?:private|public|protected)\\s[^{}();=]*?([A-Za-z_$][\\w$]*)\\s*\\(".r
+ val starts = header.findAllMatchIn(code.body).map(m => (m.start,
m.group(1))).toSeq
+ starts.zip(starts.drop(1).map(_._1) :+ code.body.length).collect {
+ case ((start, name), end) if code.body.substring(start,
end).contains(text) => name
+ }.toSet
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]