This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 5ff0ef76b18c [SPARK-59589][4.3][SQL] Hand Expand's varying output
columns to the consumer as globals
5ff0ef76b18c is described below
commit 5ff0ef76b18c7b11634bd5ac718020e367bc7463
Author: YangJie <[email protected]>
AuthorDate: Fri Nov 6 10:54:18 2026 +0800
[SPARK-59589][4.3][SQL] Hand Expand's varying output columns to the
consumer as globals
### What changes were proposed in this pull request?
This is a backport of #58881 to `branch-4.3`.
`ExpandExec.doConsume` hands each output column that varies across its
projections to the consumer as `JavaCode.global` / `JavaCode.isNullGlobal`
instead of `JavaCode.variable` / `JavaCode.isNullVariable`. The value lives in
mutable state, a field or a slot of a compacted array such as
`expand_mutableStateArray_0[1]`, not in a local variable. See #58881 for the
full description.
Differences from the `master` commit:
- `branch-4.3` does not have SPARK-33301 or SPARK-59295, so there is no
`CompactedSlot` workaround to remove, no
`spark.sql.codegen.wholeStage.splitExpressions` conf, and no `collectInputs`.
This backport only changes `ExpandExec`.
- The testing check on method parameters (`assertMethodParameters`,
`checkParameterNames`) and its tests in `CollectInputsSuite` and
`WithExpressionEvalSuite` are not backported.
### Why are the changes needed?
A consumer that moves code into a method passes the local variables the
code reads as parameters. Taken for a local variable, the Expand's slot becomes
a parameter named `expand_mutableStateArray_0[1]`, which does not compile:
```sql
CREATE OR REPLACE TEMP VIEW t AS
SELECT concat('a', CAST(id AS STRING)) AS a, concat('b', CAST(id AS
STRING)) AS b FROM range(10);
SELECT c,
concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v),
initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')) AS x,
concat(concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v),
rtrim(v), initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')), 'z')
AS y
FROM t UNPIVOT (v FOR c IN (a, b));
```
Janino fails with `')' expected instead of '['`. The stage falls back to
running without whole-stage codegen, or the query fails with
`spark.sql.codegen.fallback=false`.
### Does this PR introduce _any_ user-facing change?
Yes. Such a stage now compiles under whole-stage codegen.
### How was this patch tested?
The `WholeStageCodegenSuite` tests from #58881, without the loop over
`spark.sql.codegen.wholeStage.splitExpressions` and without the case that
builds the methods from local variables. For the subexpression method (the
query above), the switch cases of the Expand that `RewriteDistinctAggregates`
adds, and the split aggregate functions, the method calls `lpad` on the slot,
the code compiles, and the rows match those computed without whole-stage
codegen. Without the change to `ExpandExe [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
Closes #60121 from LuciferYang/SPARK-59589-4.3.
Lead-authored-by: YangJie <[email protected]>
Co-authored-by: reviewer <reviewer@local>
Signed-off-by: yangjie01 <[email protected]>
---
.../apache/spark/sql/execution/ExpandExec.scala | 10 +++-
.../sql/execution/WholeStageCodegenSuite.scala | 68 ++++++++++++++++++++++
2 files changed, 76 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
index 268123606293..5944ba0977bf 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
@@ -166,9 +166,15 @@ case class ExpandExec(
"resultValue",
v => s"$v = ${CodeGenerator.defaultValue(firstExpr.dataType)};")
+ // `addMutableState` hands back a field: a name of its own, or a slot
in a compacted array
+ // (`mutableStateArray_0[3]`) for a state not inlined. Either is
reachable from every
+ // method of the generated class and is not a local variable, so a
consumer collecting
+ // local variables to pass into a split function must not take it for
one -- a slot
+ // expression is not a legal parameter name. Whole-stage subexpression
elimination wraps
+ // its own mutable state the same way.
ExprCode(
- JavaCode.isNullVariable(isNull),
- JavaCode.variable(value, firstExpr.dataType))
+ JavaCode.isNullGlobal(isNull),
+ JavaCode.global(value, firstExpr.dataType))
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
index 104a4b46b265..2808542e41d4 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
@@ -1561,6 +1561,19 @@ class WholeStageCodegenSuite extends SharedSparkSession
s"addExact appears $enabledCount times when enabled vs $disabledCount
times when disabled")
}
+ test("SPARK-59589: a subexpression method reads an Expand's varying output
as a field") {
+ // `ExpandExec` holds an UNPIVOT's string output in a slot of a compacted
mutable state array,
+ // such as `expand_mutableStateArray_0[1]`. A large common subexpression
reading it is computed
+ // in a method of its own, which takes the local variables it reads as
parameters. Taken for a
+ // local variable, the slot became a parameter `UTF8String
expand_mutableStateArray_0[1]` and
+ // the stage did not compile. Handed out as a field, it is read as one.
+ val f = "concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v),
rtrim(v), initcap(v), " +
+ "repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y'))"
+ val stringParameter = "private void \\w*subExpr_\\d+\\([^)]*UTF8String ".r
+ checkExpandSlotRead("subExpr", unpivot => s"SELECT c, $f AS x, concat($f,
'z') AS y $unpivot",
+ code => assert(code.forall(c =>
stringParameter.findFirstIn(c.body).isEmpty)))
+ }
+
test("SPARK-60398: Expand compiles with its switch case functions spilled to
nested classes") {
// A sliding window has its Expand generate a row for each window a time
can fall in: 360 rows
// for a window of an hour sliding by 10 seconds. Without subexpression
elimination the code
@@ -1591,6 +1604,61 @@ class WholeStageCodegenSuite extends SharedSparkSession
}
}
+ test("SPARK-59589: split switch cases and aggregate functions read an
Expand's output as a " +
+ "field") {
+ // With the consume functions not split by operator, the consumer reads
the UNPIVOT's output
+ // directly, and a tiny threshold moves its code into methods: the switch
cases of the Expand
+ // that `RewriteDistinctAggregates` adds, and the aggregate functions.
+ withSQLConf(
+ SQLConf.WHOLESTAGE_SPLIT_CONSUME_FUNC_BY_OPERATOR.key -> "false",
+ SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1") {
+ checkExpandSlotRead("switchCaseCode", unpivot =>
+ s"SELECT count(DISTINCT lpad(v, 10, 'x')), count(DISTINCT rpad(v, 10,
'y')) $unpivot")
+ checkExpandSlotRead("doAggregate_sum", unpivot =>
+ s"SELECT c, sum(length(lpad(v, 10, 'x'))), sum(length(rpad(v, 10,
'y'))) $unpivot " +
+ "GROUP BY c")
+ }
+ }
+
+ /**
+ * Checks the query `query` builds from a FROM clause that UNPIVOTs two
string columns into
+ * `v`: a method named like `method` calls `lpad` on the slot of the
compacted mutable state
+ * array that `ExpandExec` holds `v` in, `check` holds of the code, the code
compiles, and the
+ * rows match those computed without whole stage codegen.
+ */
+ private def checkExpandSlotRead(
+ method: String,
+ query: String => String,
+ check: Seq[CodeAndComment] => Unit = _ => ()): Unit = {
+ withTempView("t") {
+ spark.range(10).selectExpr("concat('a', CAST(id AS STRING)) AS a",
+ "concat('b', CAST(id AS STRING)) AS b").createOrReplaceTempView("t")
+ def df: DataFrame = sql(query("FROM t UNPIVOT (v FOR c IN (a, b))"))
+ val code = genCode(df)
+ // A subexpression method or a switch case also writes into a slot of
the same array,
+ // so the array's name alone does not show a read.
+ assert(code.exists(c => methodsHolding(c,
"].lpad(").exists(_.contains(method))), method)
+ withClue(s"$method: ")(check(code))
+ code.foreach(CodeGenerator.compile)
+ val expected = withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key ->
"false") {
+ df.collect().toSeq
+ }
+ checkAnswer(df, expected)
+ }
+ }
+
+ /**
+ * The methods of the generated class `code` whose code holds `text`, by
name, whose prefix
+ * says which operator they were generated for, such as
`expand_switchCaseCode_0`.
+ */
+ private def methodsHolding(code: CodeAndComment, text: String): Set[String]
= {
+ val header =
"(?m)^\\s*(?:private|public|protected)\\s[^{}();=]*?([A-Za-z_$][\\w$]*)\\s*\\(".r
+ val starts = header.findAllMatchIn(code.body).map(m => (m.start,
m.group(1))).toSeq
+ starts.zip(starts.drop(1).map(_._1) :+ code.body.length).collect {
+ case ((start, name), end) if code.body.substring(start,
end).contains(text) => name
+ }.toSet
+ }
+
test("SPARK-60403: full outer joins compile with their functions spilled to
nested classes") {
// A full outer join outputs its rows through a function of its own, and
generates the code of
// the operators above it, for the body of that function, before adding
the function: here
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]