This is an automated email from the ASF dual-hosted git repository.

LuciferYang pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new 5ff0ef76b18c [SPARK-59589][4.3][SQL] Hand Expand's varying output 
columns to the consumer as globals
5ff0ef76b18c is described below

commit 5ff0ef76b18c7b11634bd5ac718020e367bc7463
Author: YangJie <[email protected]>
AuthorDate: Fri Nov 6 10:54:18 2026 +0800

    [SPARK-59589][4.3][SQL] Hand Expand's varying output columns to the 
consumer as globals
    
    ### What changes were proposed in this pull request?
    
    This is a backport of #58881 to `branch-4.3`.
    
    `ExpandExec.doConsume` hands each output column that varies across its 
projections to the consumer as `JavaCode.global` / `JavaCode.isNullGlobal` 
instead of `JavaCode.variable` / `JavaCode.isNullVariable`. The value lives in 
mutable state, a field or a slot of a compacted array such as 
`expand_mutableStateArray_0[1]`, not in a local variable. See #58881 for the 
full description.
    
    Differences from the `master` commit:
    - `branch-4.3` does not have SPARK-33301 or SPARK-59295, so there is no 
`CompactedSlot` workaround to remove, no 
`spark.sql.codegen.wholeStage.splitExpressions` conf, and no `collectInputs`. 
This backport only changes `ExpandExec`.
    - The testing check on method parameters (`assertMethodParameters`, 
`checkParameterNames`) and its tests in `CollectInputsSuite` and 
`WithExpressionEvalSuite` are not backported.
    
    ### Why are the changes needed?
    
    A consumer that moves code into a method passes the local variables the 
code reads as parameters. Taken for a local variable, the Expand's slot becomes 
a parameter named `expand_mutableStateArray_0[1]`, which does not compile:
    
    ```sql
    CREATE OR REPLACE TEMP VIEW t AS
      SELECT concat('a', CAST(id AS STRING)) AS a, concat('b', CAST(id AS 
STRING)) AS b FROM range(10);
    SELECT c,
      concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v), 
initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')) AS x,
      concat(concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), 
rtrim(v), initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')), 'z') 
AS y
    FROM t UNPIVOT (v FOR c IN (a, b));
    ```
    
    Janino fails with `')' expected instead of '['`. The stage falls back to 
running without whole-stage codegen, or the query fails with 
`spark.sql.codegen.fallback=false`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Such a stage now compiles under whole-stage codegen.
    
    ### How was this patch tested?
    
    The `WholeStageCodegenSuite` tests from #58881, without the loop over 
`spark.sql.codegen.wholeStage.splitExpressions` and without the case that 
builds the methods from local variables. For the subexpression method (the 
query above), the switch cases of the Expand that `RewriteDistinctAggregates` 
adds, and the split aggregate functions, the method calls `lpad` on the slot, 
the code compiles, and the rows match those computed without whole-stage 
codegen. Without the change to `ExpandExe [...]
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5
    
    Closes #60121 from LuciferYang/SPARK-59589-4.3.
    
    Lead-authored-by: YangJie <[email protected]>
    Co-authored-by: reviewer <reviewer@local>
    Signed-off-by: yangjie01 <[email protected]>
---
 .../apache/spark/sql/execution/ExpandExec.scala    | 10 +++-
 .../sql/execution/WholeStageCodegenSuite.scala     | 68 ++++++++++++++++++++++
 2 files changed, 76 insertions(+), 2 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
index 268123606293..5944ba0977bf 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
@@ -166,9 +166,15 @@ case class ExpandExec(
           "resultValue",
           v => s"$v = ${CodeGenerator.defaultValue(firstExpr.dataType)};")
 
+        // `addMutableState` hands back a field: a name of its own, or a slot 
in a compacted array
+        // (`mutableStateArray_0[3]`) for a state not inlined. Either is 
reachable from every
+        // method of the generated class and is not a local variable, so a 
consumer collecting
+        // local variables to pass into a split function must not take it for 
one -- a slot
+        // expression is not a legal parameter name. Whole-stage subexpression 
elimination wraps
+        // its own mutable state the same way.
         ExprCode(
-          JavaCode.isNullVariable(isNull),
-          JavaCode.variable(value, firstExpr.dataType))
+          JavaCode.isNullGlobal(isNull),
+          JavaCode.global(value, firstExpr.dataType))
       }
     }
 
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
index 104a4b46b265..2808542e41d4 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
@@ -1561,6 +1561,19 @@ class WholeStageCodegenSuite extends SharedSparkSession
         s"addExact appears $enabledCount times when enabled vs $disabledCount 
times when disabled")
   }
 
+  test("SPARK-59589: a subexpression method reads an Expand's varying output 
as a field") {
+    // `ExpandExec` holds an UNPIVOT's string output in a slot of a compacted 
mutable state array,
+    // such as `expand_mutableStateArray_0[1]`. A large common subexpression 
reading it is computed
+    // in a method of its own, which takes the local variables it reads as 
parameters. Taken for a
+    // local variable, the slot became a parameter `UTF8String 
expand_mutableStateArray_0[1]` and
+    // the stage did not compile. Handed out as a field, it is read as one.
+    val f = "concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), 
rtrim(v), initcap(v), " +
+      "repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y'))"
+    val stringParameter = "private void \\w*subExpr_\\d+\\([^)]*UTF8String ".r
+    checkExpandSlotRead("subExpr", unpivot => s"SELECT c, $f AS x, concat($f, 
'z') AS y $unpivot",
+      code => assert(code.forall(c => 
stringParameter.findFirstIn(c.body).isEmpty)))
+  }
+
   test("SPARK-60398: Expand compiles with its switch case functions spilled to 
nested classes") {
     // A sliding window has its Expand generate a row for each window a time 
can fall in: 360 rows
     // for a window of an hour sliding by 10 seconds. Without subexpression 
elimination the code
@@ -1591,6 +1604,61 @@ class WholeStageCodegenSuite extends SharedSparkSession
     }
   }
 
+  test("SPARK-59589: split switch cases and aggregate functions read an 
Expand's output as a " +
+      "field") {
+    // With the consume functions not split by operator, the consumer reads 
the UNPIVOT's output
+    // directly, and a tiny threshold moves its code into methods: the switch 
cases of the Expand
+    // that `RewriteDistinctAggregates` adds, and the aggregate functions.
+    withSQLConf(
+        SQLConf.WHOLESTAGE_SPLIT_CONSUME_FUNC_BY_OPERATOR.key -> "false",
+        SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1") {
+      checkExpandSlotRead("switchCaseCode", unpivot =>
+        s"SELECT count(DISTINCT lpad(v, 10, 'x')), count(DISTINCT rpad(v, 10, 
'y')) $unpivot")
+      checkExpandSlotRead("doAggregate_sum", unpivot =>
+        s"SELECT c, sum(length(lpad(v, 10, 'x'))), sum(length(rpad(v, 10, 
'y'))) $unpivot " +
+          "GROUP BY c")
+    }
+  }
+
+  /**
+   * Checks the query `query` builds from a FROM clause that UNPIVOTs two 
string columns into
+   * `v`: a method named like `method` calls `lpad` on the slot of the 
compacted mutable state
+   * array that `ExpandExec` holds `v` in, `check` holds of the code, the code 
compiles, and the
+   * rows match those computed without whole stage codegen.
+   */
+  private def checkExpandSlotRead(
+      method: String,
+      query: String => String,
+      check: Seq[CodeAndComment] => Unit = _ => ()): Unit = {
+    withTempView("t") {
+      spark.range(10).selectExpr("concat('a', CAST(id AS STRING)) AS a",
+        "concat('b', CAST(id AS STRING)) AS b").createOrReplaceTempView("t")
+      def df: DataFrame = sql(query("FROM t UNPIVOT (v FOR c IN (a, b))"))
+      val code = genCode(df)
+      // A subexpression method or a switch case also writes into a slot of 
the same array,
+      // so the array's name alone does not show a read.
+      assert(code.exists(c => methodsHolding(c, 
"].lpad(").exists(_.contains(method))), method)
+      withClue(s"$method: ")(check(code))
+      code.foreach(CodeGenerator.compile)
+      val expected = withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> 
"false") {
+        df.collect().toSeq
+      }
+      checkAnswer(df, expected)
+    }
+  }
+
+  /**
+   * The methods of the generated class `code` whose code holds `text`, by 
name, whose prefix
+   * says which operator they were generated for, such as 
`expand_switchCaseCode_0`.
+   */
+  private def methodsHolding(code: CodeAndComment, text: String): Set[String] 
= {
+    val header = 
"(?m)^\\s*(?:private|public|protected)\\s[^{}();=]*?([A-Za-z_$][\\w$]*)\\s*\\(".r
+    val starts = header.findAllMatchIn(code.body).map(m => (m.start, 
m.group(1))).toSeq
+    starts.zip(starts.drop(1).map(_._1) :+ code.body.length).collect {
+      case ((start, name), end) if code.body.substring(start, 
end).contains(text) => name
+    }.toSet
+  }
+
   test("SPARK-60403: full outer joins compile with their functions spilled to 
nested classes") {
     // A full outer join outputs its rows through a function of its own, and 
generates the code of
     // the operators above it, for the body of that function, before adding 
the function: here


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to