This is an automated email from the ASF dual-hosted git repository.

LuciferYang pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 862472cac1d2 [SPARK-59589][SQL] Hand Expand's varying output columns 
to the consumer as globals
862472cac1d2 is described below

commit 862472cac1d21050f145ca279416aeb182610bb2
Author: YangJie <[email protected]>
AuthorDate: Thu Nov 5 13:00:21 2026 +0800

    [SPARK-59589][SQL] Hand Expand's varying output columns to the consumer as 
globals
    
    ### What changes were proposed in this pull request?
    
    `ExpandExec.doConsume` keeps each output column that varies across its 
projections in mutable state, and hands it to the consumer as 
`JavaCode.variable` / `JavaCode.isNullVariable`. This PR hands it out as 
`JavaCode.global` / `JavaCode.isNullGlobal` instead.
    
    `addMutableState` returns either a field or a slot of a compacted array, 
such as `expand_mutableStateArray_0[1]`. Neither is a local variable, and 
whole-stage subexpression elimination already labels its own states as globals. 
Both labels print the same text, so the generated code changes only where a 
consumer checks whether a value is a local variable.
    
    With the column labelled correctly, the SPARK-33301 workaround in 
`CodegenContext.collectInputs`, which reads a `VariableValue` shaped like a 
slot as a field when `spark.sql.codegen.wholeStage.splitExpressions` is on, is 
no longer needed. This PR removes it (`CodeGenerator.CompactedSlot`, 
`isCompactedSlot`) and its mentions in the docs of the conf and of 
`WholeStageCodegenExec`.
    
    Under testing, a method parameter that is not a Java name now fails an 
assertion, in `collectInputs` and where the subexpression, aggregate and 
switch-case methods are built. So the next producer that hands out a slot as a 
`VariableValue`, or sets `INPUT_ROW` to a slot, fails the tests. The check is 
off outside tests, and does not cover the row a whole-stage split appends to 
its arguments yet (SPARK-60510).
    
    ### Why are the changes needed?
    
    A consumer that moves code into a method passes the local variables the 
code reads as parameters. Taken for a local variable, the Expand's slot becomes 
a parameter named `expand_mutableStateArray_0[1]`, which does not compile. An 
UNPIVOT of string columns with a large common subexpression hits this:
    
    ```sql
    CREATE OR REPLACE TEMP VIEW t AS
      SELECT concat('a', CAST(id AS STRING)) AS a, concat('b', CAST(id AS 
STRING)) AS b FROM range(10);
    SELECT c,
      concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v), 
initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')) AS x,
      concat(concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), 
rtrim(v), initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')), 'z') 
AS y
    FROM t UNPIVOT (v FOR c IN (a, b));
    ```
    
    The `subExpr` method declares `UTF8String expand_mutableStateArray_0[1]`, 
and Janino fails with `')' expected instead of '['`. The stage falls back to 
running without whole-stage codegen, or the query fails with 
`spark.sql.codegen.fallback=false`. On master the SPARK-33301 workaround makes 
it compile while `spark.sql.codegen.wholeStage.splitExpressions` is on, the 
default, and it still fails with the conf off. dongjoon-hyun reproduced it on 
4.1.3, 4.2.0 and 4.3.0, which have no such c [...]
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Such a stage now compiles under whole-stage codegen. Before, it did 
not on 4.1.3, 4.2.0 and 4.3.0, nor on master with 
`spark.sql.codegen.wholeStage.splitExpressions` off.
    
    ### How was this patch tested?
    
    - `WholeStageCodegenSuite`: one test for each method that took the slot as 
a parameter, which are the subexpression method (the query above), the switch 
cases of the Expand that `RewriteDistinctAggregates` adds, and the split 
aggregate functions. With `spark.sql.codegen.wholeStage.splitExpressions` on 
and off, the method calls `lpad` on the slot, the code compiles, and the rows 
match those computed without whole-stage codegen. With a test operator that 
hands out the Expand's output as [...]
    - `CollectInputsSuite`: the two tests for the removed slot rule are 
replaced. A slot handed out as a `GlobalValue` is never an argument, and an 
inlined field handed out as a `VariableValue` is left out of the arguments only 
by a whole-stage split method. A slot handed out as a `VariableValue`, or an 
`INPUT_ROW` that is a slot, fails the check; with the check off, as in 
production, the policies that ask for names refuse the method.
    - `WithExpressionEvalSuite`: a `With` whose input is such a slot fails the 
check; with the check off, its definition stays inline at each reference.
    - Removed: the SPARK-33301 test asserting that a `subExpr` method takes a 
`UTF8String` parameter with the conf off, which this PR makes impossible (the 
new subexpression test runs the same query), and a parameter check in the 
remaining SPARK-33301 CASE WHEN test that could not fail on its own.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5
    
    Closes #58881 from LuciferYang/SPARK-59589.
    
    Lead-authored-by: YangJie <[email protected]>
    Co-authored-by: reviewer <reviewer@local>
    Signed-off-by: yangjie01 <[email protected]>
    (cherry picked from commit fc227c422e81e88d947f55ddb72663c8f1e729c4)
    Signed-off-by: yangjie01 <[email protected]>
---
 .../expressions/codegen/CodeGenerator.scala        | 105 +++++++++++-----
 .../org/apache/spark/sql/internal/SQLConf.scala    |  11 +-
 .../expressions/WithExpressionEvalSuite.scala      |  39 ++++--
 .../expressions/codegen/CollectInputsSuite.scala   |  65 ++++++++--
 .../apache/spark/sql/execution/ExpandExec.scala    |  12 +-
 .../sql/execution/WholeStageCodegenExec.scala      |  11 +-
 .../aggregate/AggregateCodegenSupport.scala        |   2 +-
 .../sql/execution/WholeStageCodegenSuite.scala     | 135 ++++++++++++++++-----
 8 files changed, 277 insertions(+), 103 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
index ff3cfbce3007..7f71196c3111 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
@@ -17,6 +17,8 @@
 
 package org.apache.spark.sql.catalyst.expressions.codegen
 
+import javax.lang.model.SourceVersion
+
 import scala.annotation.tailrec
 import scala.collection.mutable
 import scala.collection.mutable.ArrayBuffer
@@ -122,9 +124,15 @@ case class SubExprCodes(
  *                       `genCode` is refused.
  * @param followCommonExprs follow a `With` and a `CommonExpressionRef` into 
the definitions the
  *                          code computes, rather than walk them like any 
other node.
- * @param onlyNames refuse a value no parameter can name, rather than pass 
over it.
- * @param fieldsNeedNoArgument leave a field out of the arguments, the method 
reading it as it
- *                             stands, rather than pass it.
+ * @param onlyNames refuse a value no parameter can name, rather than pass 
over an expression and
+ *                  pass a variable. A variable that is not a name fails 
testing either way, if
+ *                  the walk reaches it before its first refusal: here under 
this policy, and
+ *                  otherwise where an operator builds a method taking it
+ *                  (`CodegenContext.assertMethodParameters`). The row a whole 
stage split appends
+ *                  is not checked (SPARK-60510).
+ * @param fieldsNeedNoArgument leave a variable naming a field 
`addMutableState` registered out of
+ *                             the arguments, the method reading it as it 
stands, rather than pass
+ *                             it.
  * @param checkParamLength refuse a method whose parameters would pass the 
JVM's limit.
  */
 private[sql] case class InputPolicy(
@@ -655,7 +663,9 @@ class CodegenContext extends Logging {
    *                     class, the name is not changed
    * @return the name of the mutable state variable, which is the original 
name or fresh name if
    *         the variable is inlined to the outer class, or an array access if 
the variable is to
-   *         be stored in an array of variables of the same type.
+   *         be stored in an array of variables of the same type. To hand it 
out as an
+   *         `ExprValue`, use `JavaCode.global` or `isNullGlobal`, not 
`JavaCode.variable`: an
+   *         array access cannot name a method parameter 
(`CodegenContext.collectInputs`).
    *         A variable will be inlined into the outer class when one of the 
following conditions
    *         are satisfied:
    *         1. forceInline is true
@@ -1325,23 +1335,26 @@ class CodegenContext extends Logging {
     val args = mutable.LinkedHashMap.empty[String, VariableValue]
     val inputsToEvaluate = mutable.ArrayBuffer.empty[ExprCode]
     // Whether the code can read `v` from the method: false for a value no 
parameter can carry
-    // where the policy asks for names. `ExpandExec` hands out a 
`VariableValue` naming a slot of a
-    // compacted mutable state array, and a `SimpleExprValue` is an expression 
rather than a name
+    // where the policy asks for names. A `SimpleExprValue` is an expression 
rather than a name
     // -- `posexplode_outer` gives its position the nullness `index == -1`, 
and a `Byte` or `Short`
     // literal's value is `(byte)1`. A field or a literal needs no parameter 
and is read as it
-    // stands; so is a field handed out as a `VariableValue` where the policy 
says so, passing one
-    // being refused by the split methods' own check, a slot of a compacted 
array included.
+    // stands; so is a `VariableValue` naming a field `addMutableState` 
registered, where the
+    // policy says so, passing one being refused by the whole stage split's 
own check (`splitCall`)
+    // in tests. A slot of a compacted array is not a name at all, so it has 
to be handed out as a
+    // `GlobalValue`, and `INPUT_ROW` cannot be one. Testing checks for one 
handed out as a
+    // variable: here, where the policy asks for names, for what the walk 
reaches before its first
+    // refusal; elsewhere, in the method taking it (`assertMethodParameters`). 
The row a whole stage
+    // split appends is not checked (SPARK-60510).
     def take(v: ExprValue): Boolean = v match {
       case local: VariableValue =>
         val name = local.variableName
-        val isName = name.nonEmpty && 
Character.isJavaIdentifierStart(name.head) &&
-          name.forall(Character.isJavaIdentifierPart)
-        // A slot of a compacted array is a field under every policy, since no 
parameter can be
-        // named `array[i]`, where the whole stage split is on; with it off, 
every policy answers
-        // as it did before that split existed. Another field is passed where 
the policy says so.
-        val isField = (isCompactedSlot(name) && wholeStageSplit != 
WholeStageSplit.Off) ||
-          (policy.fieldsNeedNoArgument && mutableStateNames.contains(name))
-        val taken = isName || isField || !policy.onlyNames
+        val isName = isJavaName(name)
+        if (checkParameterNames && policy.onlyNames) {
+          assert(isName, s"$name cannot name a parameter of a method this code 
moves into; " +
+            nameFix(name))
+        }
+        val isField = policy.fieldsNeedNoArgument && 
mutableStateNames.contains(name)
+        val taken = isName || !policy.onlyNames
         if (taken && !isField) {
           args.getOrElseUpdate(name, local)
         }
@@ -1415,12 +1428,19 @@ class CodegenContext extends Logging {
   }
 
   /**
-   * Whether `name` is a slot of a compacted mutable state array, such as 
`mutableStateArray_0[3]`,
-   * which `addMutableState` hands out for a state it does not inline: a field 
of the class.
+   * Whether `name` can name a parameter: a slot of a compacted array, 
`array[i]`, cannot, nor can
+   * a keyword or a literal such as `false`.
    */
-  private def isCompactedSlot(name: String): Boolean = name match {
-    case CodeGenerator.CompactedSlot(array) => 
mutableStateNames.contains(array)
-    case _ => false
+  private def isJavaName(name: String): Boolean =
+    SourceVersion.isIdentifier(name) && !SourceVersion.isKeyword(name)
+
+  /** How the producer of `name`, a value that cannot name a parameter, can 
hand it out instead. */
+  private def nameFix(name: String): String = if (name == INPUT_ROW) {
+    "set INPUT_ROW to a local or a field added with forceInline = true"
+  } else if (SourceVersion.isKeyword(name)) {
+    "hand out a literal as a LiteralValue, or give the variable another name"
+  } else {
+    "hand out a field with JavaCode.global or isNullGlobal"
   }
 
   /**
@@ -1929,6 +1949,39 @@ class CodegenContext extends Logging {
     }
   }
 
+  /**
+   * Whether to assert that each variable a method takes is a Java name, in 
`collectInputs` and
+   * `assertParametersAreNames`. On under testing; a test turns it off to 
check what production
+   * does with such a variable.
+   */
+  private[sql] var checkParameterNames: Boolean = Utils.isTesting
+
+  /**
+   * Under testing, asserts what the method `method` an operator moves its 
code `body` into needs
+   * of its `parameters`: each is a Java name (`assertParametersAreNames`), 
and each call to a split
+   * function in `body` passes only those parameters 
(`assertSplitCallsWithin`).
+   */
+  private[sql] def assertMethodParameters(
+      body: => String,
+      parameters: Seq[String],
+      method: String): Unit = {
+    assertParametersAreNames(parameters, method)
+    assertSplitCallsWithin(body, parameters, method)
+  }
+
+  /**
+   * Where `checkParameterNames`, asserts that each of `parameters`, the 
parameters of the method
+   * `method` an operator moves its code into, is a Java name. The parameters 
are the variables
+   * `InputPolicy.operatorMethod` collects, which passes a `VariableValue` 
whatever its name.
+   */
+  private[sql] def assertParametersAreNames(parameters: Seq[String], method: 
String): Unit = {
+    if (checkParameterNames) {
+      parameters.foreach { p =>
+        assert(isJavaName(p), s"$method takes $p, which cannot name a 
parameter; ${nameFix(p)}")
+      }
+    }
+  }
+
   /**
    * Splits the generated code of expressions into multiple sequences of String
    * based on a threshold of length of a String
@@ -2150,9 +2203,8 @@ class CodegenContext extends Logging {
     // The code of the non-split pass is discarded when the split pass below 
is taken, and with it
     // the calls to every function added while generating it, such as the 
split methods of a CASE
     // WHEN or a `With` definition's method. A generation that splits the 
stage's expressions
-    // (`WholeStageSplit.Split`) removes those functions then; any other keeps 
them, as the class
-    // did before that split existed, so that a stage kept in one piece, and 
the split turned off,
-    // give back the code generated before it.
+    // (`WholeStageSplit.Split`) removes those functions then; any other, a 
stage kept in one
+    // piece or the split turned off, keeps them, as the class did before that 
split existed.
     val functionsBeforeNonSplit = functionsMark()
     val occurrencesBeforeNonSplit = wholeStageSplitOccurrences.clone()
     val blocksSplitBeforeNonSplit = wholeStageBlocksSplit
@@ -2224,7 +2276,7 @@ class CodegenContext extends Logging {
           // Generate the code for this expression tree and wrap it in a 
function.
           val fnName = freshName("subExpr")
           val inputVars = inputVarsForAllFuncs(i)
-          assertSplitCallsWithin(eval.code.toString, 
inputVars.map(_.variableName), fnName)
+          assertMethodParameters(eval.code.toString, 
inputVars.map(_.variableName), fnName)
           val argList =
             inputVars.map(v => s"${CodeGenerator.typeName(v.javaType)} 
${v.variableName}")
           val fn =
@@ -3060,9 +3112,6 @@ object CodeGenerator extends Logging {
   // The whole stage split of expressions 
(`CodegenContext.splitExpressionsWithSources`)
   // 
---------------------------------------------------------------------------------------------
 
-  /** A slot of a compacted mutable state array, such as 
`mutableStateArray_0[3]`. */
-  private[codegen] val CompactedSlot = "([A-Za-z_$][\\w$]*)\\[\\d+\\]".r
-
   /** A name and an opening parenthesis: a call, or a keyword such as `if`. */
   private[codegen] val CallSite = "(?<![\\w$])([A-Za-z_$][\\w$]*)\\s*\\(".r
 
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index d7364155caac..eeb0b3099627 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -3233,14 +3233,9 @@ object SQLConf {
         "supports it, such as a CASE WHEN with many branches, into methods 
that take the input " +
         "variables they read as parameters, the way code generation outside 
whole stage codegen " +
         "splits it. In a stage whose expressions are split, the methods that 
subexpression " +
-        "elimination's discarded first pass added are removed; and in every 
stage, a slot of " +
-        "a compacted mutable state array counts as the field it is when code 
moves into a " +
-        "method: an operator's method that took it as a parameter, and failed 
to compile, now " +
-        "compiles, and a common expression's definition reading one, which 
stayed inline, " +
-        "gets its method. When false, " +
-        "the code stays in the method of its operator, where a large enough 
expression goes " +
-        "past the JVM's 64KB method limit and fails to compile, and the 
generated code is what " +
-        "it was before this conf existed.")
+        "elimination's discarded first pass added are removed. When false, the 
code stays in " +
+        "the method of its operator, where a large enough expression goes past 
the JVM's 64KB " +
+        "method limit and fails to compile.")
       .version("4.4.0")
       .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
       .booleanConf
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/WithExpressionEvalSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/WithExpressionEvalSuite.scala
index fd9a42a14bcd..c1573afff503 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/WithExpressionEvalSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/WithExpressionEvalSuite.scala
@@ -91,8 +91,15 @@ class WithExpressionEvalSuite extends SparkFunSuite with 
SQLHelper {
   }
 
   /** Generates `e` against [[localInputContext]]: the code it emitted, and 
the methods added. */
-  private def generateWithLocalInput(e: Expression, inputs: ExprCode*): 
(String, String) = {
+  private def generateWithLocalInput(e: Expression, inputs: ExprCode*): 
(String, String) =
+    generateWithLocalInput(e, checkParameterNames = true, inputs: _*)
+
+  private def generateWithLocalInput(
+      e: Expression,
+      checkParameterNames: Boolean,
+      inputs: ExprCode*): (String, String) = {
     val ctx = localInputContext(inputs: _*)
+    ctx.checkParameterNames = checkParameterNames
     (e.genCode(ctx).code.toString, ctx.declareAddedFunctions())
   }
 
@@ -305,18 +312,9 @@ class WithExpressionEvalSuite extends SparkFunSuite with 
SQLHelper {
           s"the innermost body was emitted $count times at depth $depth")
         assert(!methods.contains("computeCommonExpr"), methods)
       }
-      // A value whose name is not one a parameter can take: `ExpandExec` 
hands out a compacted
-      // mutable state array slot as if it were a local. The marker count says 
the fallback pasted
-      // the body at both references; that a method was attempted at all is 
the case above, which
-      // gets one from the same shape with a passable name.
-      val slot = ExprCode(EmptyBlock, FalseLiteral,
-        JavaCode.variable("mutableStateArray_0[3]", IntegerType))
-      val (slotBody, slotMethods) = generateWithLocalInput(nested(2), slot)
-      assert(!slotMethods.contains("computeCommonExpr"), slotMethods)
-      assert(markersIn(slotBody + slotMethods) == 4, slotBody + slotMethods)
       // A nullness that is an expression rather than a name, the shape 
`GenerateExec` hands out
       // for the position of `posexplode_outer`: whichever locals such an 
expression names, this
-      // has no way to pass them. A plain name in its place does get a method, 
in the case above.
+      // has no way to pass them. A plain name in its place does get a method, 
in the previous test.
       val position = ExprCode(EmptyBlock, JavaCode.isNullExpression("index_0 
== -1"),
         JavaCode.variable("index_0", IntegerType))
       val (positionBody, positionMethods) =
@@ -337,6 +335,25 @@ class WithExpressionEvalSuite extends SparkFunSuite with 
SQLHelper {
     }
   }
 
+  test("SPARK-59589: a slot handed out as a local fails the testing check 
where a definition's " +
+      "method is considered") {
+    // A value whose name is not one a parameter can take, such as a slot of a 
compacted mutable
+    // state array handed out as a local. Its producer has to hand it out as a 
field; "a method for
+    // the local variables the operator has already evaluated" gets a method 
from the same shape
+    // with a passable name.
+    withSQLConf(SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1024") {
+      val slot = ExprCode(EmptyBlock, FalseLiteral,
+        JavaCode.variable("mutableStateArray_0[3]", IntegerType))
+      val e = intercept[AssertionError](generateWithLocalInput(nested(2), 
slot))
+      assert(e.getMessage.contains("mutableStateArray_0[3] cannot name a 
parameter"),
+        e.getMessage)
+      // Without the check, as in production, the method is refused and the 
bodies stay inline.
+      val (body, methods) = generateWithLocalInput(nested(2), 
checkParameterNames = false, slot)
+      assert(!methods.contains("computeCommonExpr"), methods)
+      assert(markersIn(body + methods) == 4, body + methods)
+    }
+  }
+
   test("SPARK-59295: a field needs no parameter, a subexpression elimination 
state no method") {
     withSQLConf(SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1024") {
       // An input variable the operator allocated as mutable state, which is 
what the aggregate
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CollectInputsSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CollectInputsSuite.scala
index d72901e1567d..88dce5f6150e 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CollectInputsSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/CollectInputsSuite.scala
@@ -139,29 +139,72 @@ class CollectInputsSuite extends SparkFunSuite with 
SQLHelper {
     assert(!ctx.collectInputs(Seq(input(0)), operator, Map.empty).get.readsRow)
   }
 
-  test("a slot of a compacted mutable state array is a field under every 
policy with the whole " +
-      "stage split on, never a parameter, which no method could declare") {
+  test("a field handed out as a global is never an argument; an inlined one 
handed out as a " +
+      "variable is left out only where the policy says so") {
     val ctx = context()
-    ctx.wholeStageSplit = WholeStageSplit.Record
-    // A state of a type that is not primitive is compacted into an array, and 
its name is a slot.
+    val expr = Length(BoundReference(0, StringType, nullable = false))
+    // A state of a type that is not primitive is a slot of a compacted array.
     val slot = ctx.addMutableState("UTF8String", "value")
     assert(slot.matches("\\w+\\[\\d+\\]"), slot)
-    ctx.currentVars = Seq(ExprCode(EmptyBlock, FalseLiteral, 
JavaCode.variable(slot, StringType)))
-    val expr = Length(BoundReference(0, StringType, nullable = false))
+    ctx.currentVars = Seq(ExprCode(EmptyBlock, FalseLiteral, 
JavaCode.global(slot, StringType)))
     for (policy <- Seq(InputPolicy.wholeStageSplit, commonExpr, operator)) {
       assert(names(ctx.collectInputs(Seq(expr), policy, 
Map.empty).get).isEmpty, policy)
     }
+    // An inlined state handed out as a variable is read as the field only by 
a whole stage split
+    // method.
+    val field = ctx.addMutableState("UTF8String", "field", forceInline = true)
+    ctx.currentVars = Seq(ExprCode(EmptyBlock, FalseLiteral, 
JavaCode.variable(field, StringType)))
+    assert(names(ctx.collectInputs(Seq(expr), InputPolicy.wholeStageSplit, 
Map.empty).get).isEmpty)
+    for (policy <- Seq(commonExpr, operator)) {
+      assert(names(ctx.collectInputs(Seq(expr), policy, Map.empty).get) == 
Seq(field), policy)
+    }
   }
 
-  test("with the whole stage split off, a slot is answered as before that 
split existed") {
-    // A common expression's method refuses it, so its definition stays 
inline, and an operator's
-    // method lists it among its inputs.
+  test("a slot handed out as a variable fails the testing check: while 
collecting where the " +
+      "policy asks for names, and in the method that takes it where it does 
not") {
     val ctx = context()
     val slot = ctx.addMutableState("UTF8String", "value")
     ctx.currentVars = Seq(ExprCode(EmptyBlock, FalseLiteral, 
JavaCode.variable(slot, StringType)))
     val expr = Length(BoundReference(0, StringType, nullable = false))
-    assert(ctx.collectInputs(Seq(expr), commonExpr, Map.empty).isEmpty)
-    assert(names(ctx.collectInputs(Seq(expr), operator, Map.empty).get) == 
Seq(slot))
+    for (policy <- Seq(InputPolicy.wholeStageSplit, commonExpr)) {
+      val e = intercept[AssertionError](ctx.collectInputs(Seq(expr), policy, 
Map.empty))
+      assert(e.getMessage.contains(s"$slot cannot name a parameter") &&
+        e.getMessage.contains("JavaCode.global"), e.getMessage)
+    }
+    // Without the check, as in production, these policies refuse the method.
+    ctx.checkParameterNames = false
+    for (policy <- Seq(InputPolicy.wholeStageSplit, commonExpr)) {
+      assert(ctx.collectInputs(Seq(expr), policy, Map.empty).isEmpty, policy)
+    }
+    ctx.checkParameterNames = true
+    // An operator may keep its code inline, so collecting does not check.
+    val parameters = names(ctx.collectInputs(Seq(expr), operator, 
Map.empty).get)
+    assert(parameters == Seq(slot))
+    val e = intercept[AssertionError](ctx.assertParametersAreNames(parameters, 
"m"))
+    assert(e.getMessage.contains(s"m takes $slot") && 
e.getMessage.contains("JavaCode.global"),
+      e.getMessage)
+    ctx.assertParametersAreNames(Seq("value_0", "isNull_0"), "m")
+    // A keyword or a literal cannot name a parameter either.
+    val keyword = 
intercept[AssertionError](ctx.assertParametersAreNames(Seq("false"), "m"))
+    assert(keyword.getMessage.contains("m takes false") &&
+      keyword.getMessage.contains("LiteralValue"), keyword.getMessage)
+  }
+
+  test("an INPUT_ROW that is a slot fails the testing check: while collecting 
for a common " +
+      "expression's method, and in an operator's method that takes it") {
+    val ctx = context()
+    ctx.INPUT_ROW = ctx.addMutableState("InternalRow", "row")
+    assert(ctx.INPUT_ROW.matches("\\w+\\[\\d+\\]"), ctx.INPUT_ROW)
+    ctx.currentVars = Seq(evaluated(0))
+    val e1 = intercept[AssertionError](ctx.collectInputs(Seq(input(0)), 
commonExpr, Map.empty))
+    assert(e1.getMessage.contains("set INPUT_ROW to a local"), e1.getMessage)
+    ctx.checkParameterNames = false
+    assert(ctx.collectInputs(Seq(input(0)), commonExpr, Map.empty).isEmpty)
+    ctx.checkParameterNames = true
+    val parameters = names(ctx.collectInputs(Seq(input(0)), operator, 
Map.empty).get)
+    assert(parameters.head == ctx.INPUT_ROW)
+    val e2 = 
intercept[AssertionError](ctx.assertParametersAreNames(parameters, "m"))
+    assert(e2.getMessage.contains("set INPUT_ROW to a local"), e2.getMessage)
   }
 
   test("the whole stage split walks below the state of a node that generates 
its child") {
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
index ee31909549e0..0084429493f2 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
@@ -189,9 +189,15 @@ case class ExpandExec(
             "resultValue",
             v => s"$v = ${CodeGenerator.defaultValue(firstExpr.dataType)};")
 
+          // `addMutableState` hands back a field: a name of its own, or a 
slot in a compacted array
+          // (`mutableStateArray_0[3]`) for a state not inlined. Either is 
reachable from every
+          // method of the generated class and is not a local variable, so a 
consumer collecting
+          // local variables to pass into a split function must not take it 
for one -- a slot
+          // expression is not a legal parameter name. Whole-stage 
subexpression elimination wraps
+          // its own mutable state the same way.
           ExprCode(
-            JavaCode.isNullVariable(isNull),
-            JavaCode.variable(value, firstExpr.dataType))
+            JavaCode.isNullGlobal(isNull),
+            JavaCode.global(value, firstExpr.dataType))
         }
       }
     }
@@ -232,7 +238,7 @@ case class ExpandExec(
         val paramLength = 
CodeGenerator.calculateParamLengthFromExprValues(inputVars)
         val maybeSplitUpdateCode = if 
(CodeGenerator.isValidParamLength(paramLength)) {
           val switchCaseFunc = ctx.freshName("switchCaseCode")
-          ctx.assertSplitCallsWithin(updateCode, 
inputVars.map(_.variableName), switchCaseFunc)
+          ctx.assertMethodParameters(updateCode, 
inputVars.map(_.variableName), switchCaseFunc)
           val argList = inputVars.map { v =>
             s"${CodeGenerator.typeName(v.javaType)} ${v.variableName}"
           }
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
index 21ba2d17c432..cc46d3bdecc0 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
@@ -629,9 +629,8 @@ object WholeStageCodegenExec {
    * method sizes, so the class is then dropped and compiled again. A hit 
happens where the class
    * is cached while the decision for its code is not: the decision was 
evicted, or the same body
    * was compiled earlier without being decided - with the split off, whose 
code is the recorded
-   * code without its marks, but for the compacted-slot rule, or under another 
limit. Should
-   * another thread load the class again in between, the second compile is a 
hit as well, and the
-   * sizes are unknown.
+   * code without its marks, or under another limit. Should another thread 
load the class again
+   * in between, the second compile is a hit as well, and the sizes are 
unknown.
    */
   private[execution] def trialCompile(
       code: CodeAndComment,
@@ -774,11 +773,7 @@ case class WholeStageCodegenExec(child: SparkPlan)(val 
codegenStageId: Int)
    * one piece, with the code of each expression a split could take marked, 
and only where there
    * is one is that code compiled to choose which expressions to split 
([[chooseCode]]). A stage
    * the JIT compiles whole keeps that code, which a split's calls would only 
slow down; the trial
-   * compile of the code kept is the one [[doExecute]] finds in the compile 
cache. That code
-   * differs from the code with the conf off in one place: a slot of a 
compacted mutable state
-   * array counts as the field it is when code moves into a method 
(`CodegenContext.collectInputs`).
-   * An operator's method that took it as a parameter, and failed to compile, 
now compiles; a
-   * `With` definition reading one, which stayed inline, gets its method.
+   * compile of the code kept is the one [[doExecute]] finds in the compile 
cache.
    *
    * @return the tuple of the codegen context and the actual generated source.
    */
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateCodegenSupport.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateCodegenSupport.scala
index 15d51aeab219..88903aa4e163 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateCodegenSupport.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/aggregate/AggregateCodegenSupport.scala
@@ -351,7 +351,7 @@ trait AggregateCodegenSupport
       if (inputVars.forall(_.isDefined)) {
         val splitCodes = inputVars.flatten.zipWithIndex.map { case (args, i) =>
           val doAggFunc = ctx.freshName(s"doAggregate_${aggNames(i)}")
-          ctx.assertSplitCallsWithin(aggCodeBlocks(i).toString, 
args.map(_.variableName), doAggFunc)
+          ctx.assertMethodParameters(aggCodeBlocks(i).toString, 
args.map(_.variableName), doAggFunc)
           val argList = args.map { v =>
             s"${CodeGenerator.typeName(v.javaType)} ${v.variableName}"
           }.mkString(", ")
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
index d8af608a1641..7a2753571c4e 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
@@ -28,10 +28,11 @@ import org.scalatest.time.SpanSugar._
 
 import org.apache.spark.{SparkArithmeticException, SparkException, 
SparkNumberFormatException}
 import org.apache.spark.metrics.source.CodegenMetrics
-import org.apache.spark.rdd.MapPartitionsWithEvaluatorRDD
+import org.apache.spark.rdd.{MapPartitionsWithEvaluatorRDD, RDD}
 import org.apache.spark.sql.{DataFrame, Dataset, Row, SaveMode}
+import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.expressions.{And, Attribute, Cast, 
CodegenObjectFactoryMode, Expression, IsNotNull, Literal, With}
-import org.apache.spark.sql.catalyst.expressions.codegen.{ByteCodeStats, 
CodeAndComment, CodegenContext, CodeGenerator, WholeStageSplit, 
WholeStageSplitKey}
+import org.apache.spark.sql.catalyst.expressions.codegen.{ByteCodeStats, 
CodeAndComment, CodegenContext, CodeGenerator, ExprCode, ExprValue, 
GlobalValue, JavaCode, WholeStageSplit, WholeStageSplitKey}
 import org.apache.spark.sql.catalyst.plans.logical.MergeRows
 import org.apache.spark.sql.catalyst.plans.logical.MergeRows.{Keep, Update}
 import org.apache.spark.sql.classic
@@ -889,8 +890,10 @@ class WholeStageCodegenSuite extends SharedSparkSession
     
wholeStageCodeGenExec.get.asInstanceOf[WholeStageCodegenExec].doCodeGen()._2
   }
 
-  def genCode(ds: Dataset[_]): Seq[CodeAndComment] = {
-    val plan = ds.queryExecution.executedPlan
+  def genCode(
+      ds: Dataset[_],
+      transform: SparkPlan => SparkPlan = identity): Seq[CodeAndComment] = {
+    val plan = transform(ds.queryExecution.executedPlan)
     val wholeStageCodeGenExecs = plan.collect { case p: WholeStageCodegenExec 
=> p }
     assert(wholeStageCodeGenExecs.nonEmpty, "WholeStageCodegenExec is 
expected")
     wholeStageCodeGenExecs.map(_.doCodeGen()._2)
@@ -1747,6 +1750,19 @@ class WholeStageCodegenSuite extends SharedSparkSession
       "sinh(v) should be evaluated only once per input row without function 
splitting")
   }
 
+  test("SPARK-59589: a subexpression method reads an Expand's varying output 
as a field") {
+    // `ExpandExec` holds an UNPIVOT's string output in a slot of a compacted 
mutable state array,
+    // such as `expand_mutableStateArray_0[1]`. A large common subexpression 
reading it is computed
+    // in a method of its own, which takes the local variables it reads as 
parameters. Taken for a
+    // local variable, the slot became a parameter `UTF8String 
expand_mutableStateArray_0[1]` and
+    // the stage did not compile. Handed out as a field, it is read as one, 
split on or off.
+    val f = "concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), 
rtrim(v), initcap(v), " +
+      "repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y'))"
+    val stringParameter = "private void \\w*subExpr_\\d+\\([^)]*UTF8String ".r
+    checkExpandSlotRead("subExpr", unpivot => s"SELECT c, $f AS x, concat($f, 
'z') AS y $unpivot",
+      code => assert(code.forall(c => 
stringParameter.findFirstIn(c.body).isEmpty)))
+  }
+
   test("SPARK-60398: Expand compiles with its switch case functions spilled to 
nested classes") {
     // A sliding window has its Expand generate a row for each window a time 
can fall in: 360 rows
     // for a window of an hour sliding by 10 seconds. Without subexpression 
elimination the code
@@ -1774,6 +1790,22 @@ class WholeStageCodegenSuite extends SharedSparkSession
     }
   }
 
+  test("SPARK-59589: split switch cases and aggregate functions read an 
Expand's output as a " +
+      "field") {
+    // With the consume functions not split by operator, the consumer reads 
the UNPIVOT's output
+    // directly, and a tiny threshold moves its code into methods: the switch 
cases of the Expand
+    // that `RewriteDistinctAggregates` adds, and the aggregate functions.
+    withSQLConf(
+        SQLConf.WHOLESTAGE_SPLIT_CONSUME_FUNC_BY_OPERATOR.key -> "false",
+        SQLConf.CODEGEN_METHOD_SPLIT_THRESHOLD.key -> "1") {
+      checkExpandSlotRead("switchCaseCode", unpivot =>
+        s"SELECT count(DISTINCT lpad(v, 10, 'x')), count(DISTINCT rpad(v, 10, 
'y')) $unpivot")
+      checkExpandSlotRead("doAggregate_sum", unpivot =>
+        s"SELECT c, sum(length(lpad(v, 10, 'x'))), sum(length(rpad(v, 10, 
'y'))) $unpivot " +
+          "GROUP BY c")
+    }
+  }
+
   test("SPARK-59295: a nested With in a branch is emitted once per scope under 
whole-stage") {
     // A whole-stage `Project` passes its input as local variables, which is 
where
     // `CommonExprSlots.fill` could not put a definition in a method before. 
Both `nullif`s survive
@@ -1932,6 +1964,46 @@ class WholeStageCodegenSuite extends SharedSparkSession
       query.collect().toSeq
     }
 
+  /**
+   * Checks the query `query` builds from a FROM clause that UNPIVOTs two 
string columns into
+   * `v`, with the whole stage split on and off: a method named like `method` 
calls `lpad` on the
+   * slot of the compacted mutable state array that `ExpandExec` holds `v` in, 
`check` holds of
+   * the code, the code compiles, and the rows match those computed without 
whole stage codegen.
+   * With the slot handed out as a local variable instead, building that 
method fails the testing
+   * check on its parameters.
+   */
+  private def checkExpandSlotRead(
+      method: String,
+      query: String => String,
+      check: Seq[CodeAndComment] => Unit = _ => ()): Unit = {
+    withTempView("t") {
+      spark.range(10).selectExpr("concat('a', CAST(id AS STRING)) AS a",
+        "concat('b', CAST(id AS STRING)) AS b").createOrReplaceTempView("t")
+      def df: DataFrame = sql(query("FROM t UNPIVOT (v FOR c IN (a, b))"))
+      val expected = withoutWholeStage(df)
+      Seq("true", "false").foreach { split =>
+        withSQLConf(SQLConf.WHOLESTAGE_SPLIT_EXPRESSIONS.key -> split) {
+          val code = genCode(df)
+          // A subexpression method or a switch case also writes into a slot 
of the same array,
+          // so the array's name alone does not show a read.
+          assert(code.exists(c => methodsHolding(c, 
"].lpad(").exists(_.contains(method))),
+            s"$method $split")
+          withClue(s"$method $split: ")(check(code))
+          code.foreach(CodeGenerator.compile)
+          checkAnswer(df, expected)
+        }
+      }
+      withSQLConf(SQLConf.WHOLESTAGE_SPLIT_EXPRESSIONS.key -> "false") {
+        // The bottom `ExpandExec` hands out its output as local variables, as 
before SPARK-59589.
+        val e = intercept[AssertionError](genCode(df, _.transformUp {
+          case x: ExpandExec if !x.child.exists(_.isInstanceOf[ExpandExec]) => 
LocalOutputExec(x)
+        }))
+        assert(s"\\w*$method\\w* takes 
\\w+\\[\\d+\\]".r.findFirstIn(e.getMessage).isDefined,
+          e.getMessage)
+      }
+    }
+  }
+
   test("SPARK-33301: a CASE WHEN too large for one method is split under whole 
stage codegen") {
     // Inside a stage the inputs are local variables of the stage's method, 
and the branches were
     // not split into methods at all: at 3000 branches the method went past 
the JVM's 64KB limit,
@@ -2800,11 +2872,10 @@ class WholeStageCodegenSuite extends SharedSparkSession
   test("SPARK-33301: a split method reads a slot of a compacted mutable state 
array as a field") {
     withSplitAlways {
       // `ExpandExec` holds an output of a type that is not primitive in a 
slot of a compacted
-      // mutable state array, such as `expand_mutableStateArray_0[0]`: a 
field, though not a name.
+      // mutable state array, such as `expand_mutableStateArray_0[0]`, and 
hands it out as a field.
       // An UNPIVOT of string columns gives such an output, which the 
projection above reads
       // directly when the operators' consume functions are not split into 
methods of their own.
-      // Every block reads it, the ELSE too, and each reads it as the field it 
is, taking no
-      // argument for it.
+      // Every block reads it, the ELSE too, and each reads it as the field it 
is.
       withSQLConf(SQLConf.WHOLESTAGE_SPLIT_CONSUME_FUNC_BY_OPERATOR.key -> 
"false") {
         withTempView("t") {
           spark.range(100).selectExpr("id", "concat('a', CAST(id % 50 AS 
STRING)) AS a",
@@ -2816,38 +2887,12 @@ class WholeStageCodegenSuite extends SharedSparkSession
           assert(genCode(df).exists { c =>
             methodsHolding(c, 
"expand_mutableStateArray").exists(_.contains("caseWhen"))
           })
-          
assert(splitCaseWhenParameters(df).flatten.forall(!_.contains("mutableStateArray")))
           checkAnswer(df, withoutWholeStage(df))
         }
       }
     }
   }
 
-  test("SPARK-33301: with the split on, a subexpression method reads a 
compacted slot as a field") {
-    // `ExpandExec` holds an UNPIVOT's string output in a slot of a compacted 
mutable state array.
-    // A large common subexpression reading it is computed in a method of its 
own, which takes its
-    // inputs as parameters. No parameter can be named `array[i]`, so with the 
split off, as before
-    // it, that method takes the slot as one and does not compile; with it on, 
in every stage,
-    // CASE WHEN or not, the method reads the slot as the field it is.
-    withTempView("t") {
-      spark.range(10).selectExpr("concat('a', CAST(id AS STRING)) AS a",
-        "concat('b', CAST(id AS STRING)) AS b").createOrReplaceTempView("t")
-      val f = "concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), 
rtrim(v), initcap(v), " +
-        "repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y'))"
-      def df: DataFrame =
-        sql(s"SELECT c, $f AS x, concat($f, 'z') AS y FROM t UNPIVOT (v FOR c 
IN (a, b))")
-      // A parameter named for the slot's string, whatever its name: the slot 
is read as a field.
-      val stringParameter = "private void \\w*subExpr_\\d+\\([^)]*UTF8String 
".r
-      val code = genCode(df)
-      assert(code.forall(c => stringParameter.findFirstIn(c.body).isEmpty))
-      code.foreach(CodeGenerator.compile)
-      checkAnswer(df, withoutWholeStage(df))
-      withSQLConf(SQLConf.WHOLESTAGE_SPLIT_EXPRESSIONS.key -> "false") {
-        assert(genCode(df).exists(c => 
stringParameter.findFirstIn(c.body).nonEmpty))
-      }
-    }
-  }
-
   // A Parquet table, so that the filter is not evaluated by the optimizer on 
a local relation.
   // On the first row `NOT (a > 0 AND b > 0)` is NULL, and no `isnotnull(a)` 
is inferred from it.
   // The second row passes every condition of the tests below, except `b < 0`, 
`(a > 0 OR b < 0)`
@@ -3250,3 +3295,27 @@ class WholeStageCodegenSuite extends SharedSparkSession
     }
   }
 }
+
+/** Hands out the values and null flags its child hands out as fields as local 
variables. */
+private case class LocalOutputExec(child: SparkPlan) extends UnaryExecNode 
with CodegenSupport {
+  override def output: Seq[Attribute] = child.output
+
+  override def inputRDDs(): Seq[RDD[InternalRow]] =
+    child.asInstanceOf[CodegenSupport].inputRDDs()
+
+  override protected def doProduce(ctx: CodegenContext): String =
+    child.asInstanceOf[CodegenSupport].produce(ctx, this)
+
+  override def doConsume(ctx: CodegenContext, input: Seq[ExprCode], row: 
ExprCode): String =
+    consume(ctx, input.map(e => e.copy(isNull = local(e.isNull), value = 
local(e.value))))
+
+  private def local(v: ExprValue): ExprValue = v match {
+    case GlobalValue(name, javaType) => JavaCode.variable(name, javaType)
+    case _ => v
+  }
+
+  override protected def doExecute(): RDD[InternalRow] = child.execute()
+
+  override protected def withNewChildInternal(newChild: SparkPlan): 
LocalOutputExec =
+    copy(child = newChild)
+}


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to