dongjoon-hyun opened a new pull request, #59650: URL: https://github.com/apache/spark/pull/59650
### What changes were proposed in this pull request? This PR aims to fix `ExpandExec` to call its split switch case functions by the names that `CodegenContext.addNewFunction` returns, as `subexpressionEliminationForWholeStageCodegen` and `AggregateCodegenSupport` already do. Since SPARK-35329 (apache/spark#32457, [v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0) on 2021-10-06), `ExpandExec` moves the code of each switch case into a function of its own when the switch code is large. However, it ignores the return value of `ctx.addNewFunction` and calls the function by its bare name. Once the generated class grows past `CodeGenerator.GENERATED_CLASS_SIZE_THRESHOLD` (1,000,000 characters), `addNewFunction` puts the next functions into private nested classes and returns qualified names like `expand_nestedClassInstance_0.expand_switchCaseCode_124`. A bare name does not resolve when the call comes from outside of that nested class: - from `processNext` of the outer class, when the consume code is not split into functions (e.g. `spark.sql.codegen.splitConsumeFuncByOperator=false`), or - from a sibling nested class, e.g. `expand_doConsume_0` or `hashAgg_doAggregateWithKeys_0`, which are added after the switch case functions and land in a later nested class. ### Why are the changes needed? To fix a whole-stage codegen compilation failure. A sliding time window generates an `Expand` with one projection per overlapping window, so a long window with a short slide generates enough switch case functions to spill into nested classes. For example, with the default codegen configurations, the following query (1800 projections) fails to compile because the switch case functions land in three nested classes. ```scala spark.range(10).selectExpr("timestamp_seconds(id) AS ts") .groupBy(window($"ts", "1 hour", "2 seconds")).count() ``` ``` org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 56905, Column 1: A method named "expand_switchCaseCode_0" is not declared in any enclosing class nor any supertype, nor through a static import ``` ### Does this PR introduce _any_ user-facing change? Yes. Previously, the whole-stage codegen of such queries failed to compile, and Spark fell back to the execution without whole-stage codegen, or failed the query when `spark.sql.codegen.fallback` is `false`. After this PR, the generated code compiles. The query results do not change. ### How was this patch tested? Pass the CIs with the newly added test case. Without this fix, the new test case fails like the following. ``` $ build/sbt 'sql/testOnly *WholeStageCodegenSuite -- -z "SPARK-60398"' ... [info] - SPARK-60398: Expand compiles with its switch case functions spilled to nested classes *** FAILED *** (3 seconds, 523 milliseconds) [info] java.util.concurrent.ExecutionException: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 46676, Column 1: Failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 46676, Column 1: A method named "expand_switchCaseCode_124" is not declared in any enclosing class nor any supertype, nor through a static import ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
