MaxGekk opened a new pull request, #59042:
URL: https://github.com/apache/spark/pull/59042
### What changes were proposed in this pull request?
`CodegenContext.splitExpressions` splits the generated code of expressions
into functions of at most `spark.sql.codegen.methodSplitThreshold` characters,
but leaves one call per split function in the calling method. This PR groups
those calls in turn: wherever their fold is longer than the same threshold,
consecutive calls are put into functions of their own, level by level, until
the calls left in the caller are short.
A group is folded with the caller's `foldFunctions` and wrapped with its
`makeSplitFunction`, which is exactly what `generateInnerClassesFunctionCalls`
already does with the functions of one inner class, so every caller of
`splitExpressions` already supports it. For `CaseWhen`, for example, a group
becomes a function that runs its calls inside its own `do { ... } while
(false)` and returns the result state, like the split functions themselves. A
group is only formed when it holds more than one call, so the recursion ends.
### Why are the changes needed?
The calls are what is left in the calling method, and there are as many as
the code had blocks. When the code splits into hundreds of functions, as a
`CASE WHEN` of a thousand branches does, the calls alone take the calling
method past the 8000 bytes HotSpot compiles (`-XX:+DontCompileHugeMethods`), so
the method every row goes through runs interpreted.
With whole-stage codegen off, `SELECT CASE WHEN v = 1 THEN v * 1 ... WHEN v
= 1000 THEN v * 1000 ELSE 0 END FROM (SELECT id % 1000 AS v FROM
range(2000000))` generates a `CaseWhen_0` method holding the calls that is 8060
bytes. Spark logs it as too long to be JIT compiled, and
`-XX:+PrintCompilation` shows it never compiled. A run takes about 11.1 s on a
laptop. With this change its largest generated method is a few hundred bytes,
and a run takes about 1.7 s.
### Does this PR introduce _any_ user-facing change?
No, other than the performance of queries whose expressions split into many
functions.
### How was this patch tested?
A new test in `CodeGenerationSuite`: a mutable projection of a 3000-branch
`CASE WHEN` answers correctly and Spark logs no generated method too long to be
JIT compiled. Without the change it logs `CaseWhen_0` at 10389 bytes and the
test fails.
Existing suites: `CodeGenerationSuite`, `ConditionalExpressionSuite`,
`SubexpressionEliminationSuite`, `CodegenSubexpressionEliminationSuite`,
`WithExpressionEvalSuite`, `GeneratedProjectionSuite`,
`UnsafeRowConverterSuite`, `HashExpressionsSuite`, `StringExpressionsSuite`,
`PredicateSuite` and `ComplexTypeSuite` in catalyst; `WholeStageCodegenSuite`,
`TPCDSQuerySuite`, `TPCDSQueryWithStatsSuite`, `TPCHQuerySuite`,
`DataFrameAggregateSuite`, `DataFrameSuite` and `SQLQuerySuite` in sql.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
