qingfureal opened a new pull request, #59678:
URL: https://github.com/apache/spark/pull/59678
### What changes were proposed in this pull request?
Under whole-stage codegen, `ProjectExec` generates all its outputs into one
method. This PR lets the SPARK-33301 gate split a projection's outputs, each
into a method of its own, when the method they land in is past the JIT limit.
- `CodegenContext.generateOutputsWithWholeStageSplit`, which
`ProjectExec.doConsume` now calls instead of `exprs.map(_.genCode(ctx))`,
follows `wholeStageSplit` as `splitExpressionsWithSources` does: off, the
outputs are generated as before; under `Record`, they are generated as before
and marked; under `Split`, when the gate chose this projection, each output
gets a method.
- A split output's method stores its value in fields, as subexpression
elimination's split functions do, and the output's code becomes the call. The
operator consuming the projection still evaluates each output where it reads
it, so no output is evaluated earlier than without the split. Splitting outputs
in blocks would: under ANSI mode, for example, a block evaluated ahead of a
filter above the projection raises an error for a row the filter drops.
- The method takes the locals the output's code reads, found by
`collectInputs` under a new policy, `InputPolicy.wholeStageOutput`:
`wholeStageSplit` with `hoistDeferredInputs`. The inputs the output reads that
are not evaluated yet are evaluated just ahead of its call instead of inside
its code. Only that output reads them, since `ProjectExec` defers only the
inputs one output reads (`usedInputs`).
- An output that only passes on an input or a literal stays in place, and so
does one whose inputs no method can take. A projection with fewer than two
outputs to move is left to the split within its expression (CASE WHEN today,
SPARK-60047 for the others), since moving one output just moves the large
method.
- `nextWholeStageSplitKey` and `wholeStageSplitArguments` are refactored so
that the CASE WHEN split and this one share them.
A stage whose methods the JIT compiles whole keeps its code byte for byte,
as with the CASE WHEN split.
### Why are the changes needed?
A projection of a few dozen ordinary expressions puts the stage's method
past HotSpot's 8000-byte limit for JIT-compiling a method, although no single
output is large. Spark falls back from whole-stage codegen only past
`spark.sql.codegen.hugeMethodLimit` (64KB), so the stage keeps whole-stage
codegen and runs interpreted. Measured on master (local, JDK 17), 300,000 rows,
each column `concat_ws('-', upper(s), lower(s), cast(id + i as string),
substring(s, i, 3), cast(id * i % 7 as string))`:
| Columns | Whole-stage codegen on | Whole-stage codegen off |
|---|---:|---:|
| 50 | 7.7 s | 1.7 s |
| 90 | 13.3 s | 2.9 s |
At 50 columns the largest method of the stage is 13,187 bytes. Numbers with
this change will follow.
### Does this PR introduce _any_ user-facing change?
No, other than the performance of stages with wide projections.
### How was this patch tested?
New tests in `WholeStageCodegenSuite`:
- a 50-output projection gets a method per output and every method of the
stage is under 8000 bytes; with `spark.sql.codegen.wholeStage.splitExpressions`
off it is past the limit;
- a projection the JIT compiles whole keeps the code it has with the split
off;
- an output that only passes on a column or a literal stays in place;
- under ANSI mode, a split output is not evaluated for a row a filter above
a nondeterministic projection drops (`10 DIV (id - 5)`);
- each output's method takes the column it reads, and a common
subexpression's locals;
- results match those without whole-stage codegen under a range of consumers
(end of the stage, aggregate, broadcast and sort-merge join, rollup, generator,
sort with limit), with subexpression elimination on and off.
`SPARK-33301: a split that turns no block into a method is not compiled` is
reshaped: in its query, the output split makes the CASE WHEN splittable (its
outputs' inputs are evaluated ahead of their calls), so the CASE WHEN and the
common subexpressions are now one output, which keeps the case it tests.
### Was this patch authored or co-authored using generative AI tooling?
Yes. Co-authored-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
