xumingming opened a new pull request, #57826:
URL: https://github.com/apache/spark/pull/57826
### What changes were proposed in this pull request?
Extend the subexpression-elimination benchmark in `ExpandBenchmark` (added in
SPARK-58501) from two cases to a full 2x2 matrix over
`spark.sql.optimizer.rewriteCountDistinctConditional` and
`spark.sql.subexpressionElimination`:
| Case | Expand branches | daysSince evals/row |
|---|---|---|
| rewrite off, CSE off (base) | 10 | 18 |
| rewrite on, CSE off | 2 | 18 |
| rewrite on, CSE on | 2 | 1 |
| rewrite off, CSE on | 10 | 1 |
The four cases attribute the benefit of each optimization independently: the
conditional COUNT(DISTINCT) rewrite cuts data amplification (10x to 2x),
while
subexpression elimination cuts per-row subexpression evaluations (18 to 1).
Also add an optional section filter (`ratio`, `char`, `subexpr`) and a
`smoke`
mode to `runBenchmarkSuite` so a single section can be measured without
editing
the file:
```
build/sbt "sql/Test/runMain
org.apache.spark.sql.execution.benchmark.ExpandBenchmark subexpr"
build/sbt "sql/Test/runMain
org.apache.spark.sql.execution.benchmark.ExpandBenchmark smoke"
```
Regenerate `sql/core/benchmarks/ExpandBenchmark-jdk21-results.txt`.
### Why are the changes needed?
SPARK-58501 made `ExpandExec` participate in whole-stage subexpression
elimination, so a subexpression shared across Expand branches is evaluated
once
per input row instead of once per branch occurrence. The previous benchmark
only
measured the CSE toggle on the 2-branch plan produced by the conditional
rewrite. The missing cell (rewrite off, CSE on) shows the CSE benefit holds
even
without the data-amplification reduction and lets reviewers attribute the
combined speedup to each optimization.
On Apple M3 Pro / JDK 21, 5M rows:
| Case | Best (ms) | Avg (ms) | Relative |
|---|---|---|---|
| rewrite off, CSE off | 84,450 | 84,850 | 1.0X |
| rewrite on, CSE off | 41,354 | 41,544 | 2.0X |
| rewrite on, CSE on | 10,712 | 10,843 | 7.9X |
| rewrite off, CSE on | 53,281 | 54,021 | 1.6X |
### Does this PR introduce _any_ user-facing change?
No. Benchmark/test-only change.
### How was this patch tested?
- `build/sbt "sql/Test/compile"` and `build/sbt "sql/Test/scalastyle"` pass.
- Smoke run (`... ExpandBenchmark smoke`) verifies all sections run with a
single iteration on tiny data.
- Section filter verified: `... ExpandBenchmark subexpr` runs only the
subexpression-elimination section.
- Full results committed in
`sql/core/benchmarks/ExpandBenchmark-jdk21-results.txt`
(Apple M3 Pro, JDK 21). Results for other JDKs can be regenerated via the
GitHub Actions benchmark workflow:
https://spark.apache.org/developer-tools.html#github-workflow-benchmarks
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: pi 0.84.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
