MaxGekk opened a new pull request, #59018:
URL: https://github.com/apache/spark/pull/59018
### What changes were proposed in this pull request?
`BenchmarkQueryTest.checkGeneratedCode` and `WholeStageCodegenSizeBenchmark`
measure the code whole-stage codegen generates for the TPC-DS (and TPC-H, SSB)
queries without running them. They plan each query over empty tables with AQE
disabled and call `doCodeGen()` on every `WholeStageCodegenExec`. With AQE
disabled, `BroadcastHashJoinExec` executes its broadcast while generating code.
Over an empty table the relation is `EmptyHashedRelation`, and `HashJoin` then
generates a one-line stub instead of the join ("If HashedRelation is empty,
hash inner join simply returns nothing"), so the rest of the stage is not
generated either. Both instruments therefore measure stubs for every stage with
a broadcast join. On master, with the benchmark's setup, 324 of the 2201 stages
of the TPC-DS v1.4 and v2.7 queries carry the stub, in 127 of the 129 queries.
This PR generates each stage's code from a copy of the stage in which every
broadcast exchange, including one under a `ReusedExchangeExec`, reads one row
of non-null default values instead of its child. The joins then generate the
code they generate for data. Only the code of the copy is used: the build side
is its own stage, found in and measured from the original plan, so the set of
stages is unchanged.
- `BenchmarkQueryTest` gets a companion object with `withRowBroadcasts` (the
copy) and `generatedCode` (every stage of a plan and its subqueries, with its
code), plus `joinsEmptyBuildSide`. `checkGeneratedCode` generates each
subtree's code through `withRowBroadcasts` and fails if a stub is left.
- `WholeStageCodegenSizeBenchmark` measures through `generatedCode` in place
of `codegenStringSeq`. It fails, instead of skipping the query, if a stub is
left, so it cannot report a stub as measured.
- The three results files are regenerated. The stage count is unchanged at
2247 and the largest method at 4962 bytes. The source code size grows by about
19% (21.37M to 25.42M characters), the summed per-stage largest method by about
11% (821K to 910K bytes), and the summed constant pool by about 6%.
A single row makes every join key unique, so a hash join generates its
unique-key form; a join whose real build side repeats keys also has the loop
over matches. That remaining difference is small next to a stub.
### Why are the changes needed?
The benchmark is the instrument for the SPARK-56908 umbrella, and it has
understated the generated code of most TPC-DS queries. `checkGeneratedCode`
checks stubs in the same way. `LogicalPlanTagInSparkPlanSuite` already runs it
with AQE disabled on master, and the other suites will once SPARK-59764
disables AQE for them.
### Does this PR introduce _any_ user-facing change?
No. Test and benchmark only.
### How was this patch tested?
By running the affected suites:
```
build/sbt "sql/testOnly
org.apache.spark.sql.execution.LogicalPlanTagInSparkPlanSuite
org.apache.spark.sql.TPCDSQuerySuite
org.apache.spark.sql.TPCDSQueryWithStatsSuite
org.apache.spark.sql.TPCDSQueryANSISuite org.apache.spark.sql.TPCHQuerySuite
org.apache.spark.sql.SSBQuerySuite"
```
All six passed, 635 tests. No stage generated in full crosses the 8000-byte
limit.
As a negative check, with the new assertion but without `withRowBroadcasts`,
147 of the 150 tests of `LogicalPlanTagInSparkPlanSuite` fail with the stub
message. That is the bug on master today.
The results files were generated with `SPARK_GENERATE_BENCHMARK_FILES=1`,
running the benchmark's `main` under JDK 17, 21 and 25 in turn. The benchmark
measures code sizes, not time, so the machine does not matter. A JDK 25 run
through sbt matched the direct run within about 0.01%, the same run-to-run
variation the committed files already show between JDKs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
