[
https://issues.apache.org/jira/browse/SPARK-59764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-59764.
------------------------------
Fix Version/s: 4.4.0
Resolution: Fixed
Issue resolved by pull request 59017
[https://github.com/apache/spark/pull/59017]
> BenchmarkQueryTest.checkGeneratedCode checks nothing under adaptive query
> execution
> -----------------------------------------------------------------------------------
>
> Key: SPARK-59764
> URL: https://issues.apache.org/jira/browse/SPARK-59764
> Project: Spark
> Issue Type: Test
> Components: SQL
> Affects Versions: 3.2.0, 4.1.0, 4.0.0, 4.2.0, 4.3.0, 5.0.0
> Reporter: Max Gekk
> Assignee: Max Gekk
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> {{BenchmarkQueryTest.checkGeneratedCode}} is meant to check that every
> whole-stage codegen subtree of a benchmark query compiles and that its
> largest generated method is within
> {{CodeGenerator.DEFAULT_JVM_HUGE_METHOD_LIMIT}} (8000 bytes, HotSpot's
> {{HugeMethodLimit}}). {{TPCDSQuerySuite}} (and {{TPCDSQueryWithStatsSuite}},
> {{TPCDSQueryANSISuite}}), {{TPCHQuerySuite}} and {{SSBQuerySuite}} call it on
> {{sql(query).queryExecution.executedPlan}}.
> Since adaptive query execution became the default (SPARK-33679, 3.2.0), that
> plan is an {{AdaptiveSparkPlanExec}}, a leaf node whose query stages, and so
> its {{WholeStageCodegenExec}} nodes, are created only when the query runs.
> The suites never run the queries, so the {{plan foreach}} walk in
> {{checkGeneratedCode}} finds no {{WholeStageCodegenExec}} and the method
> returns without compiling or checking anything. The tests pass vacuously.
> Evidence: walking the same way as {{checkGeneratedCode}} (the plan plus its
> subqueries) over all TPC-DS v1.4 (103), v2.7 (32) and modified (21) queries
> and all 22 TPC-H queries finds 0 whole-stage stages with
> {{spark.sql.adaptive.enabled=true}}. With
> {{spark.sql.adaptive.enabled=false}} the same walk finds about 1,700 TPC-DS
> stages (without injected statistics) and 126 TPC-H stages, and exactly one of
> them is past 8000 bytes: {{modified-q3}}'s stage (12214 bytes; 12167 with
> injected statistics), which the suite already excludes as SPARK-29128.
> Two consequences:
> * The "can be properly compiled" check and the method size check have not run
> since 3.2.0, so a codegen regression in these queries would not be caught
> here.
> * The workarounds in {{TPCDSQuerySuite}} guard an assertion that does not
> run. {{spark.sql.readSideCharPadding=false}} ("so that the generated code is
> less than 8000") is not needed with AQE off either: padding changes the
> largest method of some queries by at most about 500 bytes, and none crosses
> 8000.
> Proposed fix: set {{spark.sql.adaptive.enabled=false}} in
> {{BenchmarkQueryTest}}'s {{sparkConf}}, since these suites only compile
> plans, and assert that at least one {{WholeStageCodegenExec}} was found, so
> the check cannot become vacuous again. Keep the {{modified-q3}} exclusion,
> and consider dropping the char padding override.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
