MaxGekk commented on PR #59017: URL: https://github.com/apache/spark/pull/59017#issuecomment-5816272793
A related limitation, which this PR does not change and which I filed separately as SPARK-59765: with AQE off, `BroadcastHashJoinExec` runs the broadcast while generating code. Over the empty TPC-DS tables the relation is `EmptyHashedRelation`, so the join generates a stub, and the rest of its stage isn't generated either. On master, with `WholeStageCodegenSizeBenchmark`'s setup (SF100 stats, AQE off), 324 of the 2201 stages of the TPC-DS v1.4 and v2.7 queries contain that stub, in 127 of the 129 queries. So both `checkGeneratedCode` here and `WholeStageCodegenSizeBenchmark` (SPARK-57915) measure less code than the queries compile over real data. The JIRA proposes generating each stage's code from a copy in which every broadcast exchange reads one row of non-null default values, and asserting that no stub is left. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
