weixiuli opened a new pull request #31941: URL: https://github.com/apache/spark/pull/31941
### What changes were proposed in this pull request? Both AQE and DPP can be applied at the same time in https://issues.apache.org/jira/browse/SPARK-34168, while AQE and DPP can only enable when the join is Broadcast hash join at the beginning. This pr supports Dynamic Partition Pruning in Adaptive Execution can be applied at the same time in more scenarios , the processing idea is as follows: 1. Firstly, we should check whether the inputPlan has a BroadcastHashJoinExec of the build side before the adaptive dynamic pruning optimizer rule. 2. When the above broadcastHashJoinExec exists but the broadcastQueryStage of the build side of join has't created, we should create a broadcastQueryStage of the build side for the DPP optimizer, and cache it for AQE reuse. 3. When the above broadcastHashJoinExec exists and the broadcastQueryStage of the build side of join has created, we can reuse it again for the DPP optimizer 4. When the above broadcastHashJoinExec does't exist, we should forback the DPP optimizer. ### Why are the changes needed? To support Dynamic Partition Pruning in Adaptive Execution can be applied at the same time in more scenarios. ### Does this PR introduce _any_ user-facing change? NO ### How was this patch tested? Add unittests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
