cloud-fan commented on a change in pull request #26207: [SPARK-29552][SQL]
Execute the "OptimizeLocalShuffleReader" rule when creating new query stage and
then can optimize the shuffle reader to local shuffle reader as much as
possible.
URL: https://github.com/apache/spark/pull/26207#discussion_r337854155
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -92,6 +92,14 @@ case class AdaptiveSparkPlanExec(
// optimizations should be stage-independent.
@transient private val queryStageOptimizerRules: Seq[Rule[SparkPlan]] = Seq(
ReuseAdaptiveSubquery(conf, subqueryCache),
+ // We will revert the all local shuffle reader node in
OptimizeLocalShuffleReader rule
Review comment:
To polish it a little bit:
```
When adding local shuffle readers in `OptimizeLocalShuffleReader`, we revert
all the local
readers if additional shuffles are introduced. This may be too conservative:
maybe there is
only one local reader that introduces shuffle, and we can still keep other
local readers.
Here we re-execute this rule with the sub-plan-tree of a query stage, to
make sure necessary
local readers are added before executing the query stage.
This rule must be executed before `ReduceNumShufflePartitions`, as local
shuffle readers can't
change number of partitions.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
