viirya commented on a change in pull request #32921:
URL: https://github.com/apache/spark/pull/32921#discussion_r655832302
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/AdaptiveSparkPlanExec.scala
##########
@@ -96,6 +96,7 @@ case class AdaptiveSparkPlanExec(
@transient private val queryStageOptimizerRules: Seq[Rule[SparkPlan]] = Seq(
PlanAdaptiveDynamicPruningFilters(this),
ReuseAdaptiveSubquery(context.subqueryCache),
+ PrepareScans,
Review comment:
> Does anybody have any thoughts on this? AQE currently runs prep rules
only before creating stages. Will it be too bad to call prep rules twice? Once
before dynamic filtering and once after?
Technically, the plan after prep rules should reach a final status regarding
exchanges. Once we expect some changes to exchanges in the plan (e.g. dynamic
filtering change partitions of the stage), shouldn't we revert the stage and
re-plan the adaptive plan? In some cases, is it possible that the stage will
not exist after re-planning as the underlying partition is changed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
