maryannxue commented on a change in pull request #27452: [SPARK-30719][SQL] do
not log warning if AQE is intentionally skipped
URL: https://github.com/apache/spark/pull/27452#discussion_r374745725
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/InsertAdaptiveSparkPlan.scala
##########
@@ -40,49 +40,64 @@ case class InsertAdaptiveSparkPlan(
private val conf = adaptiveExecutionContext.session.sessionState.conf
- def containShuffle(plan: SparkPlan): Boolean = {
+ private def mayContainExchange(plan: SparkPlan): Boolean = {
plan.find {
case _: Exchange => true
case s: SparkPlan => !s.requiredChildDistribution.forall(_ ==
UnspecifiedDistribution)
}.isDefined
}
- def containSubQuery(plan: SparkPlan): Boolean = {
+ private def containSubQuery(plan: SparkPlan): Boolean = {
plan.find(_.expressions.exists(_.find {
case _: SubqueryExpression => true
case _ => false
}.isDefined)).isDefined
}
+ // AQE is only useful when the query has exchanges or sub-queries. This
method returns true if one
+ // of the following conditions is satisfied:
+ // - The config ADAPTIVE_EXECUTION_FORCE_APPLY is true.
+ // - The input query is from a sub-query. When this happens, it means
we've already decided to
+ // apply AQE for the main query and we must continue to do it.
+ // - The query may contains exchange. The exchanges are not added yet at
this point and we can
+ // only know if the query may contain exchange or not by checking
+ // `SparkPlan.requiredChildDistribution`.
+ // - The query contains sub-query.
+ private def shouldApplyAQE(plan: SparkPlan, isSubquery: Boolean): Boolean = {
Review comment:
How about we just move the logic of `mayContainExchange` and
`containSubquery` here and get rid of those methods?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
