maryannxue opened a new pull request #27554: [SPARK-30801][SQL] Subqueries 
should not be AQE-ed if main query is not
URL: https://github.com/apache/spark/pull/27554
 
 
   ### What changes were proposed in this pull request?
   This PR makes sure AQE is either enabled or disabled for the entire query, 
including the main query and all subqueries.
   Currently there are unsupported queries by AQE, e.g., queries that contain 
DPP filters. We need to make sure that if the main query is unsupported, none 
of the sub-queries should apply AQE, otherwise it can lead to performance 
regressions due to missed opportunity of sub-query reuse.
   
   ### Why are the changes needed?
   To get rid of potential perf regressions when AQE is turned on.
   
   ### Does this PR introduce any user-facing change?
   No.
   
   ### How was this patch tested?
   Updated DynamicPartitionPruningSuite:
   1. Removed the existing workaround 
`withSQLConf(SQLConf.ADAPTIVE_EXECUTION_ENABLED.key, "false")`
   2. Added `DynamicPartitionPruningSuiteAEOn` and 
`DynamicPartitionPruningSuiteAEOff` to enable testing this suite with AQE on 
and off options
   3. Added a check in `checkPartitionPruningPredicate` to verify that the 
subqueries are always in sync with the main query in terms of whether AQE is 
applied.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

Reply via email to