Hemanth Boyina created SPARK-61447:
--------------------------------------

             Summary: Avoid exponential planning cost for nested dynamic 
pruning subqueries with AQE
                 Key: SPARK-61447
                 URL: https://issues.apache.org/jira/browse/SPARK-61447
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Hemanth Boyina


With adaptive query execution enabled, a query that joins a partitioned table 
to many other partitioned tables through a chain of left joins spends 
exponential time and memory in planning.          
                                                                                
                                                                                
                                           
PartitionPruning adds a dynamic pruning filter on each right-side table. The 
subquery of the i-th filter contains the left side of the join, which already 
contains the filters from joins 1 to i-1.  So the filters nest inside each 
other, and the optimized logical plan stays linear (about 5 nodes per join).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to