weixiuli created SPARK-37542:
--------------------------------
Summary: Improve the Dynamic partition pruning
Key: SPARK-37542
URL: https://issues.apache.org/jira/browse/SPARK-37542
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.0, 3.1.2, 3.1.1, 3.1.0, 3.0.3, 3.0.2, 3.0.1, 3.0.0
Reporter: weixiuli
Currently, the dynamic partition pruning rule will insert a predicate on the
filterable table using the filter from the other side of the join and a custom
wrapper called DynamicPruning，and the predicate will be re-optimized by the AQE
or non-AQE.
But, some time the predicate may be unnecessary if the join can NOT reuse
broadcastExchange or it is not benefit，and it will be dropped by the rule of
the AQE rule or non-AQE.
We should optimize the PartitionPruning and avoid insert unnecessary predicate
to improve the performance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
