[
https://issues.apache.org/jira/browse/SPARK-37542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weixiuli updated SPARK-37542:
-----------------------------
Description:
Currently, the dynamic partition pruning rule will insert a predicate on the
filterable table using the filter from the other side of the join and a custom
wrapper called DynamicPruning，and the predicate will be re-optimized by the AQE
or non-AQE.
But, some time the predicate may be unnecessary if the join can NOT reuse
broadcastExchange or it is not benefit，and it will be dropped by the rules of
the AQE or non-AQE.
We should optimize the dynamic partition pruning rule and avoid inserting
unnecessary predicate to improve the performance.
was:
Currently, the dynamic partition pruning rule will insert a predicate on the
filterable table using the filter from the other side of the join and a custom
wrapper called DynamicPruning，and the predicate will be re-optimized by the AQE
or non-AQE.
But, some time the predicate may be unnecessary if the join can NOT reuse
broadcastExchange or it is not benefit，and it will be dropped by the rules of
the AQE or non-AQE.
We should optimize the PartitionPruning and avoid inserting unnecessary
predicate to improve the performance.
> Improve the Dynamic partition pruning
> --------------------------------------
>
> Key: SPARK-37542
> URL: https://issues.apache.org/jira/browse/SPARK-37542
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.1.0, 3.1.1, 3.1.2, 3.2.0
> Reporter: weixiuli
> Priority: Major
>
> Currently, the dynamic partition pruning rule will insert a predicate on the
> filterable table using the filter from the other side of the join and a
> custom wrapper called DynamicPruning，and the predicate will be re-optimized
> by the AQE or non-AQE.
> But, some time the predicate may be unnecessary if the join can NOT reuse
> broadcastExchange or it is not benefit，and it will be dropped by the rules of
> the AQE or non-AQE.
> We should optimize the dynamic partition pruning rule and avoid inserting
> unnecessary predicate to improve the performance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
