[
https://issues.apache.org/jira/browse/SPARK-37542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
weixiuli updated SPARK-37542:
-----------------------------
Description:
Currently, the dynamic partition pruning rule will insert a predicate on the
filterable table using the filter from the other side of the join and a custom
wrapper called DynamicPruning，and the predicate will be re-optimized by the AQE
or non-AQE.
But, some time the predicate may be unnecessary if the join can NOT reuse
broadcastExchange or it is not benefit，and it will be dropped by the rules of
the AQE or non-AQE.
We should optimize the dynamic partitioning pruning rule to avoid inserting
unnecessary predicates to improve performance.
was:
Currently, the dynamic partition pruning rule will insert a predicate on the
filterable table using the filter from the other side of the join and a custom
wrapper called DynamicPruning，and the predicate will be re-optimized by the AQE
or non-AQE.
But, some time the predicate may be unnecessary if the join can NOT reuse
broadcastExchange or it is not benefit，and it will be dropped by the rules of
the AQE or non-AQE.
We should optimize the dynamic partitioning prune rules to avoid inserting
unnecessary predicates to improve performance.
> Optimize the dynamic partitioning prune rules to avoid inserting unnecessary
> predicates to improve performance
> --------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-37542
> URL: https://issues.apache.org/jira/browse/SPARK-37542
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.1.0, 3.1.1, 3.1.2, 3.2.0
> Reporter: weixiuli
> Priority: Major
>
> Currently, the dynamic partition pruning rule will insert a predicate on the
> filterable table using the filter from the other side of the join and a
> custom wrapper called DynamicPruning，and the predicate will be re-optimized
> by the AQE or non-AQE.
> But, some time the predicate may be unnecessary if the join can NOT reuse
> broadcastExchange or it is not benefit，and it will be dropped by the rules of
> the AQE or non-AQE.
> We should optimize the dynamic partitioning pruning rule to avoid inserting
> unnecessary predicates to improve performance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
