[
https://issues.apache.org/jira/browse/SPARK-48486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18147697#comment-18147697
]
Minreng Wu commented on SPARK-48486:
------------------------------------
We recently encountered the same issue with nested joins and DPP enabled. Our
investigation matches the pattern described by chenfengbin above: later DPP
subqueries contain earlier ones, causing the same filtering-side plans to be
planned and prepared repeatedly.
The earlier [PR #47911|https://github.com/apache/spark/pull/47911] proposed
limiting DPP insertion based on plan string length. Our implementation focuses
on sharing repeated construction and avoiding repeated expansion:
* Query-scoped caching at the DPP construction entry points, allowing nested
requests to reuse previously constructed plans. Cache keys retain
planning-relevant distinctions such as join hints, and broadcast eligibility
remains determined by the existing checks.
* Identity-based memoization of completed recursive rewrites in the final
reuse rule, avoiding repeated traversal of shared subquery plans.
* Rendering shared subquery bodies once in explain output and representing
repeated DPP dependencies with references in SQL event serialization, with
corresponding SQL UI graph support.
The construction cache supports AQE on/off and both values of
{{{}reuseBroadcastOnly{}}}. It applies to deterministic inputs when subquery
reuse is enabled, which is the default.
We have tested the changes on our Spark 3.4-based branch and validated them in
our deployed environment. The planning/reuse and display/event changes are
organized into separate commits so they can be reviewed independently.
PR for review: [https://github.com/apache/spark/pull/60717]
Feedback on the approach and implementation would be appreciated. We are happy
to iterate based on the review.
> The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated
> Directed Acyclic Graph (DAG) to expand.
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-48486
> URL: https://issues.apache.org/jira/browse/SPARK-48486
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.3.0
> Reporter: chenfengbin
> Priority: Major
> Labels: pull-request-available
> Attachments: create_table.sql
>
>
> The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated
> Directed Acyclic Graph (DAG) to expand.
> for example: The partition field of table A or B is part_dt.
> select a.part_dt,{*}b.part_dt{*}
> from a left join b on b.part_dt = a.part_dt
> where a.part_dt in ('2024-05-30','2024-05-31')
> During the generation of Dynamic Partition Pruning (DPP), the tree will be
> traversed recursively more times.
> 24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with:
> part_dt#219 IN (2024-05-30,2024-05-31)
> 24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with:
> part_dt#223 IN
> (2024-05-30,2024-05-31),isnotnull(part_dt#223),dynamicpruning#234
> [part_dt#223|#223]
> 24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with:
> part_dt#219 IN (2024-05-30,2024-05-31)
> The last one is extra:24/05/31 16:14:14 INFO DataSourceStrategy: Pruning
> directories with: part_dt#219 IN (2024-05-30,2024-05-31)
> When more partitions meet the condition, it will increase by 2 to the power
> of n, then divided by 2.
> for example:
> select a.part_dt{*},b.part_dt{*},c.part_dt
> from a
> left join b on b.part_dt = a.part_dt
> left join c on c.part_dt = a.part_dt
> where a.part_dt in ('2024-05-30','2024-05-31')
> result:
> 24/05/31 16:29:19 INFO ExecuteStatement: Execute in full collect mode
> 24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with:
> part_dt#249 IN (2024-05-30,2024-05-31)
> 24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with:
> part_dt#253 IN
> (2024-05-30,2024-05-31),isnotnull(part_dt#253),dynamicpruning#261
> [part_dt#253|#253]
> 24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with:
> part_dt#257 IN
> (2024-05-30,2024-05-31),isnotnull(part_dt#257),dynamicpruning#262
> [part_dt#257|#257]
> 24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with:
> part_dt#249 IN (2024-05-30,2024-05-31)
> 24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with:
> part_dt#249 IN (2024-05-30,2024-05-31)
> 24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with:
> part_dt#253 IN
> (2024-05-30,2024-05-31),isnotnull(part_dt#253),dynamicpruning#261
> [part_dt#253|#253]
> 24/05/31 16:29:19 INFO DataSourceStrategy: Pruning directories with:
> part_dt#249 IN (2024-05-30,2024-05-31)
> The last four lines are all superfluous.
> When more than 10 conditions are met, it will cause a memory overflow when
> generating DPP, and the generated plan will be about 400 times larger.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]