chenfengbin created SPARK-48486:
-----------------------------------
Summary: The Dynamic Partition Pruning (DPP) feature in Spark can
cause the generated Directed Acyclic Graph (DAG) to expand.
Key: SPARK-48486
URL: https://issues.apache.org/jira/browse/SPARK-48486
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 3.3.0
Reporter: chenfengbin
The Dynamic Partition Pruning (DPP) feature in Spark can cause the generated
Directed Acyclic Graph (DAG) to expand.
for example: The partition field of table A or B is part_dt.
select a.part_dt,a.*,b.part_dt,b.*
from a left join b on b.part_dt = a.part_dt
where a.part_dt in ('2024-05-30','2024-05-31')
During the generation of Dynamic Partition Pruning (DPP), the tree will be
traversed recursively more times.
24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with:
part_dt#219 IN (2024-05-30,2024-05-31)
24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with:
part_dt#223 IN
(2024-05-30,2024-05-31),isnotnull(part_dt#223),dynamicpruning#234 [part_dt#223]
24/05/31 16:14:14 INFO DataSourceStrategy: Pruning directories with:
part_dt#219 IN (2024-05-30,2024-05-31)
The last one is extra：24/05/31 16:14:14 INFO DataSourceStrategy: Pruning
directories with: part_dt#219 IN (2024-05-30,2024-05-31)
When more partitions meet the condition, it will increase by 2 to the power of
n, then divided by 2.
for example:
select a.part_dt,a.*,b.part_dt,b.*
from a left join b on b.part_dt = a.part_dt where a.part_dt in
('2024-05-30','2024-05-31')
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
