[ 
https://issues.apache.org/jira/browse/SPARK-61173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61173:
-----------------------------------
    Labels: pull-request-available  (was: )

> InjectRuntimeFilter should accept a DSv2 scan with a fully pushed selective 
> filter as the bloom filter creation side
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61173
>                 URL: https://issues.apache.org/jira/browse/SPARK-61173
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Yang Jie
>            Priority: Major
>              Labels: pull-request-available
>
> `InjectRuntimeFilter.extractSelectiveFilterOverScan` counts a selective 
> predicate only at a `Filter` and accepts a leaf only below one. A DSv2 scan 
> that takes a selective filter in fully leaves no `Filter` above it, so it 
> cannot be a bloom filter's creation side, while the same dimension read 
> through V1 can, since `PruneFileSourcePartitions` keeps the `Filter`. Since 
> SPARK-60702, a key whose DPP filter physical planning drops can get a bloom 
> filter, so on a shuffle join V1 gets one and a DSv2 dimension filtered only 
> through its scan gets no runtime filter.
> The rule could also count the `pushedFilters` of a 
> `DataSourceV2ScanRelation`, only when `onlyRowWisePushdowns` holds, since the 
> bloom filter reads its creation side again. A V2 file scan reports the size 
> of the whole table in `estimateStatistics`, not of the partitions it reads, 
> so `creationSideThreshold` would still be checked against more than V1 checks.
> Found in the review of #55579 (SPARK-30628).



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to