[
https://issues.apache.org/jira/browse/SPARK-61173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61173:
-----------------------------------
Labels: pull-request-available (was: )
> InjectRuntimeFilter should accept a DSv2 scan with a fully pushed selective
> filter as the bloom filter creation side
> --------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61173
> URL: https://issues.apache.org/jira/browse/SPARK-61173
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Yang Jie
> Priority: Major
> Labels: pull-request-available
>
> `InjectRuntimeFilter.extractSelectiveFilterOverScan` counts a selective
> predicate only at a `Filter` and accepts a leaf only below one. A DSv2 scan
> that takes a selective filter in fully leaves no `Filter` above it, so it
> cannot be a bloom filter's creation side, while the same dimension read
> through V1 can, since `PruneFileSourcePartitions` keeps the `Filter`. Since
> SPARK-60702, a key whose DPP filter physical planning drops can get a bloom
> filter, so on a shuffle join V1 gets one and a DSv2 dimension filtered only
> through its scan gets no runtime filter.
> The rule could also count the `pushedFilters` of a
> `DataSourceV2ScanRelation`, only when `onlyRowWisePushdowns` holds, since the
> bloom filter reads its creation side again. A V2 file scan reports the size
> of the whole table in `estimateStatistics`, not of the partitions it reads,
> so `creationSideThreshold` would still be checked against more than V1 checks.
> Found in the review of #55579 (SPARK-30628).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]