Yang Jie created SPARK-61173:
--------------------------------
Summary: InjectRuntimeFilter should accept a DSv2 scan with a
fully pushed selective filter as the bloom filter creation side
Key: SPARK-61173
URL: https://issues.apache.org/jira/browse/SPARK-61173
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
`InjectRuntimeFilter.extractSelectiveFilterOverScan` counts a selective
predicate only at a `Filter` and accepts a leaf only below one. A DSv2 scan
that takes a selective filter in fully leaves no `Filter` above it, so it
cannot be a bloom filter's creation side, while the same dimension read through
V1 can, since `PruneFileSourcePartitions` keeps the `Filter`. Since
SPARK-60702, a key whose DPP filter physical planning drops can get a bloom
filter, so on a shuffle join V1 gets one and a DSv2 dimension filtered only
through its scan gets no runtime filter.
The rule could also count the `pushedFilters` of a `DataSourceV2ScanRelation`,
only when `onlyRowWisePushdowns` holds, since the bloom filter reads its
creation side again. A V2 file scan reports the size of the whole table in
`estimateStatistics`, not of the partitions it reads, so
`creationSideThreshold` would still be checked against more than V1 checks.
Found in the review of #55579 (SPARK-30628).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
