Peter Toth created SPARK-62098:
----------------------------------
Summary: Apply per-partition runtime bloom filters in the scan of
the application side
Key: SPARK-62098
URL: https://issues.apache.org/jira/browse/SPARK-62098
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
SPARK-60541 checks each partition of the application side against the bloom
filter of the matching partition of the creation side, in a semi join. Under a
storage-partitioned join no shuffle lies between the scan and that join, so the
semi join saves little.
This puts the per-partition bloom filters into the scan of the application side
instead, as a storage filter (SPARK-60465):
* Each split of the scan gets the bloom filter of the creation side's partition
with the split's partition key.
* A split whose partition key has no bloom filter is not read.
* The scan keeps the partitioning it reports, so it adds no shuffle and cuts no
tasks.
The semi join goes, and so do:
* the planning that judged whether it was free
({{ChoosePerPartitionRuntimeFilters}});
* {{might_contain}} reading its bloom filter from a column;
* a hash existence join reusing its bloom filter build column.
A scan that takes no storage filter gets no per-partition bloom filter.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]