[
https://issues.apache.org/jira/browse/SPARK-60541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18132290#comment-18132290
]
Peter Toth commented on SPARK-60541:
------------------------------------
I'm working on this.
> Per-partition runtime bloom filters for storage-partitioned joins
> -----------------------------------------------------------------
>
> Key: SPARK-60541
> URL: https://issues.apache.org/jira/browse/SPARK-60541
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Peter Toth
> Priority: Major
>
> Runtime bloom filters ({{InjectRuntimeFilter}}) build one bloom for the whole
> filter creation side, through a scalar subquery. In a storage-partitioned
> join both sides are already split by the partition key, so each partition
> only needs the keys of the matching partition on the other side.
> This adds a per-partition variant. When both scans report a compatible
> {{KeyedPartitioning}}, the optimizer adds a left semi join of the application
> side and an aggregate grouped by the creation side's partition transform,
> which builds one bloom per partition key. The semi join plans as part of the
> storage-partitioned join, without a shuffle.
> Compared with the global bloom:
> * each bloom holds only its partition's keys, so at the same size it has a
> lower false positive rate;
> * a creation side too large for one global bloom can still be filtered;
> * there is no driver round trip and no stage boundary for the subquery.
> A follow-up can hand the per-partition bloom to the application side's scan
> as a storage filter (SPARK-60465).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
