Peter Toth created SPARK-60541:
----------------------------------
Summary: Per-partition runtime bloom filters for
storage-partitioned joins
Key: SPARK-60541
URL: https://issues.apache.org/jira/browse/SPARK-60541
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Peter Toth
Runtime bloom filters ({{InjectRuntimeFilter}}) build one bloom for the whole
filter creation side, through a scalar subquery. In a storage-partitioned join
both sides are already split by the partition key, so each partition only needs
the keys of the matching partition on the other side.
This adds a per-partition variant. When both scans report a compatible
{{KeyedPartitioning}}, the optimizer adds a left semi join of the application
side and an aggregate grouped by the creation side's partition transform, which
builds one bloom per partition key. The semi join plans as part of the
storage-partitioned join, without a shuffle.
Compared with the global bloom:
* each bloom holds only its partition's keys, so at the same size it has a
lower false positive rate;
* a creation side too large for one global bloom can still be filtered;
* there is no driver round trip and no stage boundary for the subquery.
A follow-up can hand the per-partition bloom to the application side's scan as
a storage filter (SPARK-60465).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
