Hemanth Boyina created SPARK-61883:
--------------------------------------
Summary: Add composite runtime Bloom filter for multi-column
equi-joins
Key: SPARK-61883
URL: https://issues.apache.org/jira/browse/SPARK-61883
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Hemanth Boyina
InjectRuntimeFilter builds one runtime Bloom filter per equi-join key. For a
join on two or more columns, each key is filtered independently, so an
application-side row survives when each of its key values appears somewhere on
the creation side, even if that combination never occurs together.
{*}Example{*}: a join on (a, b) where the creation side only has pairs
(0,0)..(9,9). The per-key filters admit (3, 7) because 3 appears in a and 7
appears in b, although (3, 7) can never join.
This adds an optional composite Bloom filter over the hash of the whole key
tuple, in addition to the per-key filters, when all keys come from the same
scan on both sides of the join. It can prune key combinations that never occur
together on the creation side. It is gated by a new config,
spark.sql.optimizer.runtime.bloomFilter.composite.enabled, and is off by
default.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]