[
https://issues.apache.org/jira/browse/SPARK-61883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61883:
-----------------------------------
Labels: pull-request-available (was: )
> Add composite runtime Bloom filter for multi-column equi-joins
> --------------------------------------------------------------
>
> Key: SPARK-61883
> URL: https://issues.apache.org/jira/browse/SPARK-61883
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Hemanth Boyina
> Priority: Major
> Labels: pull-request-available
>
> InjectRuntimeFilter builds one runtime Bloom filter per equi-join key. For a
> join on two or more columns, each key is filtered independently, so an
> application-side row survives when each of its key values appears somewhere
> on the creation side, even if that combination never occurs together.
>
>
>
>
>
> {*}Example{*}: a join on (a, b) where the creation side only has pairs
> (0,0)..(9,9). The per-key filters admit (3, 7) because 3 appears in a and 7
> appears in b, although (3, 7) can never join.
>
>
>
> This adds an optional composite Bloom filter over the hash of the whole key
> tuple, in addition to the per-key filters, when all keys come from the same
> scan on both sides of the join. It can prune key combinations that never
> occur together on the creation side. It is gated by a new config,
> spark.sql.optimizer.runtime.bloomFilter.composite.enabled, and is off by
> default.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]