[ 
https://issues.apache.org/jira/browse/SPARK-61883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61883:
-----------------------------------
    Labels: pull-request-available  (was: )

> Add composite runtime Bloom filter for multi-column equi-joins
> --------------------------------------------------------------
>
>                 Key: SPARK-61883
>                 URL: https://issues.apache.org/jira/browse/SPARK-61883
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Hemanth Boyina
>            Priority: Major
>              Labels: pull-request-available
>
> InjectRuntimeFilter builds one runtime Bloom filter per equi-join key. For a 
> join on two or more columns, each key is filtered independently, so an 
> application-side row survives when each of its  key values appears somewhere 
> on the creation side, even if that combination never occurs together.         
>                                                                               
>              
>                                                                               
>                                                                               
>                                                
>      {*}Example{*}: a join on (a, b) where the creation side only has pairs 
> (0,0)..(9,9). The per-key filters admit (3, 7) because 3 appears in a and 7 
> appears in b, although (3, 7) can never join.            
>                                                                               
>                                                                               
>                                                
> This adds an optional composite Bloom filter over the hash of the whole key 
> tuple, in addition to the per-key filters, when all keys come from the same 
> scan on both sides of the join. It can prune key combinations that never 
> occur together on the creation side. It is gated by a new config, 
> spark.sql.optimizer.runtime.bloomFilter.composite.enabled, and is off by 
> default.                 
>               



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to