Hemanth Boyina created SPARK-61883:
--------------------------------------

             Summary: Add composite runtime Bloom filter for multi-column 
equi-joins
                 Key: SPARK-61883
                 URL: https://issues.apache.org/jira/browse/SPARK-61883
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Hemanth Boyina


InjectRuntimeFilter builds one runtime Bloom filter per equi-join key. For a 
join on two or more columns, each key is filtered independently, so an 
application-side row survives when each of its  key values appears somewhere on 
the creation side, even if that combination never occurs together.              
                                                                                
      
                                                                                
                                                                                
                                           
     {*}Example{*}: a join on (a, b) where the creation side only has pairs 
(0,0)..(9,9). The per-key filters admit (3, 7) because 3 appears in a and 7 
appears in b, although (3, 7) can never join.            
                                                                                
                                                                                
                                           
This adds an optional composite Bloom filter over the hash of the whole key 
tuple, in addition to the per-key filters, when all keys come from the same 
scan on both sides of the join. It can prune key combinations that never occur 
together on the creation side. It is gated by a new config, 
spark.sql.optimizer.runtime.bloomFilter.composite.enabled, and is off by 
default.                 
              



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to