This is an automated email from the ASF dual-hosted git repository.

peter-toth pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 678c475fa75d [SPARK-60808][SQL] Fix lost join rows when a runtime 
bloom filter's creation side is non-deterministic
678c475fa75d is described below

commit 678c475fa75d7196f35e934600e00c808e9d9706
Author: Peter Toth <[email protected]>
AuthorDate: Fri Nov 6 20:43:37 2026 +0100

    [SPARK-60808][SQL] Fix lost join rows when a runtime bloom filter's 
creation side is non-deterministic
    
    ### What changes were proposed in this pull request?
    
    `InjectRuntimeFilter` no longer builds a runtime bloom filter from a 
creation side whose filter or key is non-deterministic.
    
    The check runs on the bloom filter's subquery after column pruning, next to 
the existing Python UDF check. So a non-deterministic column that the bloom 
filter does not read still allows the bloom filter.
    
    Known limit: the check sees only the non-determinism that the plan shows. A 
leaf can hide it when reading the leaf again returns other rows, for example:
    - a cache whose plan is not repeatable, if part of the cache is computed 
again;
    - a V2 scan with a pushed limit, top-N or sample;
    - a `LogicalRDD` that is not a materialized checkpoint, such as the 
DataFrame of a `foreachBatch` micro-batch.
    
    DPP has the same gap, see the comment on 
`PartitionPruning.isCheaplyRecomputableMaterializedPlan`. Closing it needs a 
repeatability check for each leaf type. Master has one only for some leaves, 
and the 4.0 to 4.2 branches have none. #58635 adds such a check for its 
`RUNTIME_FILTER` hint, so the default path can use it on master in a follow-up.
    
    ### Why are the changes needed?
    
    The bloom filter's subquery evaluates the creation side a second time, 
apart from the join. With a non-deterministic filter or key, the two 
evaluations can keep different rows or give different keys. The bloom filter 
can then drop application side rows that match a creation side row the join 
sees, and the query loses rows.
    
    Example from the ticket, where every `dim.k` is in `fact`, so `lost` must 
be 0:
    ```sql
    CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM range(200);
    CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k, id v 
FROM range(20000);
    SET spark.sql.autoBroadcastJoinThreshold=-1;
    SET 
spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=1;
    SELECT count(*) total, count_if(f.k IS NULL) lost
    FROM (SELECT k FROM dim WHERE cast(reflect('java.lang.Math', 'random') AS 
DOUBLE) < 0.5) d
    LEFT JOIN fact f ON d.k = f.k;
    ```
    Before this change, three runs gave `lost` = 47, 46 and 47. After it, 
`lost` is 0.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A join whose bloom filter creation side has a non-deterministic filter 
or key no longer loses rows. Such a join no longer gets a bloom filter built 
from that side. That includes a sampling filter like `rand() < 0.1`, even 
though its two scans usually keep the same rows.
    
    ### How was this patch tested?
    
    New tests in `InjectRuntimeFilterSuite`:
    - the bloom filters of three creation sides:
      - a non-deterministic filter: no bloom filter;
      - a non-deterministic key: no bloom filter;
      - a non-deterministic column that the bloom filter does not read: the 
bloom filter stays;
    - the example above, with `checkAnswer` on `lost`, and a deterministic 
creation side on the same tables that gets a bloom filter.
    
    Before this change, the first two creation sides got a bloom filter, and 
the example lost rows. Checking the creation side plan instead of the pruned 
subquery fails the third creation side.
    
    `InjectRuntimeFilterSuite`, `BloomFilterAggregateQuerySuite`, the DPP 
suites, the runtime filter suites, `StorageFilterPushDownSuite` and the plan 
stability suites pass, with no plan change.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code with Claude Opus 5.5
    
    Closes #60176 from 
peter-toth/SPARK-60808-bloom-nondeterministic-creation-side.
    
    Authored-by: Peter Toth <[email protected]>
    Signed-off-by: Peter Toth <[email protected]>
---
 .../catalyst/optimizer/InjectRuntimeFilter.scala   |  6 ++++
 .../spark/sql/InjectRuntimeFilterSuite.scala       | 34 ++++++++++++++++++++++
 2 files changed, 40 insertions(+)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
index aef521e8d0a7..ac67141c4e60 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
@@ -88,6 +88,12 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with 
PredicateHelper with J
     if (aggregate.containsPattern(PYTHON_UDF)) {
       return filterApplicationSidePlan
     }
+    // The subquery evaluates the creation side again, apart from the join. A 
non-deterministic
+    // filter or key can give it different keys than the join sees. This check 
does not see into
+    // a leaf that returns other rows when read again, like a cache whose plan 
is not repeatable.
+    if (!aggregate.deterministic) {
+      return filterApplicationSidePlan
+    }
     val bloomFilterSubquery = ScalarSubquery(aggregate, Nil)
     val filter = BloomFilterMightContain(bloomFilterSubquery,
       new XxHash64(Seq(filterApplicationSideKey)))
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
index beabadd17829..62b1e42cc565 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
@@ -1108,6 +1108,40 @@ class InjectRuntimeFilterSuite extends SharedSparkSession
     }
   }
 
+  test("SPARK-60808: do not add bloom filter whose creation side is not 
deterministic") {
+    
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
 -> "3000",
+      SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+      val queries = Seq(
+        "select * from bf2 where a2 = 62 and rand() < 0.5",
+        "select if(rand() < 0.5, c2, null) c2 from bf2 where a2 = 62",
+        // The bloom filter does not read `r`.
+        "select c2, rand() r from bf2 where a2 = 62"
+      ).map(creationSide => s"select * from bf1 join ($creationSide) t on 
bf1.c1 = t.c2")
+      assert(queries.map(q => 
getNumBloomFilters(sql(q).queryExecution.optimizedPlan)) ==
+        Seq(0, 0, 1))
+    }
+  }
+
+  test("SPARK-60808: a non-deterministic creation side does not lose join 
rows") {
+    withTable("dim", "fact") {
+      sql("CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM 
range(200)")
+      sql("CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k 
FROM range(20000)")
+      
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
 -> "1",
+        SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+        // Every `dim.k` is in `fact`. `rand()` would pick the same rows in 
each scan of the same
+        // files, so the filter calls `java.lang.Math.random` instead.
+        checkAnswer(sql(
+          """SELECT count_if(f.k IS NULL)
+            |FROM (SELECT k FROM dim
+            |  WHERE cast(reflect('java.lang.Math', 'random') AS DOUBLE) < 
0.5) d
+            |LEFT JOIN fact f ON d.k = f.k""".stripMargin), Row(0L))
+        // A deterministic creation side gets a bloom filter with these tables 
and settings.
+        assert(getNumBloomFilters(sql("SELECT * FROM (SELECT k FROM dim WHERE 
k < 100) d " +
+          "LEFT JOIN fact f ON d.k = f.k").queryExecution.optimizedPlan) == 1)
+      }
+    }
+  }
+
   test("SPARK-58310: skip runtime bloom filter when pruned creation side 
contains Python UDF") {
     import IntegratedUDFTestUtils._
 


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to