This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new 9f83f3447811 [SPARK-60808][4.2][SQL] Fix lost join rows when a runtime 
bloom filter's creation side is non-deterministic
9f83f3447811 is described below

commit 9f83f3447811b60037bf86d680be779955ad3104
Author: Peter Toth <[email protected]>
AuthorDate: Fri Nov 6 20:15:57 2026 -0800

    [SPARK-60808][4.2][SQL] Fix lost join rows when a runtime bloom filter's 
creation side is non-deterministic
    
    ### Differences from #60176
    
    This is the branch-4.2 backport of #60176. The description below is the 
original one.
    - The cherry-pick conflicted, because branch-4.2 does not have SPARK-58310 
(#57485), the check that skips a creation side with a Python UDF, and its test. 
So the new check directly follows the `aggregate` val in `injectBloomFilter`, 
and the new tests follow the "simple expressions only" test. The code and the 
tests are otherwise the same as on master. "Next to the existing Python UDF 
check" in the description below does not hold on this branch.
    - On branch-4.2, `InjectRuntimeFilterSuite`, 
`BloomFilterAggregateQuerySuite` and the plan stability suites pass (348 
tests), with no plan change. Without the fix, both new tests fail.
    - `dev/lint-scala` passes.
    - This commit also applies cleanly to branch-4.1 and branch-4.0, where 
`InjectRuntimeFilter.scala` is the same as on branch-4.2.
    
    ### What changes were proposed in this pull request?
    
    `InjectRuntimeFilter` no longer builds a runtime bloom filter from a 
creation side whose filter or key is non-deterministic.
    
    The check runs on the bloom filter's subquery after column pruning, next to 
the existing Python UDF check. So a non-deterministic column that the bloom 
filter does not read still allows the bloom filter.
    
    Known limit: the check sees only the non-determinism that the plan shows. A 
leaf can hide it when reading the leaf again returns other rows, for example:
    - a cache whose plan is not repeatable, if part of the cache is computed 
again;
    - a V2 scan with a pushed limit, top-N or sample;
    - a `LogicalRDD` that is not a materialized checkpoint, such as the 
DataFrame of a `foreachBatch` micro-batch.
    
    DPP has the same gap, see the comment on 
`PartitionPruning.isCheaplyRecomputableMaterializedPlan`. Closing it needs a 
repeatability check for each leaf type. Master has one only for some leaves, 
and the 4.0 to 4.2 branches have none. #58635 adds such a check for its 
`RUNTIME_FILTER` hint, so the default path can use it on master in a follow-up.
    
    ### Why are the changes needed?
    
    The bloom filter's subquery evaluates the creation side a second time, 
apart from the join. With a non-deterministic filter or key, the two 
evaluations can keep different rows or give different keys. The bloom filter 
can then drop application side rows that match a creation side row the join 
sees, and the query loses rows.
    
    Example from the ticket, where every `dim.k` is in `fact`, so `lost` must 
be 0:
    ```sql
    CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM range(200);
    CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k, id v 
FROM range(20000);
    SET spark.sql.autoBroadcastJoinThreshold=-1;
    SET 
spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=1;
    SELECT count(*) total, count_if(f.k IS NULL) lost
    FROM (SELECT k FROM dim WHERE cast(reflect('java.lang.Math', 'random') AS 
DOUBLE) < 0.5) d
    LEFT JOIN fact f ON d.k = f.k;
    ```
    Before this change, three runs gave `lost` = 47, 46 and 47. After it, 
`lost` is 0.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. A join whose bloom filter creation side has a non-deterministic filter 
or key no longer loses rows. Such a join no longer gets a bloom filter built 
from that side. That includes a sampling filter like `rand() < 0.1`, even 
though its two scans usually keep the same rows.
    
    ### How was this patch tested?
    
    New tests in `InjectRuntimeFilterSuite`:
    - the bloom filters of three creation sides:
      - a non-deterministic filter: no bloom filter;
      - a non-deterministic key: no bloom filter;
      - a non-deterministic column that the bloom filter does not read: the 
bloom filter stays;
    - the example above, with `checkAnswer` on `lost`, and a deterministic 
creation side on the same tables that gets a bloom filter.
    
    Before this change, the first two creation sides got a bloom filter, and 
the example lost rows. Checking the creation side plan instead of the pruned 
subquery fails the third creation side.
    
    `InjectRuntimeFilterSuite`, `BloomFilterAggregateQuerySuite`, the DPP 
suites, the runtime filter suites, `StorageFilterPushDownSuite` and the plan 
stability suites pass, with no plan change.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code with Claude Opus 5.5
    
    Closes #60186 from 
peter-toth/SPARK-60808-bloom-nondeterministic-creation-side-4.2.
    
    Authored-by: Peter Toth <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../catalyst/optimizer/InjectRuntimeFilter.scala   |  6 ++++
 .../spark/sql/InjectRuntimeFilterSuite.scala       | 34 ++++++++++++++++++++++
 2 files changed, 40 insertions(+)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
index 176e927b2d21..1083a4a1c216 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
@@ -69,6 +69,12 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with 
PredicateHelper with J
     val alias = Alias(bloomFilterAgg.toAggregateExpression(), "bloomFilter")()
     val aggregate =
       ConstantFolding(ColumnPruning(Aggregate(Nil, Seq(alias), 
filterCreationSidePlan)))
+    // The subquery evaluates the creation side again, apart from the join. A 
non-deterministic
+    // filter or key can give it different keys than the join sees. This check 
does not see into
+    // a leaf that returns other rows when read again, like a cache whose plan 
is not repeatable.
+    if (!aggregate.deterministic) {
+      return filterApplicationSidePlan
+    }
     val bloomFilterSubquery = ScalarSubquery(aggregate, Nil)
     val filter = BloomFilterMightContain(bloomFilterSubquery,
       new XxHash64(Seq(filterApplicationSideKey)))
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
index 53b0132b29ce..4d0ea85a6dfe 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
@@ -517,6 +517,40 @@ class InjectRuntimeFilterSuite extends SharedSparkSession
     }
   }
 
+  test("SPARK-60808: do not add bloom filter whose creation side is not 
deterministic") {
+    
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
 -> "3000",
+      SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+      val queries = Seq(
+        "select * from bf2 where a2 = 62 and rand() < 0.5",
+        "select if(rand() < 0.5, c2, null) c2 from bf2 where a2 = 62",
+        // The bloom filter does not read `r`.
+        "select c2, rand() r from bf2 where a2 = 62"
+      ).map(creationSide => s"select * from bf1 join ($creationSide) t on 
bf1.c1 = t.c2")
+      assert(queries.map(q => 
getNumBloomFilters(sql(q).queryExecution.optimizedPlan)) ==
+        Seq(0, 0, 1))
+    }
+  }
+
+  test("SPARK-60808: a non-deterministic creation side does not lose join 
rows") {
+    withTable("dim", "fact") {
+      sql("CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM 
range(200)")
+      sql("CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k 
FROM range(20000)")
+      
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
 -> "1",
+        SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+        // Every `dim.k` is in `fact`. `rand()` would pick the same rows in 
each scan of the same
+        // files, so the filter calls `java.lang.Math.random` instead.
+        checkAnswer(sql(
+          """SELECT count_if(f.k IS NULL)
+            |FROM (SELECT k FROM dim
+            |  WHERE cast(reflect('java.lang.Math', 'random') AS DOUBLE) < 
0.5) d
+            |LEFT JOIN fact f ON d.k = f.k""".stripMargin), Row(0L))
+        // A deterministic creation side gets a bloom filter with these tables 
and settings.
+        assert(getNumBloomFilters(sql("SELECT * FROM (SELECT k FROM dim WHERE 
k < 100) d " +
+          "LEFT JOIN fact f ON d.k = f.k").queryExecution.optimizedPlan) == 1)
+      }
+    }
+  }
+
   test("Support Left Semi join in row level runtime filters") {
     
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
 -> "3000",
       SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "32") {


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to