This is an automated email from the ASF dual-hosted git repository.
peter-toth pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 678c475fa75d [SPARK-60808][SQL] Fix lost join rows when a runtime
bloom filter's creation side is non-deterministic
678c475fa75d is described below
commit 678c475fa75d7196f35e934600e00c808e9d9706
Author: Peter Toth <[email protected]>
AuthorDate: Fri Nov 6 20:43:37 2026 +0100
[SPARK-60808][SQL] Fix lost join rows when a runtime bloom filter's
creation side is non-deterministic
### What changes were proposed in this pull request?
`InjectRuntimeFilter` no longer builds a runtime bloom filter from a
creation side whose filter or key is non-deterministic.
The check runs on the bloom filter's subquery after column pruning, next to
the existing Python UDF check. So a non-deterministic column that the bloom
filter does not read still allows the bloom filter.
Known limit: the check sees only the non-determinism that the plan shows. A
leaf can hide it when reading the leaf again returns other rows, for example:
- a cache whose plan is not repeatable, if part of the cache is computed
again;
- a V2 scan with a pushed limit, top-N or sample;
- a `LogicalRDD` that is not a materialized checkpoint, such as the
DataFrame of a `foreachBatch` micro-batch.
DPP has the same gap, see the comment on
`PartitionPruning.isCheaplyRecomputableMaterializedPlan`. Closing it needs a
repeatability check for each leaf type. Master has one only for some leaves,
and the 4.0 to 4.2 branches have none. #58635 adds such a check for its
`RUNTIME_FILTER` hint, so the default path can use it on master in a follow-up.
### Why are the changes needed?
The bloom filter's subquery evaluates the creation side a second time,
apart from the join. With a non-deterministic filter or key, the two
evaluations can keep different rows or give different keys. The bloom filter
can then drop application side rows that match a creation side row the join
sees, and the query loses rows.
Example from the ticket, where every `dim.k` is in `fact`, so `lost` must
be 0:
```sql
CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM range(200);
CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k, id v
FROM range(20000);
SET spark.sql.autoBroadcastJoinThreshold=-1;
SET
spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=1;
SELECT count(*) total, count_if(f.k IS NULL) lost
FROM (SELECT k FROM dim WHERE cast(reflect('java.lang.Math', 'random') AS
DOUBLE) < 0.5) d
LEFT JOIN fact f ON d.k = f.k;
```
Before this change, three runs gave `lost` = 47, 46 and 47. After it,
`lost` is 0.
### Does this PR introduce _any_ user-facing change?
Yes. A join whose bloom filter creation side has a non-deterministic filter
or key no longer loses rows. Such a join no longer gets a bloom filter built
from that side. That includes a sampling filter like `rand() < 0.1`, even
though its two scans usually keep the same rows.
### How was this patch tested?
New tests in `InjectRuntimeFilterSuite`:
- the bloom filters of three creation sides:
- a non-deterministic filter: no bloom filter;
- a non-deterministic key: no bloom filter;
- a non-deterministic column that the bloom filter does not read: the
bloom filter stays;
- the example above, with `checkAnswer` on `lost`, and a deterministic
creation side on the same tables that gets a bloom filter.
Before this change, the first two creation sides got a bloom filter, and
the example lost rows. Checking the creation side plan instead of the pruned
subquery fails the third creation side.
`InjectRuntimeFilterSuite`, `BloomFilterAggregateQuerySuite`, the DPP
suites, the runtime filter suites, `StorageFilterPushDownSuite` and the plan
stability suites pass, with no plan change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code with Claude Opus 5.5
Closes #60176 from
peter-toth/SPARK-60808-bloom-nondeterministic-creation-side.
Authored-by: Peter Toth <[email protected]>
Signed-off-by: Peter Toth <[email protected]>
---
.../catalyst/optimizer/InjectRuntimeFilter.scala | 6 ++++
.../spark/sql/InjectRuntimeFilterSuite.scala | 34 ++++++++++++++++++++++
2 files changed, 40 insertions(+)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
index aef521e8d0a7..ac67141c4e60 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
@@ -88,6 +88,12 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with
PredicateHelper with J
if (aggregate.containsPattern(PYTHON_UDF)) {
return filterApplicationSidePlan
}
+ // The subquery evaluates the creation side again, apart from the join. A
non-deterministic
+ // filter or key can give it different keys than the join sees. This check
does not see into
+ // a leaf that returns other rows when read again, like a cache whose plan
is not repeatable.
+ if (!aggregate.deterministic) {
+ return filterApplicationSidePlan
+ }
val bloomFilterSubquery = ScalarSubquery(aggregate, Nil)
val filter = BloomFilterMightContain(bloomFilterSubquery,
new XxHash64(Seq(filterApplicationSideKey)))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
index beabadd17829..62b1e42cc565 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
@@ -1108,6 +1108,40 @@ class InjectRuntimeFilterSuite extends SharedSparkSession
}
}
+ test("SPARK-60808: do not add bloom filter whose creation side is not
deterministic") {
+
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
-> "3000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+ val queries = Seq(
+ "select * from bf2 where a2 = 62 and rand() < 0.5",
+ "select if(rand() < 0.5, c2, null) c2 from bf2 where a2 = 62",
+ // The bloom filter does not read `r`.
+ "select c2, rand() r from bf2 where a2 = 62"
+ ).map(creationSide => s"select * from bf1 join ($creationSide) t on
bf1.c1 = t.c2")
+ assert(queries.map(q =>
getNumBloomFilters(sql(q).queryExecution.optimizedPlan)) ==
+ Seq(0, 0, 1))
+ }
+ }
+
+ test("SPARK-60808: a non-deterministic creation side does not lose join
rows") {
+ withTable("dim", "fact") {
+ sql("CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM
range(200)")
+ sql("CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k
FROM range(20000)")
+
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
-> "1",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+ // Every `dim.k` is in `fact`. `rand()` would pick the same rows in
each scan of the same
+ // files, so the filter calls `java.lang.Math.random` instead.
+ checkAnswer(sql(
+ """SELECT count_if(f.k IS NULL)
+ |FROM (SELECT k FROM dim
+ | WHERE cast(reflect('java.lang.Math', 'random') AS DOUBLE) <
0.5) d
+ |LEFT JOIN fact f ON d.k = f.k""".stripMargin), Row(0L))
+ // A deterministic creation side gets a bloom filter with these tables
and settings.
+ assert(getNumBloomFilters(sql("SELECT * FROM (SELECT k FROM dim WHERE
k < 100) d " +
+ "LEFT JOIN fact f ON d.k = f.k").queryExecution.optimizedPlan) == 1)
+ }
+ }
+ }
+
test("SPARK-58310: skip runtime bloom filter when pruned creation side
contains Python UDF") {
import IntegratedUDFTestUtils._
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]