This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 9f83f3447811 [SPARK-60808][4.2][SQL] Fix lost join rows when a runtime
bloom filter's creation side is non-deterministic
9f83f3447811 is described below
commit 9f83f3447811b60037bf86d680be779955ad3104
Author: Peter Toth <[email protected]>
AuthorDate: Fri Nov 6 20:15:57 2026 -0800
[SPARK-60808][4.2][SQL] Fix lost join rows when a runtime bloom filter's
creation side is non-deterministic
### Differences from #60176
This is the branch-4.2 backport of #60176. The description below is the
original one.
- The cherry-pick conflicted, because branch-4.2 does not have SPARK-58310
(#57485), the check that skips a creation side with a Python UDF, and its test.
So the new check directly follows the `aggregate` val in `injectBloomFilter`,
and the new tests follow the "simple expressions only" test. The code and the
tests are otherwise the same as on master. "Next to the existing Python UDF
check" in the description below does not hold on this branch.
- On branch-4.2, `InjectRuntimeFilterSuite`,
`BloomFilterAggregateQuerySuite` and the plan stability suites pass (348
tests), with no plan change. Without the fix, both new tests fail.
- `dev/lint-scala` passes.
- This commit also applies cleanly to branch-4.1 and branch-4.0, where
`InjectRuntimeFilter.scala` is the same as on branch-4.2.
### What changes were proposed in this pull request?
`InjectRuntimeFilter` no longer builds a runtime bloom filter from a
creation side whose filter or key is non-deterministic.
The check runs on the bloom filter's subquery after column pruning, next to
the existing Python UDF check. So a non-deterministic column that the bloom
filter does not read still allows the bloom filter.
Known limit: the check sees only the non-determinism that the plan shows. A
leaf can hide it when reading the leaf again returns other rows, for example:
- a cache whose plan is not repeatable, if part of the cache is computed
again;
- a V2 scan with a pushed limit, top-N or sample;
- a `LogicalRDD` that is not a materialized checkpoint, such as the
DataFrame of a `foreachBatch` micro-batch.
DPP has the same gap, see the comment on
`PartitionPruning.isCheaplyRecomputableMaterializedPlan`. Closing it needs a
repeatability check for each leaf type. Master has one only for some leaves,
and the 4.0 to 4.2 branches have none. #58635 adds such a check for its
`RUNTIME_FILTER` hint, so the default path can use it on master in a follow-up.
### Why are the changes needed?
The bloom filter's subquery evaluates the creation side a second time,
apart from the join. With a non-deterministic filter or key, the two
evaluations can keep different rows or give different keys. The bloom filter
can then drop application side rows that match a creation side row the join
sees, and the query loses rows.
Example from the ticket, where every `dim.k` is in `fact`, so `lost` must
be 0:
```sql
CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM range(200);
CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k, id v
FROM range(20000);
SET spark.sql.autoBroadcastJoinThreshold=-1;
SET
spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=1;
SELECT count(*) total, count_if(f.k IS NULL) lost
FROM (SELECT k FROM dim WHERE cast(reflect('java.lang.Math', 'random') AS
DOUBLE) < 0.5) d
LEFT JOIN fact f ON d.k = f.k;
```
Before this change, three runs gave `lost` = 47, 46 and 47. After it,
`lost` is 0.
### Does this PR introduce _any_ user-facing change?
Yes. A join whose bloom filter creation side has a non-deterministic filter
or key no longer loses rows. Such a join no longer gets a bloom filter built
from that side. That includes a sampling filter like `rand() < 0.1`, even
though its two scans usually keep the same rows.
### How was this patch tested?
New tests in `InjectRuntimeFilterSuite`:
- the bloom filters of three creation sides:
- a non-deterministic filter: no bloom filter;
- a non-deterministic key: no bloom filter;
- a non-deterministic column that the bloom filter does not read: the
bloom filter stays;
- the example above, with `checkAnswer` on `lost`, and a deterministic
creation side on the same tables that gets a bloom filter.
Before this change, the first two creation sides got a bloom filter, and
the example lost rows. Checking the creation side plan instead of the pruned
subquery fails the third creation side.
`InjectRuntimeFilterSuite`, `BloomFilterAggregateQuerySuite`, the DPP
suites, the runtime filter suites, `StorageFilterPushDownSuite` and the plan
stability suites pass, with no plan change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code with Claude Opus 5.5
Closes #60186 from
peter-toth/SPARK-60808-bloom-nondeterministic-creation-side-4.2.
Authored-by: Peter Toth <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../catalyst/optimizer/InjectRuntimeFilter.scala | 6 ++++
.../spark/sql/InjectRuntimeFilterSuite.scala | 34 ++++++++++++++++++++++
2 files changed, 40 insertions(+)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
index 176e927b2d21..1083a4a1c216 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
@@ -69,6 +69,12 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with
PredicateHelper with J
val alias = Alias(bloomFilterAgg.toAggregateExpression(), "bloomFilter")()
val aggregate =
ConstantFolding(ColumnPruning(Aggregate(Nil, Seq(alias),
filterCreationSidePlan)))
+ // The subquery evaluates the creation side again, apart from the join. A
non-deterministic
+ // filter or key can give it different keys than the join sees. This check
does not see into
+ // a leaf that returns other rows when read again, like a cache whose plan
is not repeatable.
+ if (!aggregate.deterministic) {
+ return filterApplicationSidePlan
+ }
val bloomFilterSubquery = ScalarSubquery(aggregate, Nil)
val filter = BloomFilterMightContain(bloomFilterSubquery,
new XxHash64(Seq(filterApplicationSideKey)))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
index 53b0132b29ce..4d0ea85a6dfe 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
@@ -517,6 +517,40 @@ class InjectRuntimeFilterSuite extends SharedSparkSession
}
}
+ test("SPARK-60808: do not add bloom filter whose creation side is not
deterministic") {
+
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
-> "3000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+ val queries = Seq(
+ "select * from bf2 where a2 = 62 and rand() < 0.5",
+ "select if(rand() < 0.5, c2, null) c2 from bf2 where a2 = 62",
+ // The bloom filter does not read `r`.
+ "select c2, rand() r from bf2 where a2 = 62"
+ ).map(creationSide => s"select * from bf1 join ($creationSide) t on
bf1.c1 = t.c2")
+ assert(queries.map(q =>
getNumBloomFilters(sql(q).queryExecution.optimizedPlan)) ==
+ Seq(0, 0, 1))
+ }
+ }
+
+ test("SPARK-60808: a non-deterministic creation side does not lose join
rows") {
+ withTable("dim", "fact") {
+ sql("CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM
range(200)")
+ sql("CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k
FROM range(20000)")
+
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
-> "1",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+ // Every `dim.k` is in `fact`. `rand()` would pick the same rows in
each scan of the same
+ // files, so the filter calls `java.lang.Math.random` instead.
+ checkAnswer(sql(
+ """SELECT count_if(f.k IS NULL)
+ |FROM (SELECT k FROM dim
+ | WHERE cast(reflect('java.lang.Math', 'random') AS DOUBLE) <
0.5) d
+ |LEFT JOIN fact f ON d.k = f.k""".stripMargin), Row(0L))
+ // A deterministic creation side gets a bloom filter with these tables
and settings.
+ assert(getNumBloomFilters(sql("SELECT * FROM (SELECT k FROM dim WHERE
k < 100) d " +
+ "LEFT JOIN fact f ON d.k = f.k").queryExecution.optimizedPlan) == 1)
+ }
+ }
+ }
+
test("Support Left Semi join in row level runtime filters") {
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
-> "3000",
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "32") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]