This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 357e17a06a68 [SPARK-60808][4.0][SQL] Fix lost join rows when a runtime
bloom filter's creation side is non-deterministic
357e17a06a68 is described below
commit 357e17a06a68ca894158302d26cc8ebba30ec8a2
Author: Peter Toth <[email protected]>
AuthorDate: Sat Nov 7 09:50:57 2026 -0800
[SPARK-60808][4.0][SQL] Fix lost join rows when a runtime bloom filter's
creation side is non-deterministic
### Differences from #60176
This is the branch-4.0 backport of #60176. The description below is the
original one.
- The commit of the branch-4.2 backport (#60186) cherry-picks cleanly. Like
branch-4.2, branch-4.0 does not have SPARK-58310 (#57485), the check that skips
a creation side with a Python UDF, and its test. So the new check directly
follows the `aggregate` val in `injectBloomFilter`, and the new tests follow
the "simple expressions only" test. The code and the tests are otherwise the
same as on master. "Next to the existing Python UDF check" in the description
below does not hold on thi [...]
- On branch-4.0, `InjectRuntimeFilterSuite`,
`BloomFilterAggregateQuerySuite` and the plan stability suites pass (347
tests), with no plan change. Without the fix, both new tests fail.
- `dev/lint-scala` passes.
### What changes were proposed in this pull request?
`InjectRuntimeFilter` no longer builds a runtime bloom filter from a
creation side whose filter or key is non-deterministic.
The check runs on the bloom filter's subquery after column pruning, next to
the existing Python UDF check. So a non-deterministic column that the bloom
filter does not read still allows the bloom filter.
Known limit: the check sees only the non-determinism that the plan shows. A
leaf can hide it when reading the leaf again returns other rows, for example:
- a cache whose plan is not repeatable, if part of the cache is computed
again;
- a V2 scan with a pushed limit, top-N or sample;
- a `LogicalRDD` that is not a materialized checkpoint, such as the
DataFrame of a `foreachBatch` micro-batch.
DPP has the same gap, see the comment on
`PartitionPruning.isCheaplyRecomputableMaterializedPlan`. Closing it needs a
repeatability check for each leaf type. Master has one only for some leaves,
and the 4.0 to 4.2 branches have none. #58635 adds such a check for its
`RUNTIME_FILTER` hint, so the default path can use it on master in a follow-up.
### Why are the changes needed?
The bloom filter's subquery evaluates the creation side a second time,
apart from the join. With a non-deterministic filter or key, the two
evaluations can keep different rows or give different keys. The bloom filter
can then drop application side rows that match a creation side row the join
sees, and the query loses rows.
Example from the ticket, where every `dim.k` is in `fact`, so `lost` must
be 0:
```sql
CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM range(200);
CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k, id v
FROM range(20000);
SET spark.sql.autoBroadcastJoinThreshold=-1;
SET
spark.sql.optimizer.runtime.bloomFilter.applicationSideScanSizeThreshold=1;
SELECT count(*) total, count_if(f.k IS NULL) lost
FROM (SELECT k FROM dim WHERE cast(reflect('java.lang.Math', 'random') AS
DOUBLE) < 0.5) d
LEFT JOIN fact f ON d.k = f.k;
```
Before this change, three runs gave `lost` = 47, 46 and 47. After it,
`lost` is 0.
### Does this PR introduce _any_ user-facing change?
Yes. A join whose bloom filter creation side has a non-deterministic filter
or key no longer loses rows. Such a join no longer gets a bloom filter built
from that side. That includes a sampling filter like `rand() < 0.1`, even
though its two scans usually keep the same rows.
### How was this patch tested?
New tests in `InjectRuntimeFilterSuite`:
- the bloom filters of three creation sides:
- a non-deterministic filter: no bloom filter;
- a non-deterministic key: no bloom filter;
- a non-deterministic column that the bloom filter does not read: the
bloom filter stays;
- the example above, with `checkAnswer` on `lost`, and a deterministic
creation side on the same tables that gets a bloom filter.
Before this change, the first two creation sides got a bloom filter, and
the example lost rows. Checking the creation side plan instead of the pruned
subquery fails the third creation side.
`InjectRuntimeFilterSuite`, `BloomFilterAggregateQuerySuite`, the DPP
suites, the runtime filter suites, `StorageFilterPushDownSuite` and the plan
stability suites pass, with no plan change.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code with Claude Opus 5.5
Closes #60201 from
peter-toth/SPARK-60808-bloom-nondeterministic-creation-side-4.0.
Authored-by: Peter Toth <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../catalyst/optimizer/InjectRuntimeFilter.scala | 6 ++++
.../spark/sql/InjectRuntimeFilterSuite.scala | 34 ++++++++++++++++++++++
2 files changed, 40 insertions(+)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
index 176e927b2d21..1083a4a1c216 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/InjectRuntimeFilter.scala
@@ -69,6 +69,12 @@ object InjectRuntimeFilter extends Rule[LogicalPlan] with
PredicateHelper with J
val alias = Alias(bloomFilterAgg.toAggregateExpression(), "bloomFilter")()
val aggregate =
ConstantFolding(ColumnPruning(Aggregate(Nil, Seq(alias),
filterCreationSidePlan)))
+ // The subquery evaluates the creation side again, apart from the join. A
non-deterministic
+ // filter or key can give it different keys than the join sees. This check
does not see into
+ // a leaf that returns other rows when read again, like a cache whose plan
is not repeatable.
+ if (!aggregate.deterministic) {
+ return filterApplicationSidePlan
+ }
val bloomFilterSubquery = ScalarSubquery(aggregate, Nil)
val filter = BloomFilterMightContain(bloomFilterSubquery,
new XxHash64(Seq(filterApplicationSideKey)))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
index 7d7185ae6c13..5ced0082812d 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/InjectRuntimeFilterSuite.scala
@@ -517,6 +517,40 @@ class InjectRuntimeFilterSuite extends QueryTest with
SQLTestUtils with SharedSp
}
}
+ test("SPARK-60808: do not add bloom filter whose creation side is not
deterministic") {
+
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
-> "3000",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+ val queries = Seq(
+ "select * from bf2 where a2 = 62 and rand() < 0.5",
+ "select if(rand() < 0.5, c2, null) c2 from bf2 where a2 = 62",
+ // The bloom filter does not read `r`.
+ "select c2, rand() r from bf2 where a2 = 62"
+ ).map(creationSide => s"select * from bf1 join ($creationSide) t on
bf1.c1 = t.c2")
+ assert(queries.map(q =>
getNumBloomFilters(sql(q).queryExecution.optimizedPlan)) ==
+ Seq(0, 0, 1))
+ }
+ }
+
+ test("SPARK-60808: a non-deterministic creation side does not lose join
rows") {
+ withTable("dim", "fact") {
+ sql("CREATE TABLE dim USING parquet AS SELECT cast(id AS INT) k FROM
range(200)")
+ sql("CREATE TABLE fact USING parquet AS SELECT cast(id % 200 AS INT) k
FROM range(20000)")
+
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
-> "1",
+ SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "-1") {
+ // Every `dim.k` is in `fact`. `rand()` would pick the same rows in
each scan of the same
+ // files, so the filter calls `java.lang.Math.random` instead.
+ checkAnswer(sql(
+ """SELECT count_if(f.k IS NULL)
+ |FROM (SELECT k FROM dim
+ | WHERE cast(reflect('java.lang.Math', 'random') AS DOUBLE) <
0.5) d
+ |LEFT JOIN fact f ON d.k = f.k""".stripMargin), Row(0L))
+ // A deterministic creation side gets a bloom filter with these tables
and settings.
+ assert(getNumBloomFilters(sql("SELECT * FROM (SELECT k FROM dim WHERE
k < 100) d " +
+ "LEFT JOIN fact f ON d.k = f.k").queryExecution.optimizedPlan) == 1)
+ }
+ }
+ }
+
test("Support Left Semi join in row level runtime filters") {
withSQLConf(SQLConf.RUNTIME_BLOOM_FILTER_APPLICATION_SIDE_SCAN_SIZE_THRESHOLD.key
-> "3000",
SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key -> "32") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]