This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new dcad297056ab [SPARK-60980][SQL] Fix query failures by `CombineFilters`
when a condition that can fail follows a non-deterministic or throwable one
dcad297056ab is described below
commit dcad297056abde10e29491be847941b8e32132ab
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Nov 5 23:47:47 2026 -0800
[SPARK-60980][SQL] Fix query failures by `CombineFilters` when a condition
that can fail follows a non-deterministic or throwable one
### What changes were proposed in this pull request?
This PR fixes `CombineFilters` so that it does not move a condition that
can fail, such as a division in ANSI mode, ahead of a non-deterministic or
throwable condition written before it.
`CombineFilters` combines `Filter(fc, Filter(nc, child))` by adding
conjuncts of the outer condition `fc` to the inner filter, which evaluates them
first, and keeping the others in a filter above it. Since #32405 (SPARK-35273),
it keeps only the non-deterministic and throwable conjuncts above. It moves
every other conjunct into the inner filter with `partition`, no matter where it
is written. So a conjunct written after a non-deterministic or throwable one is
evaluated ahead of it, on [...]
This PR splits the outer conjuncts with `span` instead.
| Outer conjunct | Before | After |
|---|---|---|
| Written before the first non-deterministic or throwable conjunct |
Combined | Combined |
| Non-deterministic or throwable | Kept above | Kept above |
| Written after it and cannot fail, e.g. `p = 1` | Combined | Combined |
| Written after it and can fail, e.g. `10L / (id - 3) > 0` | Combined |
Kept above |
A conjunct cannot fail when `ExprUtils.canEvaluateUnconditionally` accepts
it: comparisons, `IN`, `IS NULL` and so on over columns and literals. Moving
such a conjunct ahead only reduces the rows that the other conjuncts see. This
keeps the partition pruning and filter pushdown that #32405 added for a
condition like `rand() < 0.5 AND p = 1`. The conjuncts kept above stay in their
written order.
Spark does not guarantee the evaluation order of conjuncts in general.
However, `CombineFilters` already keeps a throwable conjunct after the
conjuncts written before it, and #59848 (SPARK-60602) and #59944 (SPARK-60633)
do the same for filter pushdown. This PR covers the conjuncts that can fail
without being throwable.
### Why are the changes needed?
A query that should succeed fails. With ANSI mode, the default since Apache
Spark 4.0.0, a division or a cast is enough.
```sql
-- Issue 1: It should return no row, but fails with DIVIDE_BY_ZERO.
SELECT id FROM (SELECT id FROM range(0, 10, 1, 1) WHERE id >= 0)
WHERE rand(10) * 10 > 100 AND 10L / (id - 3) > 0;
-- Issue 2: It should return 5, 6, 7, 8 and 9, but fails with
DIVIDE_BY_ZERO.
SELECT id FROM (SELECT id FROM range(0, 10, 1, 2) WHERE id >= 0)
WHERE spark_partition_id() = 1 AND 10L / (id - 3) > 0;
```
`rand(10) * 10 > 100` is false on every row, and `spark_partition_id() = 1`
is false for `id = 3`, so the division should never run for `id = 3`. Both
queries succeed without the inner `WHERE id >= 0`. With it, `CombineFilters`
moves the division into the inner filter, ahead of the non-deterministic
conjunct. A non-deterministic UDF or a throwable conjunct written first, such
as a `CASE WHEN` with `raise_error`, causes the same failure.
### Does this PR introduce _any_ user-facing change?
Yes. The queries above succeed. This is a long-standing issue since
[v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0) (2021-10-06). It
is easier to hit since
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19),
which enables ANSI mode by default.
A conjunct that `ExprUtils.canEvaluateUnconditionally` does not accept is
no longer moved into the inner filter when a non-deterministic or throwable
conjunct is written before it. For example, `rand() < 0.5 AND upper(p) = '1'`
over another filter does not prune partitions by `p` anymore. A query with a
single filter is not affected.
### How was this patch tested?
Pass the CIs with the newly added test cases. Both fail without this PR.
- `FilterPushdownSuite`: with `rand(0) > 0.1` or `raise_error('boom') IS
NULL` as the non-deterministic or throwable conjunct, a division written before
it is combined, a division written after it is kept above, and an equality
written after it is still combined.
- `DataFrameSuite`: Issue 1, written with the DataFrame API, returns no row
in ANSI mode instead of failing with `DIVIDE_BY_ZERO`.
I also ran the catalyst optimizer suites and the `PlanStabilitySuite`s
locally. No golden files change. With a temporary suite (not included), I ran
Issues 1 and 2 before and after this PR, along with similar queries using a
non-deterministic UDF and a throwable conjunct. I ran each with whole-stage
codegen on and off.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60157 from dongjoon-hyun/SPARK-60980.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 0042c2b6b096b0e58e9352623ef2a253994d2fb9)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/catalyst/optimizer/Optimizer.scala | 10 +++++++---
.../sql/catalyst/optimizer/FilterPushdownSuite.scala | 19 +++++++++++++++++++
.../scala/org/apache/spark/sql/DataFrameSuite.scala | 12 ++++++++++++
3 files changed, 38 insertions(+), 3 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index 6c751bc09af0..1ac14446c08c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -2406,10 +2406,14 @@ object CombineFilters extends Rule[LogicalPlan] with
PredicateHelper {
// The query execution/optimization does not guarantee the expressions are
evaluated in order.
// We only can combine them if and only if both are deterministic and the
outer condition is not
// throwable (inner can be throwable as it was going to be evaluated first
anyways).
+ // Combining a condition written after a non-deterministic or throwable
one evaluates it ahead
+ // of that one, so it must also be unable to fail. `throwable` does not
report e.g. a division
+ // by zero in ANSI mode, which the condition before it may have kept from
being evaluated.
case Filter(fc, nf @ Filter(nc, grandChild)) if nc.deterministic =>
- val (combineCandidates, rest) =
- splitConjunctivePredicates(fc).partition(p => p.deterministic &&
!p.throwable)
- val mergedFilter = (ExpressionSet(combineCandidates) --
+ val (leading, following) =
+ splitConjunctivePredicates(fc).span(p => p.deterministic &&
!p.throwable)
+ val (cannotFail, rest) =
following.partition(ExprUtils.canEvaluateUnconditionally)
+ val mergedFilter = (ExpressionSet(leading ++ cannotFail) --
ExpressionSet(splitConjunctivePredicates(nc))).reduceOption(And) match
{
case Some(ac) =>
Filter(And(nc, ac), grandChild)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
index bf9bb54eefa5..536f9fe79f44 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
@@ -135,6 +135,25 @@ class FilterPushdownSuite extends PlanTest {
comparePlans(optimized, originalQuery)
}
+ test("SPARK-60980: a condition that can fail is not combined ahead of a
non-deterministic " +
+ "or throwable one") {
+ Seq(Rand(0) > 0.1, IsNull(RaiseError(Literal("boom")))).foreach { blocker
=>
+ val originalQuery =
+ testRelation
+ .where($"c" === 1)
+ .where($"a" / $"b" > 1 && blocker && $"b" / $"a" > 1 && $"a" === 1)
+
+ val optimized = Optimize.execute(originalQuery.analyze)
+ val correctAnswer =
+ testRelation
+ .where($"c" === 1 && $"a" / $"b" > 1 && $"a" === 1)
+ .where(blocker && $"b" / $"a" > 1)
+ .analyze
+
+ comparePlans(optimized, correctAnswer)
+ }
+ }
+
test("SPARK-16164: Filter pushdown should keep the ordering in the logical
plan") {
val originalQuery =
testRelation
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index 0edb2861b2d3..db85393ef360 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -3010,6 +3010,18 @@ class DataFrameSuite extends SharedSparkSession
}
}
+ test("SPARK-60980: a filter condition that can fail is not evaluated ahead
of a " +
+ "non-deterministic one") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ // The first conjunct of the outer filter is false on every row, so the
`AND` short-circuit
+ // keeps the division by zero on id 3 from being evaluated.
+ val df = spark.range(0, 10, 1, 1).toDF()
+ .where($"id" >= 0)
+ .where(rand(10) * 10 > 100 && lit(10L) / ($"id" - 3) > 0)
+ checkAnswer(df, Nil)
+ }
+ }
+
test("SPARK-60528: collect a nested empty struct") {
val df = spark.range(1).selectExpr("id AS i", "named_struct('a', struct())
AS s")
checkAnswer(df, Row(0L, Row(Row())))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]