This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new dcad297056ab [SPARK-60980][SQL] Fix query failures by `CombineFilters` 
when a condition that can fail follows a non-deterministic or throwable one
dcad297056ab is described below

commit dcad297056abde10e29491be847941b8e32132ab
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Nov 5 23:47:47 2026 -0800

    [SPARK-60980][SQL] Fix query failures by `CombineFilters` when a condition 
that can fail follows a non-deterministic or throwable one
    
    ### What changes were proposed in this pull request?
    
    This PR fixes `CombineFilters` so that it does not move a condition that 
can fail, such as a division in ANSI mode, ahead of a non-deterministic or 
throwable condition written before it.
    
    `CombineFilters` combines `Filter(fc, Filter(nc, child))` by adding 
conjuncts of the outer condition `fc` to the inner filter, which evaluates them 
first, and keeping the others in a filter above it. Since #32405 (SPARK-35273), 
it keeps only the non-deterministic and throwable conjuncts above. It moves 
every other conjunct into the inner filter with `partition`, no matter where it 
is written. So a conjunct written after a non-deterministic or throwable one is 
evaluated ahead of it, on [...]
    
    This PR splits the outer conjuncts with `span` instead.
    
    | Outer conjunct | Before | After |
    |---|---|---|
    | Written before the first non-deterministic or throwable conjunct | 
Combined | Combined |
    | Non-deterministic or throwable | Kept above | Kept above |
    | Written after it and cannot fail, e.g. `p = 1` | Combined | Combined |
    | Written after it and can fail, e.g. `10L / (id - 3) > 0` | Combined | 
Kept above |
    
    A conjunct cannot fail when `ExprUtils.canEvaluateUnconditionally` accepts 
it: comparisons, `IN`, `IS NULL` and so on over columns and literals. Moving 
such a conjunct ahead only reduces the rows that the other conjuncts see. This 
keeps the partition pruning and filter pushdown that #32405 added for a 
condition like `rand() < 0.5 AND p = 1`. The conjuncts kept above stay in their 
written order.
    
    Spark does not guarantee the evaluation order of conjuncts in general. 
However, `CombineFilters` already keeps a throwable conjunct after the 
conjuncts written before it, and #59848 (SPARK-60602) and #59944 (SPARK-60633) 
do the same for filter pushdown. This PR covers the conjuncts that can fail 
without being throwable.
    
    ### Why are the changes needed?
    
    A query that should succeed fails. With ANSI mode, the default since Apache 
Spark 4.0.0, a division or a cast is enough.
    
    ```sql
    -- Issue 1: It should return no row, but fails with DIVIDE_BY_ZERO.
    SELECT id FROM (SELECT id FROM range(0, 10, 1, 1) WHERE id >= 0)
    WHERE rand(10) * 10 > 100 AND 10L / (id - 3) > 0;
    
    -- Issue 2: It should return 5, 6, 7, 8 and 9, but fails with 
DIVIDE_BY_ZERO.
    SELECT id FROM (SELECT id FROM range(0, 10, 1, 2) WHERE id >= 0)
    WHERE spark_partition_id() = 1 AND 10L / (id - 3) > 0;
    ```
    
    `rand(10) * 10 > 100` is false on every row, and `spark_partition_id() = 1` 
is false for `id = 3`, so the division should never run for `id = 3`. Both 
queries succeed without the inner `WHERE id >= 0`. With it, `CombineFilters` 
moves the division into the inner filter, ahead of the non-deterministic 
conjunct. A non-deterministic UDF or a throwable conjunct written first, such 
as a `CASE WHEN` with `raise_error`, causes the same failure.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The queries above succeed. This is a long-standing issue since 
[v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0) (2021-10-06). It 
is easier to hit since 
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19), 
which enables ANSI mode by default.
    
    A conjunct that `ExprUtils.canEvaluateUnconditionally` does not accept is 
no longer moved into the inner filter when a non-deterministic or throwable 
conjunct is written before it. For example, `rand() < 0.5 AND upper(p) = '1'` 
over another filter does not prune partitions by `p` anymore. A query with a 
single filter is not affected.
    
    ### How was this patch tested?
    
    Pass the CIs with the newly added test cases. Both fail without this PR.
    
    - `FilterPushdownSuite`: with `rand(0) > 0.1` or `raise_error('boom') IS 
NULL` as the non-deterministic or throwable conjunct, a division written before 
it is combined, a division written after it is kept above, and an equality 
written after it is still combined.
    - `DataFrameSuite`: Issue 1, written with the DataFrame API, returns no row 
in ANSI mode instead of failing with `DIVIDE_BY_ZERO`.
    
    I also ran the catalyst optimizer suites and the `PlanStabilitySuite`s 
locally. No golden files change. With a temporary suite (not included), I ran 
Issues 1 and 2 before and after this PR, along with similar queries using a 
non-deterministic UDF and a throwable conjunct. I ran each with whole-stage 
codegen on and off.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60157 from dongjoon-hyun/SPARK-60980.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 0042c2b6b096b0e58e9352623ef2a253994d2fb9)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/catalyst/optimizer/Optimizer.scala      | 10 +++++++---
 .../sql/catalyst/optimizer/FilterPushdownSuite.scala  | 19 +++++++++++++++++++
 .../scala/org/apache/spark/sql/DataFrameSuite.scala   | 12 ++++++++++++
 3 files changed, 38 insertions(+), 3 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index 6c751bc09af0..1ac14446c08c 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -2406,10 +2406,14 @@ object CombineFilters extends Rule[LogicalPlan] with 
PredicateHelper {
     // The query execution/optimization does not guarantee the expressions are 
evaluated in order.
     // We only can combine them if and only if both are deterministic and the 
outer condition is not
     // throwable (inner can be throwable as it was going to be evaluated first 
anyways).
+    // Combining a condition written after a non-deterministic or throwable 
one evaluates it ahead
+    // of that one, so it must also be unable to fail. `throwable` does not 
report e.g. a division
+    // by zero in ANSI mode, which the condition before it may have kept from 
being evaluated.
     case Filter(fc, nf @ Filter(nc, grandChild)) if nc.deterministic =>
-      val (combineCandidates, rest) =
-        splitConjunctivePredicates(fc).partition(p => p.deterministic && 
!p.throwable)
-      val mergedFilter = (ExpressionSet(combineCandidates) --
+      val (leading, following) =
+        splitConjunctivePredicates(fc).span(p => p.deterministic && 
!p.throwable)
+      val (cannotFail, rest) = 
following.partition(ExprUtils.canEvaluateUnconditionally)
+      val mergedFilter = (ExpressionSet(leading ++ cannotFail) --
         ExpressionSet(splitConjunctivePredicates(nc))).reduceOption(And) match 
{
         case Some(ac) =>
           Filter(And(nc, ac), grandChild)
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
index bf9bb54eefa5..536f9fe79f44 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
@@ -135,6 +135,25 @@ class FilterPushdownSuite extends PlanTest {
     comparePlans(optimized, originalQuery)
   }
 
+  test("SPARK-60980: a condition that can fail is not combined ahead of a 
non-deterministic " +
+    "or throwable one") {
+    Seq(Rand(0) > 0.1, IsNull(RaiseError(Literal("boom")))).foreach { blocker 
=>
+      val originalQuery =
+        testRelation
+          .where($"c" === 1)
+          .where($"a" / $"b" > 1 && blocker && $"b" / $"a" > 1 && $"a" === 1)
+
+      val optimized = Optimize.execute(originalQuery.analyze)
+      val correctAnswer =
+        testRelation
+          .where($"c" === 1 && $"a" / $"b" > 1 && $"a" === 1)
+          .where(blocker && $"b" / $"a" > 1)
+          .analyze
+
+      comparePlans(optimized, correctAnswer)
+    }
+  }
+
   test("SPARK-16164: Filter pushdown should keep the ordering in the logical 
plan") {
     val originalQuery =
       testRelation
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index 0edb2861b2d3..db85393ef360 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -3010,6 +3010,18 @@ class DataFrameSuite extends SharedSparkSession
     }
   }
 
+  test("SPARK-60980: a filter condition that can fail is not evaluated ahead 
of a " +
+    "non-deterministic one") {
+    withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+      // The first conjunct of the outer filter is false on every row, so the 
`AND` short-circuit
+      // keeps the division by zero on id 3 from being evaluated.
+      val df = spark.range(0, 10, 1, 1).toDF()
+        .where($"id" >= 0)
+        .where(rand(10) * 10 > 100 && lit(10L) / ($"id" - 3) > 0)
+      checkAnswer(df, Nil)
+    }
+  }
+
   test("SPARK-60528: collect a nested empty struct") {
     val df = spark.range(1).selectExpr("id AS i", "named_struct('a', struct()) 
AS s")
     checkAnswer(df, Row(0L, Row(Row())))


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to