This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new b98123545915 [SPARK-60633][SQL] Do not push a throwable filter 
condition through an operator ahead of the conditions before it
b98123545915 is described below

commit b98123545915a84403c7fa89e8d977ee0b2f5aed
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Sun Nov 1 14:22:02 2026 -0800

    [SPARK-60633][SQL] Do not push a throwable filter condition through an 
operator ahead of the conditions before it
    
    ### What changes were proposed in this pull request?
    
    `PushPredicateThroughNonJoin` splits a filter into the conjuncts it can 
push below an operator and the ones it keeps above. Apart from the `Filter` 
over `Project` case fixed by SPARK-60602, these cases ignore 
`Expression.throwable`:
    
    - the generic `UnaryNode` case (`pushDownPredicate`), for every operator in 
`canPushThrough`: `BatchEvalPython`, `ArrowEvalPython`, `Generate`, `Expand`, 
`Sort`, `Repartition`, ...;
    - `Aggregate`, `Window`, `Union` and `EventTimeWatermark`.
    
    So a throwable conjunct (`raise_error`, `assert_true`) can be pushed below 
the operator on its own, ahead of a conjunct written before it that has to stay 
above.
    
    This PR gives all of these cases, and the `Project` case, one shared split, 
`splitPushable`. Every conjunct written before the first one that cannot be 
pushed is pushed, as before. After that one, only conjuncts that are not 
throwable are pushed. The conjuncts left above keep the order they were written 
in. Before, the generic and `Window` cases moved the non-deterministic 
conjuncts to the end, which could also put a throwable conjunct ahead of a 
non-deterministic one written before it.
    
    Two more rules are added:
    - In the `Aggregate` case, a conjunct over a throwable alias of the 
aggregate counts as throwable, since the pushed conjunct has the alias 
substituted. The `Project` case already does this.
    - The generic case never pushes a throwable conjunct through a `Generate` 
without `outer` or a `ScriptTransformation`, even when it is written first. 
These operators can output no row for an input row, so below them the conjunct 
would see rows it never sees above them. The other operators of 
`canPushThrough` output at least one row for each input row.
    
    Moving a non-throwable conjunct below a throwable one is still allowed, 
since that can only reduce the rows the throwable conjunct sees. A throwable 
conjunct with only pushable conjuncts before it is still pushed, so the 
`Aggregate` pushdown of throwable predicates from SPARK-46707 still works. 
`CombineFilters`, `PushPredicateThroughJoin` and SPARK-60602 already follow 
these rules.
    
    ### Why are the changes needed?
    
    A throwable conjunct can run on rows that the `AND` short-circuit would 
have kept from it, so a query that should succeed fails. For example, with a 
Python UDF `f`:
    
    ```python
    spark.range(0, 10, 1, 1).select("id", f("id").alias("f")) \
        .where((col("f") > 5) & assert_true(col("id") > 5).isNull()).collect()
    ```
    
    After `ExtractPythonUDFs`, `SparkOptimizer` runs 
`PushPredicateThroughNonJoin` again. It pushes `assert_true(id > 5) IS NULL` 
alone below the `BatchEvalPython` / `ArrowEvalPython` node, and the query fails 
with `USER_RAISED_EXCEPTION`, whatever 
`spark.sql.optimizer.avoidDoubleFilterEval` is set to. The same happens without 
Python:
    
    ```scala
    spark.range(10).select($"id", explode(array($"id")) as "x")
      .where($"x" > 5 && assert_true($"id" > 5).isNull)          // Generate
    spark.range(10).select($"id", explode(filter(array($"id"), x => x > 5)) as 
"x")
      .where(assert_true($"id" > 5).isNull)                      // Generate 
that drops rows
    spark.range(10).groupBy($"id").agg(max($"id") as "m")
      .where($"m" > 5 && assert_true($"id" > 5).isNull)          // Aggregate
    spark.range(10).select($"id", max($"id").over(Window.partitionBy($"id")) as 
"m")
      .where($"m" > 5 && assert_true($"id" > 5).isNull)          // Window
    df.union(df).where(rand(0) < 0 && assert_true($"id" > 100).isNull)  // Union
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Queries like the ones above failed with `USER_RAISED_EXCEPTION` and 
now succeed. This affects every release that has `Expression.throwable` (4.0.0 
and later).
    
    This only covers expressions marked `throwable` (`raise_error`, which 
`assert_true` becomes, `sequence` with a step, and some JSON expressions). An 
expression that fails under ANSI mode, such as a division by zero, is not 
marked and is pushed as before. A conjunct over a throwable alias is covered in 
the `Project` and `Aggregate` cases, but not in the generic case, which does 
not see the aliases below the operator.
    
    Filters without throwable conjuncts are pushed as before. The one other 
change is that conjuncts left above the operator in the generic and `Window` 
cases now keep their written order, instead of the non-deterministic ones 
moving to the end.
    
    ### How was this patch tested?
    
    New tests in `FilterPushdownSuite` for `BatchEvalPython` / 
`ArrowEvalPython`, `Sort`, `Generate`, `Aggregate` (including a conjunct over a 
throwable alias), `Window`, `Union` and `EventTimeWatermark`. They check that a 
throwable conjunct stays above, that a later non-throwable conjunct is still 
pushed, that a throwable conjunct with nothing kept above before it is still 
pushed, and, using `assertTopFilterConjuncts`, the written order of the 
conjuncts left above.
    
    End-to-end tests:
    - `DataFrameSuite` covers `Generate` (including one that drops rows), 
`Aggregate` (including a throwable alias), `Window` and `Union`, with 
whole-stage codegen on and off. Each query is built inside the config loop, so 
it is planned under that config.
    - `PythonUDFSuite` covers a regular Python UDF and a scalar pandas UDF, 
with `avoidDoubleFilterEval` on and off.
    
    Each new test fails without the change. All tests in 
`org.apache.spark.sql.catalyst.optimizer` pass, and so do `DataFrameSuite`, 
`PythonUDFSuite`, `ExtractPythonUDFsSuite`, `BatchEvalPythonExecSuite`, 
`GeneratorFunctionSuite` and the `FilterPushdown*` suites in `sql/core`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    Closes #59944 from viirya/SPARK-60633.
    
    Authored-by: Liang-Chi Hsieh <[email protected]>
    Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
 .../spark/sql/catalyst/optimizer/Optimizer.scala   | 104 ++++++++-----
 .../catalyst/optimizer/FilterPushdownSuite.scala   | 163 +++++++++++++++++++++
 .../org/apache/spark/sql/DataFrameSuite.scala      |  42 ++++++
 .../sql/execution/python/PythonUDFSuite.scala      |  25 +++-
 4 files changed, 295 insertions(+), 39 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index 210ca2ce3d66..c5d54c753bca 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -2647,13 +2647,10 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
         // first expensive one are all pushed. After it, a throwable component 
(e.g.
         // `raise_error`, possibly inside a projected alias) stays above with 
the expensive ones,
         // so it is not evaluated on rows the components before it would have 
filtered out.
-        val (leading, rest) = originalAndRewrittenConditionsWithUsedAlias.span 
{
-          case (_, used, _) => isCheap(used)
-        }
-        val (cheapRest, expensiveWithUsed) = rest.partition {
-          case (_, used, replaced) => isCheap(used) && !replaced.throwable
-        }
-        val cheapWithUsed = leading ++ cheapRest
+        val (cheapWithUsed, expensiveWithUsed) = splitPushable(
+          originalAndRewrittenConditionsWithUsedAlias)(
+          canPush = { case (_, used, _) => isCheap(used) },
+          throwable = { case (_, _, replaced) => replaced.throwable })
         // Cheap filters (cases 1 & 2) pushed below the projection.
         val baseChild: LogicalPlan = if (cheapWithUsed.nonEmpty) {
           val combinedCheapFilter = cheapWithUsed.map(_._3).reduce(And)
@@ -2707,7 +2704,9 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
     // grouping keys or constants. The Aggregate operator can't reduce 
distinct values of grouping
     // keys so the filter won't see any new data after push down. Float/double 
grouping keys are
     // normalized by the Aggregate, so the pushed filter reads them normalized 
wherever that could
-    // change its result (see `pushBelowAggregate`).
+    // change its result (see `pushBelowAggregate`). A throwable predicate 
written after one that
+    // stays up is not pushed, though (see `splitPushable`). This includes a 
predicate over a
+    // throwable alias of the aggregate.
     case filter @ Filter(condition, aggregate: Aggregate)
       if aggregate.aggregateExpressions.forall(_.deterministic)
         && aggregate.groupingExpressions.nonEmpty =>
@@ -2726,20 +2725,15 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
     // pushed beneath must satisfy the following conditions:
     // 1. All the expressions are part of window partitioning key. The 
expressions can be compound.
     // 2. Deterministic.
-    // 3. Placed before any non-deterministic predicates.
+    // 3. Not throwable, if a predicate written before it stays up (see 
`splitPushable`).
     case filter @ Filter(condition, w: Window)
       if w.partitionSpec.forall(_.isInstanceOf[AttributeReference]) =>
       val partitionAttrs = AttributeSet(w.partitionSpec.flatMap(_.references))
 
-      val (candidates, nonDeterministic) =
-        splitConjunctivePredicates(condition).partition(_.deterministic)
-
-      val (pushDown, rest) = candidates.partition { cond =>
-        cond.references.subsetOf(partitionAttrs)
+      val (pushDown, stayUp) = splitPushableConjuncts(condition) { cond =>
+        cond.deterministic && cond.references.subsetOf(partitionAttrs)
       }
 
-      val stayUp = rest ++ nonDeterministic
-
       if (pushDown.nonEmpty) {
         val pushDownPredicate = pushDown.reduce(And)
         val newWindow = w.copy(child = Filter(pushDownPredicate, w.child))
@@ -2750,7 +2744,7 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
 
     case filter @ Filter(condition, union: Union) =>
       // Union could change the rows, so non-deterministic predicate can't be 
pushed down
-      val (pushDown, stayUp) = 
splitConjunctivePredicates(condition).partition(_.deterministic)
+      val (pushDown, stayUp) = 
splitPushableConjuncts(condition)(_.deterministic)
 
       if (pushDown.nonEmpty) {
         val pushDownCond = pushDown.reduceLeft(And)
@@ -2780,7 +2774,7 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
       }
 
     case filter @ Filter(condition, watermark: EventTimeWatermark) =>
-      val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition 
{ p =>
+      val (pushDown, stayUp) = splitPushableConjuncts(condition) { p =>
         p.deterministic && !p.references.contains(watermark.eventTime)
       }
 
@@ -2796,7 +2790,7 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
 
     case filter @ Filter(_, u: UnaryNode)
         if canPushThrough(u) && u.expressions.forall(_.deterministic) =>
-      pushDownPredicate(filter, u.child) { predicate =>
+      pushDownPredicate(filter, u.child, keepThrowableUp = canDropRows(u)) { 
predicate =>
         u.withNewChildren(Seq(Filter(predicate, u.child)))
       }
   }
@@ -2821,10 +2815,22 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
     case _ => false
   }
 
+  /**
+   * Whether an operator of [[canPushThrough]] can output no row for an input 
row. Below such an
+   * operator, a pushed condition sees the input rows it drops, which the 
condition above it
+   * would never see.
+   */
+  private def canDropRows(p: UnaryNode): Boolean = p match {
+    case g: Generate => !g.outer
+    case _: ScriptTransformation => true
+    case _ => false
+  }
+
   // Splits `conds`, the conjuncts of a filter on top of `aggregate`, into the 
ones to push below
   // it, rewritten in terms of the aggregate's child, and the ones that stay 
above it. A conjunct
   // can be pushed if it is deterministic and, after alias expansion, only 
reads the aggregate's
-  // child (that is, grouping keys and constants). Expressions above the 
aggregate see its
+  // child (that is, grouping keys and constants), unless it is throwable and 
a conjunct written
+  // before it stays above (see `splitPushable`). Expressions above the 
aggregate see its
   // float/double grouping keys normalized (see NormalizeFloatingNumbers: -0.0 
becomes 0.0 and every
   // NaN becomes one canonical NaN), so a pushed conjunct reads them 
normalized too, wherever the
   // raw value could change its result. If the form of the keys the aggregate 
outputs may still
@@ -2847,9 +2853,8 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
         .map(trimAliases)
     }
     val mayChangeKeyForm = NormalizeFloatingNumbers.mayChangeKeyForm(aggregate)
-    val pushDown = Seq.newBuilder[Expression]
-    val stayUp = Seq.newBuilder[Expression]
-    conds.foreach { cond =>
+    // (condition, condition with aliases expanded, pushed form or None if it 
cannot be pushed)
+    val candidates = conds.map { cond =>
       val replaced = replaceAlias(cond, aliasMap)
       val pushable = cond.deterministic && cond.references.nonEmpty &&
         replaced.references.subsetOf(aggregate.child.outputSet)
@@ -2858,13 +2863,15 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
       } else {
         replaced
       }
-      if (pushable && (!mayChangeKeyForm || normalized.fastEquals(replaced))) {
-        pushDown += normalized
-      } else {
-        stayUp += cond
-      }
+      val canPush = pushable && (!mayChangeKeyForm || 
normalized.fastEquals(replaced))
+      (cond, replaced, if (canPush) Some(normalized) else None)
     }
-    (pushDown.result(), stayUp.result())
+    // The aliases are expanded in the pushed conjunct, so a conjunct over a 
throwable alias is
+    // throwable too.
+    val (pushDown, stayUp) = splitPushable(candidates)(
+      canPush = _._3.isDefined,
+      throwable = { case (_, replaced, _) => replaced.throwable })
+    (pushDown.map(_._3.get), stayUp.map(_._1))
   }
 
   // Rewrites `e` so that it gives the same result on the raw values of 
`grouping` as `e` gives on
@@ -2903,20 +2910,18 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
 
   private def pushDownPredicate(
       filter: Filter,
-      grandchild: LogicalPlan)(insertFilter: Expression => LogicalPlan): 
LogicalPlan = {
+      grandchild: LogicalPlan,
+      keepThrowableUp: Boolean)(insertFilter: Expression => LogicalPlan): 
LogicalPlan = {
     // Only push down the predicates that is deterministic and all the 
referenced attributes
-    // come from grandchild.
+    // come from grandchild. With `keepThrowableUp`, a throwable predicate is 
not pushed even when
+    // it is written first, as the operator in between can drop rows it would 
then see.
     // TODO: non-deterministic predicates could be pushed through some 
operators that do not change
     // the rows.
-    val (candidates, nonDeterministic) =
-      splitConjunctivePredicates(filter.condition).partition(_.deterministic)
-
-    val (pushDown, rest) = candidates.partition { cond =>
-      cond.references.subsetOf(grandchild.outputSet)
+    val (pushDown, stayUp) = splitPushableConjuncts(filter.condition) { cond =>
+      cond.deterministic && !(keepThrowableUp && cond.throwable) &&
+        cond.references.subsetOf(grandchild.outputSet)
     }
 
-    val stayUp = rest ++ nonDeterministic
-
     if (pushDown.nonEmpty) {
       val newChild = insertFilter(pushDown.reduceLeft(And))
       if (stayUp.nonEmpty) {
@@ -2929,6 +2934,29 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
     }
   }
 
+  /**
+   * Splits `conds`, the conjuncts of a filter in their written order, into 
the ones to push below
+   * an operator and the ones to keep above it. `canPush` tells whether the 
operator lets a
+   * conjunct through, and `throwable` whether it may throw (e.g. 
`raise_error`). Every conjunct
+   * written before the first one that cannot be pushed is pushed. After it, 
only the conjuncts
+   * that are not throwable are pushed, so a throwable conjunct is not 
evaluated on rows the
+   * conjuncts before it would have filtered out. Moving a non-throwable 
conjunct below a
+   * throwable one is fine, as it can only reduce the rows the throwable one 
sees. The conjuncts
+   * to keep above are returned in the order they were written.
+   */
+  private def splitPushable[T](conds: Seq[T])(
+      canPush: T => Boolean, throwable: T => Boolean): (Seq[T], Seq[T]) = {
+    val (leading, rest) = conds.span(canPush)
+    // The head of `rest`, if any, is the first conjunct that cannot be pushed.
+    val (pushedRest, stayUp) = rest.drop(1).partition(c => canPush(c) && 
!throwable(c))
+    (leading ++ pushedRest, rest.take(1) ++ stayUp)
+  }
+
+  /** [[splitPushable]] over the conjuncts of `condition`, checking each for 
`throwable`. */
+  private def splitPushableConjuncts(condition: Expression)(
+      canPush: Expression => Boolean): (Seq[Expression], Seq[Expression]) =
+    splitPushable(splitConjunctivePredicates(condition))(canPush, _.throwable)
+
   /**
    * Splits `project` into a stack of [[Project]]s with the expensive 
conditions in
    * `expensiveConds` between them, so an expensive expression only runs on 
the rows the filters
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
index d10757b7219a..132db7b97887 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
@@ -821,6 +821,169 @@ class FilterPushdownSuite extends PlanTest {
     assertTopFilterConjuncts(Optimize.execute(originalQuery), 
Seq(nonDeterministic, unplaced))
   }
 
+  // Raises for the rows with a <= 5, so it must only see the rows the 
conditions before it kept.
+  private val throwableOnA = attrA > 5 || IsNull(RaiseError(Literal("boom")))
+
+  test("SPARK-60633: a throwable condition is not pushed through a Python UDF 
evaluation " +
+    "ahead of the conditions before it") {
+    Seq(PythonEvalType.SQL_BATCHED_UDF, 
PythonEvalType.SQL_SCALAR_PANDAS_UDF).foreach { evalType =>
+      withClue(s"eval type $evalType: ") {
+        val pythonUDF = PythonUDF(
+          "pythonUDF", null, IntegerType, Seq(attrA), evalType, 
udfDeterministic = true)
+        val f = AttributeReference("f", IntegerType)()
+        def evalPython(child: LogicalPlan): LogicalPlan =
+          if (evalType == PythonEvalType.SQL_BATCHED_UDF) {
+            BatchEvalPython(Seq(pythonUDF), Seq(f), child)
+          } else {
+            ArrowEvalPython(Seq(pythonUDF), Seq(f), child, evalType)
+          }
+
+        // f > 5 cannot be pushed, so the throwable condition after it stays 
above too. b > 1
+        // is still pushed, as that only reduces the rows the throwable 
condition sees.
+        val originalQuery = Filter(f > 5 && throwableOnA && attrB > 1, 
evalPython(testRelation))
+        val correctAnswer =
+          Filter(f > 5 && throwableOnA, evalPython(Filter(attrB > 1, 
testRelation)))
+        comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+        // With nothing left above before it, a throwable condition is still 
pushed.
+        val throwableFirst = Filter(throwableOnA && f > 5, 
evalPython(testRelation))
+        val throwableFirstAnswer = Filter(f > 5, 
evalPython(Filter(throwableOnA, testRelation)))
+        comparePlans(Optimize.execute(throwableFirst), throwableFirstAnswer)
+
+        // The conditions left above keep their written order.
+        val nonDeterministic = Rand(10) > 0.5
+        val mixed = Filter(nonDeterministic && f > 5 && throwableOnA, 
evalPython(testRelation))
+        assertTopFilterConjuncts(
+          Optimize.execute(mixed), Seq(nonDeterministic, f > 5, throwableOnA))
+      }
+    }
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a sort ahead 
of a " +
+    "non-deterministic condition before it") {
+    val originalQuery = testRelation.orderBy($"b".asc)
+      .where(Rand(10) > 0.5 && throwableOnA && $"b" > 1)
+      .analyze
+    val Seq(nonDeterministic, throwable, _) = splitConjunctivePredicates(
+      originalQuery.asInstanceOf[Filter].condition)
+    val correctAnswer = testRelation.where($"b" > 1).orderBy($"b".asc)
+      .where(Rand(10) > 0.5 && throwableOnA)
+      .analyze
+
+    val optimized = Optimize.execute(originalQuery)
+    comparePlans(optimized, correctAnswer)
+    assertTopFilterConjuncts(optimized, Seq(nonDeterministic, throwable))
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a generate 
that can drop " +
+    "rows") {
+    val throwable = $"a" > 5 || IsNull(RaiseError(Literal("boom")))
+    // The generate outputs no row for an empty array, so below it the 
throwable condition would
+    // see rows it never sees above it, even though nothing is written before 
it.
+    val originalQuery = testRelationWithArrayType
+      .generate(Explode($"c_arr"), alias = Some("arr"))
+      .where(throwable && $"b" > 1)
+      .analyze
+    val correctAnswer = testRelationWithArrayType.where($"b" > 1)
+      .generate(Explode($"c_arr"), alias = Some("arr"))
+      .where(throwable)
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+    // An outer generate outputs a row for every input row, so the condition 
is still pushed.
+    val outerQuery = testRelationWithArrayType
+      .generate(Explode($"c_arr"), outer = true, alias = Some("arr"))
+      .where(throwable && $"b" > 1)
+      .analyze
+    val outerAnswer = testRelationWithArrayType.where(throwable && $"b" > 1)
+      .generate(Explode($"c_arr"), outer = true, alias = Some("arr"))
+      .analyze
+    comparePlans(Optimize.execute(outerQuery), outerAnswer)
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through an aggregate 
ahead of the " +
+    "conditions before it") {
+    val originalQuery = testRelation.groupBy($"a")($"a", count($"b") as "c")
+      .where($"c" > 1 && throwableOnA && $"a" > 1)
+      .analyze
+    val correctAnswer = testRelation.where($"a" > 1).groupBy($"a")($"a", 
count($"b") as "c")
+      .where($"c" > 1 && throwableOnA)
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+    // With nothing left above before it, a throwable condition is still 
pushed.
+    val throwableFirst = testRelation.groupBy($"a")($"a", count($"b") as "c")
+      .where(throwableOnA && $"c" > 1)
+      .analyze
+    val throwableFirstAnswer = testRelation.where(throwableOnA)
+      .groupBy($"a")($"a", count($"b") as "c")
+      .where($"c" > 1)
+      .analyze
+    comparePlans(Optimize.execute(throwableFirst), throwableFirstAnswer)
+  }
+
+  test("SPARK-60633: a condition over a throwable alias is not pushed through 
an aggregate " +
+    "ahead of the conditions before it") {
+    val originalQuery = testRelation
+      .groupBy($"a")($"a", count($"b") as "c", throwableOnA as "x")
+      .where($"c" > 1 && $"x" && $"a" > 1)
+      .analyze
+    // `x` is not throwable itself, but pushing `x` evaluates the throwable 
expression behind it.
+    val correctAnswer = testRelation.where($"a" > 1)
+      .groupBy($"a")($"a", count($"b") as "c", throwableOnA as "x")
+      .where($"c" > 1 && $"x")
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a window 
ahead of the " +
+    "conditions before it") {
+    val winExpr = windowExpr(count($"b"),
+      windowSpec($"a" :: Nil, $"b".asc :: Nil, UnspecifiedFrame))
+    val originalQuery = testRelation
+      .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+      .where($"window" > 1 && throwableOnA && $"a" > 1)
+      .analyze
+    val correctAnswer = testRelation.where($"a" > 1)
+      .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+      .where($"window" > 1 && throwableOnA)
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+    // The conditions left above keep their written order.
+    val withRand = testRelation
+      .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+      .where(Rand(10) > 0.5 && $"window" > 1 && $"a" > 1)
+      .analyze
+    val Seq(nonDeterministic, stayUp, _) = splitConjunctivePredicates(
+      withRand.asInstanceOf[Filter].condition)
+    assertTopFilterConjuncts(Optimize.execute(withRand), Seq(nonDeterministic, 
stayUp))
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a union ahead 
of a " +
+    "non-deterministic condition before it") {
+    val testRelation2 = LocalRelation($"d".int, $"e".int, $"f".int)
+    val originalQuery = Union(Seq(testRelation, testRelation2))
+      .where(Rand(10) > 0.5 && throwableOnA && $"b" > 1)
+      .analyze
+    val correctAnswer = Union(Seq(testRelation.where($"b" > 1), 
testRelation2.where($"e" > 1)))
+      .where(Rand(10) > 0.5 && throwableOnA)
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a watermark 
ahead of the " +
+    "conditions before it") {
+    val interval = new CalendarInterval(2, 2, 2000L)
+    val relation = LocalRelation(Seq(attrA, $"b".timestamp, attrC), Nil, 
isStreaming = true)
+    val nodeId = UUID.randomUUID()
+    val originalQuery = EventTimeWatermark(nodeId, $"b", interval, relation)
+      .where($"b" === new java.sql.Timestamp(0) && throwableOnA && $"c" === 5)
+    val correctAnswer = EventTimeWatermark(nodeId, $"b", interval, 
relation.where($"c" === 5))
+      .where($"b" === new java.sql.Timestamp(0) && throwableOnA)
+    comparePlans(Optimize.execute(originalQuery.analyze), 
correctAnswer.analyze)
+  }
+
   test("SPARK-55014: do not split the projection when configured") {
     withSQLConf(SQLConf.SPLIT_PROJECTION_FOR_EXPENSIVE_FILTERS.key -> "false") 
{
       val originalQuery = testStringRelation
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index d7d38f637c74..d2ba7f17ee96 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -2949,6 +2949,48 @@ class DataFrameSuite extends SharedSparkSession
     }
   }
 
+  test("SPARK-60633: a throwable filter condition is not pushed through an 
operator ahead of " +
+    "its order") {
+    def df: DataFrame = spark.range(0, 10, 1, 1).toDF()
+    // `assert_true(id > 5)` only holds for the rows the conjunct before it 
keeps, and that
+    // conjunct cannot be pushed through the operator below the filter. Each 
query is built
+    // inside the config loop, so it is planned under that config.
+    def generateQuery(): DataFrame = df.select($"id", explode(array($"id")) as 
"x")
+      .where($"x" > 5 && assert_true($"id" > 5).isNull)
+    def aggregateQuery(): DataFrame = df.groupBy($"id").agg(max($"id") as "m")
+      .where($"m" > 5 && assert_true($"id" > 5).isNull)
+    def windowQuery(): DataFrame =
+      df.select($"id", max($"id").over(Window.partitionBy($"id")) as "m")
+        .where($"m" > 5 && assert_true($"id" > 5).isNull)
+    // `rand(0) < 0` keeps no row, so `assert_true` never runs.
+    def unionQuery(): DataFrame =
+      df.union(df).where(rand(0) < 0 && assert_true($"id" > 100).isNull)
+    // The generate only outputs the rows with `id > 5`, so the filter above 
it never raises,
+    // although its only condition is written first.
+    def droppingGenerateQuery(): DataFrame =
+      df.select($"id", explode(filter(array($"id"), x => x > 5)) as "x")
+        .where(assert_true($"id" > 5).isNull)
+    // The throwable part is an aggregate result column, which the filter only 
reads.
+    def aggregateAliasQuery(): DataFrame =
+      df.groupBy($"id").agg(max($"id") as "m", assert_true($"id" > 5) as "x")
+        .where($"m" > 5 && $"x".isNull)
+    val expected = (6L until 10L).map(i => Row(i, i))
+    Seq(true, false).foreach { codegen =>
+      withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegen.toString) {
+        checkAnswer(generateQuery(), expected)
+        checkAnswer(aggregateQuery(), expected)
+        checkAnswer(windowQuery(), expected)
+        checkAnswer(unionQuery(), Nil)
+        checkAnswer(droppingGenerateQuery(), expected)
+        // An interpreted aggregate computes every result column for every 
group, so the
+        // throwable alias only waits for the filter when codegen defers it 
past the filter.
+        if (codegen) {
+          checkAnswer(aggregateAliasQuery(), (6L until 10L).map(i => Row(i, i, 
null)))
+        }
+      }
+    }
+  }
+
   test("SPARK-60528: collect a nested empty struct") {
     val df = spark.range(1).selectExpr("id AS i", "named_struct('a', struct()) 
AS s")
     checkAnswer(df, Row(0L, Row(Row())))
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
index f600711f12a3..1421fce8be27 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
@@ -26,7 +26,7 @@ import org.apache.spark.{SparkEnv, SparkException}
 import org.apache.spark.internal.config.Python.PYTHON_UDF_PIPELINED_EXECUTION
 import org.apache.spark.sql.{AnalysisException, DataFrame, 
IntegratedUDFTestUtils, Row}
 import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.functions.{aggregate, array, avg, col, count, lit, 
transform}
+import org.apache.spark.sql.functions.{aggregate, array, assert_true, avg, 
col, count, lit, transform}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSparkSession
 import org.apache.spark.sql.types.{LongType, StringType, StructType}
@@ -408,4 +408,27 @@ class PythonUDFSuite extends SharedSparkSession {
       }
     }
   }
+
+  test("SPARK-60633: a throwable filter condition is not pushed below a Python 
UDF " +
+    "evaluation ahead of its order") {
+    assume(shouldTestPythonUDFs)
+    val udfs = Seq(pythonTestUDF) ++ (if (shouldTestPandasUDFs) {
+      Seq(TestScalarPandasUDF(name = "pandasUDF", Some(LongType)))
+    } else {
+      Nil
+    })
+    udfs.foreach { testUDF =>
+      // `assert_true(id > 5)` only holds for the rows `f > 5` keeps. Once the 
UDF is extracted
+      // into its own operator, `f > 5` cannot be pushed below it. The query 
is built inside the
+      // config loop, so it is planned under that config.
+      def df(): DataFrame = spark.range(0, 10, 1, 1)
+        .select($"id", testUDF($"id") as "f")
+        .where($"f" > 5 && assert_true($"id" > 5).isNull)
+      Seq(true, false).foreach { avoid =>
+        withSQLConf(SQLConf.AVOID_DOUBLE_FILTER_EVAL.key -> avoid.toString) {
+          checkAnswer(df(), (6L until 10L).map(i => Row(i, i)))
+        }
+      }
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to