This is an automated email from the ASF dual-hosted git repository.

viirya pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new 907abcbaf769 [SPARK-60633][4.2][SQL] Do not push a throwable filter 
condition through an operator ahead of the conditions before it
907abcbaf769 is described below

commit 907abcbaf769efaf8895c221f6dbf83a341d5d0f
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Sun Nov 1 18:53:15 2026 -0800

    [SPARK-60633][4.2][SQL] Do not push a throwable filter condition through an 
operator ahead of the conditions before it
    
    ### What changes were proposed in this pull request?
    
    This is a backport of #59944 to `branch-4.2`.
    
    `PushPredicateThroughNonJoin` no longer pushes a throwable conjunct 
(`raise_error`, `assert_true`) below an operator ahead of a conjunct written 
before it that stays above, in the generic `UnaryNode` case and the 
`Aggregate`, `Window`, `Union` and `EventTimeWatermark` cases. It also never 
pushes a throwable conjunct through a `Generate` without `outer` or a 
`ScriptTransformation`, and in the `Aggregate` case a conjunct over a throwable 
alias counts as throwable. See #59944 for the ful [...]
    
    Differences from the `master` commit:
    - `branch-4.2` does not have SPARK-60150 (`pushBelowAggregate`), so the 
`Aggregate` case applies `splitPushable` in place.
    - `branch-4.2` does not have SPARK-55014, so the `Project` case keeps its 
shape without the projection split, using `splitPushable` for the 
cheap/expensive split from SPARK-60602.
    - `FilterPushdownSuite` gets the `assertTopFilterConjuncts` helper, which 
`master` has from SPARK-60602.
    
    ### Why are the changes needed?
    
    A throwable conjunct can run on rows that the `AND` short-circuit would 
have kept from it, so a query that should succeed fails with 
`USER_RAISED_EXCEPTION`. For example:
    
    ```scala
    spark.range(10).select($"id", explode(array($"id")) as "x")
      .where($"x" > 5 && assert_true($"id" > 5).isNull)          // Generate
    spark.range(10).groupBy($"id").agg(max($"id") as "m")
      .where($"m" > 5 && assert_true($"id" > 5).isNull)          // Aggregate
    ```
    
    The same happens below a Python UDF evaluation (`BatchEvalPython` / 
`ArrowEvalPython`), a `Window` and a `Union`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Queries like the ones above failed with `USER_RAISED_EXCEPTION` and 
now succeed. Filters without throwable conjuncts are pushed as before, except 
that conjuncts left above the operator in the generic and `Window` cases keep 
their written order, instead of the non-deterministic ones moving to the end.
    
    ### How was this patch tested?
    
    The tests from #59944. `catalyst/testOnly 
org.apache.spark.sql.catalyst.optimizer.*` and `sql/testOnly` of 
`DataFrameSuite`, `PythonUDFSuite`, `*FilterPushdown*`, 
`GeneratorFunctionSuite` and `DataFrameAggregateSuite` on JDK 17, and 
scalastyle.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    This pull request and its description were written by Isaac.
    
    Closes #60012 from viirya/SPARK-60633-4.2.
    
    Authored-by: Liang-Chi Hsieh <[email protected]>
    Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
 .../spark/sql/catalyst/optimizer/Optimizer.scala   |  99 +++++++-----
 .../catalyst/optimizer/FilterPushdownSuite.scala   | 171 +++++++++++++++++++++
 .../org/apache/spark/sql/DataFrameSuite.scala      |  42 +++++
 .../sql/execution/python/PythonUDFSuite.scala      |  27 +++-
 4 files changed, 301 insertions(+), 38 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index b4bca2dd0d32..b62f606b9050 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -2151,13 +2151,10 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
         // first expensive one are all pushed. After it, a throwable component 
(e.g.
         // `raise_error`, possibly inside a projected alias) stays above with 
the expensive ones,
         // so it is not evaluated on rows the components before it would have 
filtered out.
-        val (leading, rest) = originalAndRewrittenConditionsWithUsedAlias.span 
{
-          case (_, used, _) => isCheap(used)
-        }
-        val (cheapRest, expensiveWithUsed) = rest.partition {
-          case (_, used, replaced) => isCheap(used) && !replaced.throwable
-        }
-        val cheapWithUsed = leading ++ cheapRest
+        val (cheapWithUsed, expensiveWithUsed) = splitPushable(
+          originalAndRewrittenConditionsWithUsedAlias)(
+          canPush = { case (_, used, _) => isCheap(used) },
+          throwable = { case (_, _, replaced) => replaced.throwable })
         // Short circuit if we do not have any cheap filters return the 
original filter as is.
         if (cheapWithUsed.isEmpty) {
           f
@@ -2185,7 +2182,9 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
     // We can push down deterministic predicate through Aggregate, including 
throwable predicate.
     // If we can push down a filter through Aggregate, it means the filter 
only references the
     // grouping keys or constants. The Aggregate operator can't reduce 
distinct values of grouping
-    // keys so the filter won't see any new data after push down.
+    // keys so the filter won't see any new data after push down. A throwable 
predicate written
+    // after one that stays up is not pushed, though (see `splitPushable`). 
This includes a
+    // predicate over a throwable alias of the aggregate.
     case filter @ Filter(condition, aggregate: Aggregate)
       if aggregate.aggregateExpressions.forall(_.deterministic)
         && aggregate.groupingExpressions.nonEmpty =>
@@ -2193,19 +2192,20 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
 
       // For each filter, expand the alias and check if the filter can be 
evaluated using
       // attributes produced by the aggregate operator's child operator.
-      val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition 
{ cond =>
-        val replaced = replaceAlias(cond, aliasMap)
-        cond.deterministic && cond.references.nonEmpty &&
-          replaced.references.subsetOf(aggregate.child.outputSet)
-      }
+      val (pushDown, stayUp) = splitPushable(
+        splitConjunctivePredicates(condition).map(cond => (cond, 
replaceAlias(cond, aliasMap))))(
+        canPush = { case (cond, replaced) =>
+          cond.deterministic && cond.references.nonEmpty &&
+            replaced.references.subsetOf(aggregate.child.outputSet)
+        },
+        throwable = { case (_, replaced) => replaced.throwable })
 
       if (pushDown.nonEmpty) {
-        val pushDownPredicate = pushDown.reduce(And)
-        val replaced = replaceAlias(pushDownPredicate, aliasMap)
+        val replaced = pushDown.map(_._2).reduce(And)
         val newAggregate = aggregate.copy(child = Filter(replaced, 
aggregate.child))
         // If there is no more filter to stay up, just eliminate the filter.
         // Otherwise, create "Filter(stayUp) <- Aggregate <- 
Filter(pushDownPredicate)".
-        if (stayUp.isEmpty) newAggregate else Filter(stayUp.reduce(And), 
newAggregate)
+        if (stayUp.isEmpty) newAggregate else 
Filter(stayUp.map(_._1).reduce(And), newAggregate)
       } else {
         filter
       }
@@ -2214,20 +2214,15 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
     // pushed beneath must satisfy the following conditions:
     // 1. All the expressions are part of window partitioning key. The 
expressions can be compound.
     // 2. Deterministic.
-    // 3. Placed before any non-deterministic predicates.
+    // 3. Not throwable, if a predicate written before it stays up (see 
`splitPushable`).
     case filter @ Filter(condition, w: Window)
       if w.partitionSpec.forall(_.isInstanceOf[AttributeReference]) =>
       val partitionAttrs = AttributeSet(w.partitionSpec.flatMap(_.references))
 
-      val (candidates, nonDeterministic) =
-        splitConjunctivePredicates(condition).partition(_.deterministic)
-
-      val (pushDown, rest) = candidates.partition { cond =>
-        cond.references.subsetOf(partitionAttrs)
+      val (pushDown, stayUp) = splitPushableConjuncts(condition) { cond =>
+        cond.deterministic && cond.references.subsetOf(partitionAttrs)
       }
 
-      val stayUp = rest ++ nonDeterministic
-
       if (pushDown.nonEmpty) {
         val pushDownPredicate = pushDown.reduce(And)
         val newWindow = w.copy(child = Filter(pushDownPredicate, w.child))
@@ -2238,7 +2233,7 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
 
     case filter @ Filter(condition, union: Union) =>
       // Union could change the rows, so non-deterministic predicate can't be 
pushed down
-      val (pushDown, stayUp) = 
splitConjunctivePredicates(condition).partition(_.deterministic)
+      val (pushDown, stayUp) = 
splitPushableConjuncts(condition)(_.deterministic)
 
       if (pushDown.nonEmpty) {
         val pushDownCond = pushDown.reduceLeft(And)
@@ -2268,7 +2263,7 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
       }
 
     case filter @ Filter(condition, watermark: EventTimeWatermark) =>
-      val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition 
{ p =>
+      val (pushDown, stayUp) = splitPushableConjuncts(condition) { p =>
         p.deterministic && !p.references.contains(watermark.eventTime)
       }
 
@@ -2284,7 +2279,7 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
 
     case filter @ Filter(_, u: UnaryNode)
         if canPushThrough(u) && u.expressions.forall(_.deterministic) =>
-      pushDownPredicate(filter, u.child) { predicate =>
+      pushDownPredicate(filter, u.child, keepThrowableUp = canDropRows(u)) { 
predicate =>
         u.withNewChildren(Seq(Filter(predicate, u.child)))
       }
   }
@@ -2307,22 +2302,31 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
     case _ => false
   }
 
+  /**
+   * Whether an operator of [[canPushThrough]] can output no row for an input 
row. Below such an
+   * operator, a pushed condition sees the input rows it drops, which the 
condition above it
+   * would never see.
+   */
+  private def canDropRows(p: UnaryNode): Boolean = p match {
+    case g: Generate => !g.outer
+    case _: ScriptTransformation => true
+    case _ => false
+  }
+
   private def pushDownPredicate(
       filter: Filter,
-      grandchild: LogicalPlan)(insertFilter: Expression => LogicalPlan): 
LogicalPlan = {
+      grandchild: LogicalPlan,
+      keepThrowableUp: Boolean)(insertFilter: Expression => LogicalPlan): 
LogicalPlan = {
     // Only push down the predicates that is deterministic and all the 
referenced attributes
-    // come from grandchild.
+    // come from grandchild. With `keepThrowableUp`, a throwable predicate is 
not pushed even when
+    // it is written first, as the operator in between can drop rows it would 
then see.
     // TODO: non-deterministic predicates could be pushed through some 
operators that do not change
     // the rows.
-    val (candidates, nonDeterministic) =
-      splitConjunctivePredicates(filter.condition).partition(_.deterministic)
-
-    val (pushDown, rest) = candidates.partition { cond =>
-      cond.references.subsetOf(grandchild.outputSet)
+    val (pushDown, stayUp) = splitPushableConjuncts(filter.condition) { cond =>
+      cond.deterministic && !(keepThrowableUp && cond.throwable) &&
+        cond.references.subsetOf(grandchild.outputSet)
     }
 
-    val stayUp = rest ++ nonDeterministic
-
     if (pushDown.nonEmpty) {
       val newChild = insertFilter(pushDown.reduceLeft(And))
       if (stayUp.nonEmpty) {
@@ -2335,6 +2339,29 @@ object PushPredicateThroughNonJoin extends 
Rule[LogicalPlan] with PredicateHelpe
     }
   }
 
+  /**
+   * Splits `conds`, the conjuncts of a filter in their written order, into 
the ones to push below
+   * an operator and the ones to keep above it. `canPush` tells whether the 
operator lets a
+   * conjunct through, and `throwable` whether it may throw (e.g. 
`raise_error`). Every conjunct
+   * written before the first one that cannot be pushed is pushed. After it, 
only the conjuncts
+   * that are not throwable are pushed, so a throwable conjunct is not 
evaluated on rows the
+   * conjuncts before it would have filtered out. Moving a non-throwable 
conjunct below a
+   * throwable one is fine, as it can only reduce the rows the throwable one 
sees. The conjuncts
+   * to keep above are returned in the order they were written.
+   */
+  private def splitPushable[T](conds: Seq[T])(
+      canPush: T => Boolean, throwable: T => Boolean): (Seq[T], Seq[T]) = {
+    val (leading, rest) = conds.span(canPush)
+    // The head of `rest`, if any, is the first conjunct that cannot be pushed.
+    val (pushedRest, stayUp) = rest.drop(1).partition(c => canPush(c) && 
!throwable(c))
+    (leading ++ pushedRest, rest.take(1) ++ stayUp)
+  }
+
+  /** [[splitPushable]] over the conjuncts of `condition`, checking each for 
`throwable`. */
+  private def splitPushableConjuncts(condition: Expression)(
+      canPush: Expression => Boolean): (Seq[Expression], Seq[Expression]) =
+    splitPushable(splitConjunctivePredicates(condition))(canPush, _.throwable)
+
   /**
    * Whether `e` is expensive. Version 0 of the regex pattern cost estimate 
treats every LIKE and
    * RLIKE as expensive. It is checked here rather than in 
`StringRegexExpression.expensive`,
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
index 80197e30e77f..0d9af1ef835e 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
@@ -380,6 +380,177 @@ class FilterPushdownSuite extends PlanTest {
     comparePlans(optimized, correctAnswer)
   }
 
+  // `comparePlans` sorts the conjuncts of each filter, so check the top 
filter's order directly.
+  private def assertTopFilterConjuncts(plan: LogicalPlan, expected: 
Seq[Expression]): Unit = {
+    val actual = plan.collectFirst { case Filter(condition, _) => condition }
+      .map(splitConjunctivePredicates).getOrElse(Nil)
+    assert(actual.map(_.canonicalized) === expected.map(_.canonicalized),
+      s"Top filter conjuncts ${actual.mkString(", ")} are not 
${expected.mkString(", ")}")
+  }
+
+  // Raises for the rows with a <= 5, so it must only see the rows the 
conditions before it kept.
+  private val throwableOnA = attrA > 5 || IsNull(RaiseError(Literal("boom")))
+
+  test("SPARK-60633: a throwable condition is not pushed through a Python UDF 
evaluation " +
+    "ahead of the conditions before it") {
+    Seq(PythonEvalType.SQL_BATCHED_UDF, 
PythonEvalType.SQL_SCALAR_PANDAS_UDF).foreach { evalType =>
+      withClue(s"eval type $evalType: ") {
+        val pythonUDF = PythonUDF(
+          "pythonUDF", null, IntegerType, Seq(attrA), evalType, 
udfDeterministic = true)
+        val f = AttributeReference("f", IntegerType)()
+        def evalPython(child: LogicalPlan): LogicalPlan =
+          if (evalType == PythonEvalType.SQL_BATCHED_UDF) {
+            BatchEvalPython(Seq(pythonUDF), Seq(f), child)
+          } else {
+            ArrowEvalPython(Seq(pythonUDF), Seq(f), child, evalType)
+          }
+
+        // f > 5 cannot be pushed, so the throwable condition after it stays 
above too. b > 1
+        // is still pushed, as that only reduces the rows the throwable 
condition sees.
+        val originalQuery = Filter(f > 5 && throwableOnA && attrB > 1, 
evalPython(testRelation))
+        val correctAnswer =
+          Filter(f > 5 && throwableOnA, evalPython(Filter(attrB > 1, 
testRelation)))
+        comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+        // With nothing left above before it, a throwable condition is still 
pushed.
+        val throwableFirst = Filter(throwableOnA && f > 5, 
evalPython(testRelation))
+        val throwableFirstAnswer = Filter(f > 5, 
evalPython(Filter(throwableOnA, testRelation)))
+        comparePlans(Optimize.execute(throwableFirst), throwableFirstAnswer)
+
+        // The conditions left above keep their written order.
+        val nonDeterministic = Rand(10) > 0.5
+        val mixed = Filter(nonDeterministic && f > 5 && throwableOnA, 
evalPython(testRelation))
+        assertTopFilterConjuncts(
+          Optimize.execute(mixed), Seq(nonDeterministic, f > 5, throwableOnA))
+      }
+    }
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a sort ahead 
of a " +
+    "non-deterministic condition before it") {
+    val originalQuery = testRelation.orderBy($"b".asc)
+      .where(Rand(10) > 0.5 && throwableOnA && $"b" > 1)
+      .analyze
+    val Seq(nonDeterministic, throwable, _) = splitConjunctivePredicates(
+      originalQuery.asInstanceOf[Filter].condition)
+    val correctAnswer = testRelation.where($"b" > 1).orderBy($"b".asc)
+      .where(Rand(10) > 0.5 && throwableOnA)
+      .analyze
+
+    val optimized = Optimize.execute(originalQuery)
+    comparePlans(optimized, correctAnswer)
+    assertTopFilterConjuncts(optimized, Seq(nonDeterministic, throwable))
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a generate 
that can drop " +
+    "rows") {
+    val throwable = $"a" > 5 || IsNull(RaiseError(Literal("boom")))
+    // The generate outputs no row for an empty array, so below it the 
throwable condition would
+    // see rows it never sees above it, even though nothing is written before 
it.
+    val originalQuery = testRelationWithArrayType
+      .generate(Explode($"c_arr"), alias = Some("arr"))
+      .where(throwable && $"b" > 1)
+      .analyze
+    val correctAnswer = testRelationWithArrayType.where($"b" > 1)
+      .generate(Explode($"c_arr"), alias = Some("arr"))
+      .where(throwable)
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+    // An outer generate outputs a row for every input row, so the condition 
is still pushed.
+    val outerQuery = testRelationWithArrayType
+      .generate(Explode($"c_arr"), outer = true, alias = Some("arr"))
+      .where(throwable && $"b" > 1)
+      .analyze
+    val outerAnswer = testRelationWithArrayType.where(throwable && $"b" > 1)
+      .generate(Explode($"c_arr"), outer = true, alias = Some("arr"))
+      .analyze
+    comparePlans(Optimize.execute(outerQuery), outerAnswer)
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through an aggregate 
ahead of the " +
+    "conditions before it") {
+    val originalQuery = testRelation.groupBy($"a")($"a", count($"b") as "c")
+      .where($"c" > 1 && throwableOnA && $"a" > 1)
+      .analyze
+    val correctAnswer = testRelation.where($"a" > 1).groupBy($"a")($"a", 
count($"b") as "c")
+      .where($"c" > 1 && throwableOnA)
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+    // With nothing left above before it, a throwable condition is still 
pushed.
+    val throwableFirst = testRelation.groupBy($"a")($"a", count($"b") as "c")
+      .where(throwableOnA && $"c" > 1)
+      .analyze
+    val throwableFirstAnswer = testRelation.where(throwableOnA)
+      .groupBy($"a")($"a", count($"b") as "c")
+      .where($"c" > 1)
+      .analyze
+    comparePlans(Optimize.execute(throwableFirst), throwableFirstAnswer)
+  }
+
+  test("SPARK-60633: a condition over a throwable alias is not pushed through 
an aggregate " +
+    "ahead of the conditions before it") {
+    val originalQuery = testRelation
+      .groupBy($"a")($"a", count($"b") as "c", throwableOnA as "x")
+      .where($"c" > 1 && $"x" && $"a" > 1)
+      .analyze
+    // `x` is not throwable itself, but pushing `x` evaluates the throwable 
expression behind it.
+    val correctAnswer = testRelation.where($"a" > 1)
+      .groupBy($"a")($"a", count($"b") as "c", throwableOnA as "x")
+      .where($"c" > 1 && $"x")
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a window 
ahead of the " +
+    "conditions before it") {
+    val winExpr = windowExpr(count($"b"),
+      windowSpec($"a" :: Nil, $"b".asc :: Nil, UnspecifiedFrame))
+    val originalQuery = testRelation
+      .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+      .where($"window" > 1 && throwableOnA && $"a" > 1)
+      .analyze
+    val correctAnswer = testRelation.where($"a" > 1)
+      .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+      .where($"window" > 1 && throwableOnA)
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+    // The conditions left above keep their written order.
+    val withRand = testRelation
+      .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+      .where(Rand(10) > 0.5 && $"window" > 1 && $"a" > 1)
+      .analyze
+    val Seq(nonDeterministic, stayUp, _) = splitConjunctivePredicates(
+      withRand.asInstanceOf[Filter].condition)
+    assertTopFilterConjuncts(Optimize.execute(withRand), Seq(nonDeterministic, 
stayUp))
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a union ahead 
of a " +
+    "non-deterministic condition before it") {
+    val testRelation2 = LocalRelation($"d".int, $"e".int, $"f".int)
+    val originalQuery = Union(Seq(testRelation, testRelation2))
+      .where(Rand(10) > 0.5 && throwableOnA && $"b" > 1)
+      .analyze
+    val correctAnswer = Union(Seq(testRelation.where($"b" > 1), 
testRelation2.where($"e" > 1)))
+      .where(Rand(10) > 0.5 && throwableOnA)
+      .analyze
+    comparePlans(Optimize.execute(originalQuery), correctAnswer)
+  }
+
+  test("SPARK-60633: a throwable condition is not pushed through a watermark 
ahead of the " +
+    "conditions before it") {
+    val interval = new CalendarInterval(2, 2, 2000L)
+    val relation = LocalRelation(Seq(attrA, $"b".timestamp, attrC), Nil, 
isStreaming = true)
+    val nodeId = UUID.randomUUID()
+    val originalQuery = EventTimeWatermark(nodeId, $"b", interval, relation)
+      .where($"b" === new java.sql.Timestamp(0) && throwableOnA && $"c" === 5)
+    val correctAnswer = EventTimeWatermark(nodeId, $"b", interval, 
relation.where($"c" === 5))
+      .where($"b" === new java.sql.Timestamp(0) && throwableOnA)
+    comparePlans(Optimize.execute(originalQuery.analyze), 
correctAnswer.analyze)
+  }
+
   test("nondeterministic: can always push down filter through project with 
deterministic field") {
     val originalQuery = testRelation
       .select($"a")
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index 273c24d00c03..ec2231bacb1f 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -2832,6 +2832,48 @@ class DataFrameSuite extends SharedSparkSession
     val df = classic.Dataset.ofRows(spark, relation)
     checkAnswer(df.select("b"), Row(2))
   }
+
+  test("SPARK-60633: a throwable filter condition is not pushed through an 
operator ahead of " +
+    "its order") {
+    def df: DataFrame = spark.range(0, 10, 1, 1).toDF()
+    // `assert_true(id > 5)` only holds for the rows the conjunct before it 
keeps, and that
+    // conjunct cannot be pushed through the operator below the filter. Each 
query is built
+    // inside the config loop, so it is planned under that config.
+    def generateQuery(): DataFrame = df.select($"id", explode(array($"id")) as 
"x")
+      .where($"x" > 5 && assert_true($"id" > 5).isNull)
+    def aggregateQuery(): DataFrame = df.groupBy($"id").agg(max($"id") as "m")
+      .where($"m" > 5 && assert_true($"id" > 5).isNull)
+    def windowQuery(): DataFrame =
+      df.select($"id", max($"id").over(Window.partitionBy($"id")) as "m")
+        .where($"m" > 5 && assert_true($"id" > 5).isNull)
+    // `rand(0) < 0` keeps no row, so `assert_true` never runs.
+    def unionQuery(): DataFrame =
+      df.union(df).where(rand(0) < 0 && assert_true($"id" > 100).isNull)
+    // The generate only outputs the rows with `id > 5`, so the filter above 
it never raises,
+    // although its only condition is written first.
+    def droppingGenerateQuery(): DataFrame =
+      df.select($"id", explode(filter(array($"id"), x => x > 5)) as "x")
+        .where(assert_true($"id" > 5).isNull)
+    // The throwable part is an aggregate result column, which the filter only 
reads.
+    def aggregateAliasQuery(): DataFrame =
+      df.groupBy($"id").agg(max($"id") as "m", assert_true($"id" > 5) as "x")
+        .where($"m" > 5 && $"x".isNull)
+    val expected = (6L until 10L).map(i => Row(i, i))
+    Seq(true, false).foreach { codegen =>
+      withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegen.toString) {
+        checkAnswer(generateQuery(), expected)
+        checkAnswer(aggregateQuery(), expected)
+        checkAnswer(windowQuery(), expected)
+        checkAnswer(unionQuery(), Nil)
+        checkAnswer(droppingGenerateQuery(), expected)
+        // An interpreted aggregate computes every result column for every 
group, so the
+        // throwable alias only waits for the filter when codegen defers it 
past the filter.
+        if (codegen) {
+          checkAnswer(aggregateAliasQuery(), (6L until 10L).map(i => Row(i, i, 
null)))
+        }
+      }
+    }
+  }
 }
 
 case class GroupByKey(a: Int, b: Int)
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
index 22584231b328..17f496baa24a 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
@@ -17,8 +17,8 @@
 
 package org.apache.spark.sql.execution.python
 
-import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils, Row}
-import org.apache.spark.sql.functions.{array, avg, col, count, transform}
+import org.apache.spark.sql.{AnalysisException, DataFrame, 
IntegratedUDFTestUtils, Row}
+import org.apache.spark.sql.functions.{array, assert_true, avg, col, count, 
transform}
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.test.SharedSparkSession
 import org.apache.spark.sql.types.LongType
@@ -265,4 +265,27 @@ class PythonUDFSuite extends SharedSparkSession {
       }
     }
   }
+
+  test("SPARK-60633: a throwable filter condition is not pushed below a Python 
UDF " +
+    "evaluation ahead of its order") {
+    assume(shouldTestPythonUDFs)
+    val udfs = Seq(pythonTestUDF) ++ (if (shouldTestPandasUDFs) {
+      Seq(TestScalarPandasUDF(name = "pandasUDF", Some(LongType)))
+    } else {
+      Nil
+    })
+    udfs.foreach { testUDF =>
+      // `assert_true(id > 5)` only holds for the rows `f > 5` keeps. Once the 
UDF is extracted
+      // into its own operator, `f > 5` cannot be pushed below it. The query 
is built inside the
+      // config loop, so it is planned under that config.
+      def df(): DataFrame = spark.range(0, 10, 1, 1)
+        .select($"id", testUDF($"id") as "f")
+        .where($"f" > 5 && assert_true($"id" > 5).isNull)
+      Seq(true, false).foreach { avoid =>
+        withSQLConf(SQLConf.AVOID_DOUBLE_FILTER_EVAL.key -> avoid.toString) {
+          checkAnswer(df(), (6L until 10L).map(i => Row(i, i)))
+        }
+      }
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to