This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 907abcbaf769 [SPARK-60633][4.2][SQL] Do not push a throwable filter
condition through an operator ahead of the conditions before it
907abcbaf769 is described below
commit 907abcbaf769efaf8895c221f6dbf83a341d5d0f
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Sun Nov 1 18:53:15 2026 -0800
[SPARK-60633][4.2][SQL] Do not push a throwable filter condition through an
operator ahead of the conditions before it
### What changes were proposed in this pull request?
This is a backport of #59944 to `branch-4.2`.
`PushPredicateThroughNonJoin` no longer pushes a throwable conjunct
(`raise_error`, `assert_true`) below an operator ahead of a conjunct written
before it that stays above, in the generic `UnaryNode` case and the
`Aggregate`, `Window`, `Union` and `EventTimeWatermark` cases. It also never
pushes a throwable conjunct through a `Generate` without `outer` or a
`ScriptTransformation`, and in the `Aggregate` case a conjunct over a throwable
alias counts as throwable. See #59944 for the ful [...]
Differences from the `master` commit:
- `branch-4.2` does not have SPARK-60150 (`pushBelowAggregate`), so the
`Aggregate` case applies `splitPushable` in place.
- `branch-4.2` does not have SPARK-55014, so the `Project` case keeps its
shape without the projection split, using `splitPushable` for the
cheap/expensive split from SPARK-60602.
- `FilterPushdownSuite` gets the `assertTopFilterConjuncts` helper, which
`master` has from SPARK-60602.
### Why are the changes needed?
A throwable conjunct can run on rows that the `AND` short-circuit would
have kept from it, so a query that should succeed fails with
`USER_RAISED_EXCEPTION`. For example:
```scala
spark.range(10).select($"id", explode(array($"id")) as "x")
.where($"x" > 5 && assert_true($"id" > 5).isNull) // Generate
spark.range(10).groupBy($"id").agg(max($"id") as "m")
.where($"m" > 5 && assert_true($"id" > 5).isNull) // Aggregate
```
The same happens below a Python UDF evaluation (`BatchEvalPython` /
`ArrowEvalPython`), a `Window` and a `Union`.
### Does this PR introduce _any_ user-facing change?
Yes. Queries like the ones above failed with `USER_RAISED_EXCEPTION` and
now succeed. Filters without throwable conjuncts are pushed as before, except
that conjuncts left above the operator in the generic and `Window` cases keep
their written order, instead of the non-deterministic ones moving to the end.
### How was this patch tested?
The tests from #59944. `catalyst/testOnly
org.apache.spark.sql.catalyst.optimizer.*` and `sql/testOnly` of
`DataFrameSuite`, `PythonUDFSuite`, `*FilterPushdown*`,
`GeneratorFunctionSuite` and `DataFrameAggregateSuite` on JDK 17, and
scalastyle.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes #60012 from viirya/SPARK-60633-4.2.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
.../spark/sql/catalyst/optimizer/Optimizer.scala | 99 +++++++-----
.../catalyst/optimizer/FilterPushdownSuite.scala | 171 +++++++++++++++++++++
.../org/apache/spark/sql/DataFrameSuite.scala | 42 +++++
.../sql/execution/python/PythonUDFSuite.scala | 27 +++-
4 files changed, 301 insertions(+), 38 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index b4bca2dd0d32..b62f606b9050 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -2151,13 +2151,10 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// first expensive one are all pushed. After it, a throwable component
(e.g.
// `raise_error`, possibly inside a projected alias) stays above with
the expensive ones,
// so it is not evaluated on rows the components before it would have
filtered out.
- val (leading, rest) = originalAndRewrittenConditionsWithUsedAlias.span
{
- case (_, used, _) => isCheap(used)
- }
- val (cheapRest, expensiveWithUsed) = rest.partition {
- case (_, used, replaced) => isCheap(used) && !replaced.throwable
- }
- val cheapWithUsed = leading ++ cheapRest
+ val (cheapWithUsed, expensiveWithUsed) = splitPushable(
+ originalAndRewrittenConditionsWithUsedAlias)(
+ canPush = { case (_, used, _) => isCheap(used) },
+ throwable = { case (_, _, replaced) => replaced.throwable })
// Short circuit if we do not have any cheap filters return the
original filter as is.
if (cheapWithUsed.isEmpty) {
f
@@ -2185,7 +2182,9 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// We can push down deterministic predicate through Aggregate, including
throwable predicate.
// If we can push down a filter through Aggregate, it means the filter
only references the
// grouping keys or constants. The Aggregate operator can't reduce
distinct values of grouping
- // keys so the filter won't see any new data after push down.
+ // keys so the filter won't see any new data after push down. A throwable
predicate written
+ // after one that stays up is not pushed, though (see `splitPushable`).
This includes a
+ // predicate over a throwable alias of the aggregate.
case filter @ Filter(condition, aggregate: Aggregate)
if aggregate.aggregateExpressions.forall(_.deterministic)
&& aggregate.groupingExpressions.nonEmpty =>
@@ -2193,19 +2192,20 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// For each filter, expand the alias and check if the filter can be
evaluated using
// attributes produced by the aggregate operator's child operator.
- val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition
{ cond =>
- val replaced = replaceAlias(cond, aliasMap)
- cond.deterministic && cond.references.nonEmpty &&
- replaced.references.subsetOf(aggregate.child.outputSet)
- }
+ val (pushDown, stayUp) = splitPushable(
+ splitConjunctivePredicates(condition).map(cond => (cond,
replaceAlias(cond, aliasMap))))(
+ canPush = { case (cond, replaced) =>
+ cond.deterministic && cond.references.nonEmpty &&
+ replaced.references.subsetOf(aggregate.child.outputSet)
+ },
+ throwable = { case (_, replaced) => replaced.throwable })
if (pushDown.nonEmpty) {
- val pushDownPredicate = pushDown.reduce(And)
- val replaced = replaceAlias(pushDownPredicate, aliasMap)
+ val replaced = pushDown.map(_._2).reduce(And)
val newAggregate = aggregate.copy(child = Filter(replaced,
aggregate.child))
// If there is no more filter to stay up, just eliminate the filter.
// Otherwise, create "Filter(stayUp) <- Aggregate <-
Filter(pushDownPredicate)".
- if (stayUp.isEmpty) newAggregate else Filter(stayUp.reduce(And),
newAggregate)
+ if (stayUp.isEmpty) newAggregate else
Filter(stayUp.map(_._1).reduce(And), newAggregate)
} else {
filter
}
@@ -2214,20 +2214,15 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// pushed beneath must satisfy the following conditions:
// 1. All the expressions are part of window partitioning key. The
expressions can be compound.
// 2. Deterministic.
- // 3. Placed before any non-deterministic predicates.
+ // 3. Not throwable, if a predicate written before it stays up (see
`splitPushable`).
case filter @ Filter(condition, w: Window)
if w.partitionSpec.forall(_.isInstanceOf[AttributeReference]) =>
val partitionAttrs = AttributeSet(w.partitionSpec.flatMap(_.references))
- val (candidates, nonDeterministic) =
- splitConjunctivePredicates(condition).partition(_.deterministic)
-
- val (pushDown, rest) = candidates.partition { cond =>
- cond.references.subsetOf(partitionAttrs)
+ val (pushDown, stayUp) = splitPushableConjuncts(condition) { cond =>
+ cond.deterministic && cond.references.subsetOf(partitionAttrs)
}
- val stayUp = rest ++ nonDeterministic
-
if (pushDown.nonEmpty) {
val pushDownPredicate = pushDown.reduce(And)
val newWindow = w.copy(child = Filter(pushDownPredicate, w.child))
@@ -2238,7 +2233,7 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
case filter @ Filter(condition, union: Union) =>
// Union could change the rows, so non-deterministic predicate can't be
pushed down
- val (pushDown, stayUp) =
splitConjunctivePredicates(condition).partition(_.deterministic)
+ val (pushDown, stayUp) =
splitPushableConjuncts(condition)(_.deterministic)
if (pushDown.nonEmpty) {
val pushDownCond = pushDown.reduceLeft(And)
@@ -2268,7 +2263,7 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
}
case filter @ Filter(condition, watermark: EventTimeWatermark) =>
- val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition
{ p =>
+ val (pushDown, stayUp) = splitPushableConjuncts(condition) { p =>
p.deterministic && !p.references.contains(watermark.eventTime)
}
@@ -2284,7 +2279,7 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
case filter @ Filter(_, u: UnaryNode)
if canPushThrough(u) && u.expressions.forall(_.deterministic) =>
- pushDownPredicate(filter, u.child) { predicate =>
+ pushDownPredicate(filter, u.child, keepThrowableUp = canDropRows(u)) {
predicate =>
u.withNewChildren(Seq(Filter(predicate, u.child)))
}
}
@@ -2307,22 +2302,31 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
case _ => false
}
+ /**
+ * Whether an operator of [[canPushThrough]] can output no row for an input
row. Below such an
+ * operator, a pushed condition sees the input rows it drops, which the
condition above it
+ * would never see.
+ */
+ private def canDropRows(p: UnaryNode): Boolean = p match {
+ case g: Generate => !g.outer
+ case _: ScriptTransformation => true
+ case _ => false
+ }
+
private def pushDownPredicate(
filter: Filter,
- grandchild: LogicalPlan)(insertFilter: Expression => LogicalPlan):
LogicalPlan = {
+ grandchild: LogicalPlan,
+ keepThrowableUp: Boolean)(insertFilter: Expression => LogicalPlan):
LogicalPlan = {
// Only push down the predicates that is deterministic and all the
referenced attributes
- // come from grandchild.
+ // come from grandchild. With `keepThrowableUp`, a throwable predicate is
not pushed even when
+ // it is written first, as the operator in between can drop rows it would
then see.
// TODO: non-deterministic predicates could be pushed through some
operators that do not change
// the rows.
- val (candidates, nonDeterministic) =
- splitConjunctivePredicates(filter.condition).partition(_.deterministic)
-
- val (pushDown, rest) = candidates.partition { cond =>
- cond.references.subsetOf(grandchild.outputSet)
+ val (pushDown, stayUp) = splitPushableConjuncts(filter.condition) { cond =>
+ cond.deterministic && !(keepThrowableUp && cond.throwable) &&
+ cond.references.subsetOf(grandchild.outputSet)
}
- val stayUp = rest ++ nonDeterministic
-
if (pushDown.nonEmpty) {
val newChild = insertFilter(pushDown.reduceLeft(And))
if (stayUp.nonEmpty) {
@@ -2335,6 +2339,29 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
}
}
+ /**
+ * Splits `conds`, the conjuncts of a filter in their written order, into
the ones to push below
+ * an operator and the ones to keep above it. `canPush` tells whether the
operator lets a
+ * conjunct through, and `throwable` whether it may throw (e.g.
`raise_error`). Every conjunct
+ * written before the first one that cannot be pushed is pushed. After it,
only the conjuncts
+ * that are not throwable are pushed, so a throwable conjunct is not
evaluated on rows the
+ * conjuncts before it would have filtered out. Moving a non-throwable
conjunct below a
+ * throwable one is fine, as it can only reduce the rows the throwable one
sees. The conjuncts
+ * to keep above are returned in the order they were written.
+ */
+ private def splitPushable[T](conds: Seq[T])(
+ canPush: T => Boolean, throwable: T => Boolean): (Seq[T], Seq[T]) = {
+ val (leading, rest) = conds.span(canPush)
+ // The head of `rest`, if any, is the first conjunct that cannot be pushed.
+ val (pushedRest, stayUp) = rest.drop(1).partition(c => canPush(c) &&
!throwable(c))
+ (leading ++ pushedRest, rest.take(1) ++ stayUp)
+ }
+
+ /** [[splitPushable]] over the conjuncts of `condition`, checking each for
`throwable`. */
+ private def splitPushableConjuncts(condition: Expression)(
+ canPush: Expression => Boolean): (Seq[Expression], Seq[Expression]) =
+ splitPushable(splitConjunctivePredicates(condition))(canPush, _.throwable)
+
/**
* Whether `e` is expensive. Version 0 of the regex pattern cost estimate
treats every LIKE and
* RLIKE as expensive. It is checked here rather than in
`StringRegexExpression.expensive`,
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
index 80197e30e77f..0d9af1ef835e 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
@@ -380,6 +380,177 @@ class FilterPushdownSuite extends PlanTest {
comparePlans(optimized, correctAnswer)
}
+ // `comparePlans` sorts the conjuncts of each filter, so check the top
filter's order directly.
+ private def assertTopFilterConjuncts(plan: LogicalPlan, expected:
Seq[Expression]): Unit = {
+ val actual = plan.collectFirst { case Filter(condition, _) => condition }
+ .map(splitConjunctivePredicates).getOrElse(Nil)
+ assert(actual.map(_.canonicalized) === expected.map(_.canonicalized),
+ s"Top filter conjuncts ${actual.mkString(", ")} are not
${expected.mkString(", ")}")
+ }
+
+ // Raises for the rows with a <= 5, so it must only see the rows the
conditions before it kept.
+ private val throwableOnA = attrA > 5 || IsNull(RaiseError(Literal("boom")))
+
+ test("SPARK-60633: a throwable condition is not pushed through a Python UDF
evaluation " +
+ "ahead of the conditions before it") {
+ Seq(PythonEvalType.SQL_BATCHED_UDF,
PythonEvalType.SQL_SCALAR_PANDAS_UDF).foreach { evalType =>
+ withClue(s"eval type $evalType: ") {
+ val pythonUDF = PythonUDF(
+ "pythonUDF", null, IntegerType, Seq(attrA), evalType,
udfDeterministic = true)
+ val f = AttributeReference("f", IntegerType)()
+ def evalPython(child: LogicalPlan): LogicalPlan =
+ if (evalType == PythonEvalType.SQL_BATCHED_UDF) {
+ BatchEvalPython(Seq(pythonUDF), Seq(f), child)
+ } else {
+ ArrowEvalPython(Seq(pythonUDF), Seq(f), child, evalType)
+ }
+
+ // f > 5 cannot be pushed, so the throwable condition after it stays
above too. b > 1
+ // is still pushed, as that only reduces the rows the throwable
condition sees.
+ val originalQuery = Filter(f > 5 && throwableOnA && attrB > 1,
evalPython(testRelation))
+ val correctAnswer =
+ Filter(f > 5 && throwableOnA, evalPython(Filter(attrB > 1,
testRelation)))
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+ // With nothing left above before it, a throwable condition is still
pushed.
+ val throwableFirst = Filter(throwableOnA && f > 5,
evalPython(testRelation))
+ val throwableFirstAnswer = Filter(f > 5,
evalPython(Filter(throwableOnA, testRelation)))
+ comparePlans(Optimize.execute(throwableFirst), throwableFirstAnswer)
+
+ // The conditions left above keep their written order.
+ val nonDeterministic = Rand(10) > 0.5
+ val mixed = Filter(nonDeterministic && f > 5 && throwableOnA,
evalPython(testRelation))
+ assertTopFilterConjuncts(
+ Optimize.execute(mixed), Seq(nonDeterministic, f > 5, throwableOnA))
+ }
+ }
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a sort ahead
of a " +
+ "non-deterministic condition before it") {
+ val originalQuery = testRelation.orderBy($"b".asc)
+ .where(Rand(10) > 0.5 && throwableOnA && $"b" > 1)
+ .analyze
+ val Seq(nonDeterministic, throwable, _) = splitConjunctivePredicates(
+ originalQuery.asInstanceOf[Filter].condition)
+ val correctAnswer = testRelation.where($"b" > 1).orderBy($"b".asc)
+ .where(Rand(10) > 0.5 && throwableOnA)
+ .analyze
+
+ val optimized = Optimize.execute(originalQuery)
+ comparePlans(optimized, correctAnswer)
+ assertTopFilterConjuncts(optimized, Seq(nonDeterministic, throwable))
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a generate
that can drop " +
+ "rows") {
+ val throwable = $"a" > 5 || IsNull(RaiseError(Literal("boom")))
+ // The generate outputs no row for an empty array, so below it the
throwable condition would
+ // see rows it never sees above it, even though nothing is written before
it.
+ val originalQuery = testRelationWithArrayType
+ .generate(Explode($"c_arr"), alias = Some("arr"))
+ .where(throwable && $"b" > 1)
+ .analyze
+ val correctAnswer = testRelationWithArrayType.where($"b" > 1)
+ .generate(Explode($"c_arr"), alias = Some("arr"))
+ .where(throwable)
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+ // An outer generate outputs a row for every input row, so the condition
is still pushed.
+ val outerQuery = testRelationWithArrayType
+ .generate(Explode($"c_arr"), outer = true, alias = Some("arr"))
+ .where(throwable && $"b" > 1)
+ .analyze
+ val outerAnswer = testRelationWithArrayType.where(throwable && $"b" > 1)
+ .generate(Explode($"c_arr"), outer = true, alias = Some("arr"))
+ .analyze
+ comparePlans(Optimize.execute(outerQuery), outerAnswer)
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through an aggregate
ahead of the " +
+ "conditions before it") {
+ val originalQuery = testRelation.groupBy($"a")($"a", count($"b") as "c")
+ .where($"c" > 1 && throwableOnA && $"a" > 1)
+ .analyze
+ val correctAnswer = testRelation.where($"a" > 1).groupBy($"a")($"a",
count($"b") as "c")
+ .where($"c" > 1 && throwableOnA)
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+ // With nothing left above before it, a throwable condition is still
pushed.
+ val throwableFirst = testRelation.groupBy($"a")($"a", count($"b") as "c")
+ .where(throwableOnA && $"c" > 1)
+ .analyze
+ val throwableFirstAnswer = testRelation.where(throwableOnA)
+ .groupBy($"a")($"a", count($"b") as "c")
+ .where($"c" > 1)
+ .analyze
+ comparePlans(Optimize.execute(throwableFirst), throwableFirstAnswer)
+ }
+
+ test("SPARK-60633: a condition over a throwable alias is not pushed through
an aggregate " +
+ "ahead of the conditions before it") {
+ val originalQuery = testRelation
+ .groupBy($"a")($"a", count($"b") as "c", throwableOnA as "x")
+ .where($"c" > 1 && $"x" && $"a" > 1)
+ .analyze
+ // `x` is not throwable itself, but pushing `x` evaluates the throwable
expression behind it.
+ val correctAnswer = testRelation.where($"a" > 1)
+ .groupBy($"a")($"a", count($"b") as "c", throwableOnA as "x")
+ .where($"c" > 1 && $"x")
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a window
ahead of the " +
+ "conditions before it") {
+ val winExpr = windowExpr(count($"b"),
+ windowSpec($"a" :: Nil, $"b".asc :: Nil, UnspecifiedFrame))
+ val originalQuery = testRelation
+ .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+ .where($"window" > 1 && throwableOnA && $"a" > 1)
+ .analyze
+ val correctAnswer = testRelation.where($"a" > 1)
+ .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+ .where($"window" > 1 && throwableOnA)
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+ // The conditions left above keep their written order.
+ val withRand = testRelation
+ .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+ .where(Rand(10) > 0.5 && $"window" > 1 && $"a" > 1)
+ .analyze
+ val Seq(nonDeterministic, stayUp, _) = splitConjunctivePredicates(
+ withRand.asInstanceOf[Filter].condition)
+ assertTopFilterConjuncts(Optimize.execute(withRand), Seq(nonDeterministic,
stayUp))
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a union ahead
of a " +
+ "non-deterministic condition before it") {
+ val testRelation2 = LocalRelation($"d".int, $"e".int, $"f".int)
+ val originalQuery = Union(Seq(testRelation, testRelation2))
+ .where(Rand(10) > 0.5 && throwableOnA && $"b" > 1)
+ .analyze
+ val correctAnswer = Union(Seq(testRelation.where($"b" > 1),
testRelation2.where($"e" > 1)))
+ .where(Rand(10) > 0.5 && throwableOnA)
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a watermark
ahead of the " +
+ "conditions before it") {
+ val interval = new CalendarInterval(2, 2, 2000L)
+ val relation = LocalRelation(Seq(attrA, $"b".timestamp, attrC), Nil,
isStreaming = true)
+ val nodeId = UUID.randomUUID()
+ val originalQuery = EventTimeWatermark(nodeId, $"b", interval, relation)
+ .where($"b" === new java.sql.Timestamp(0) && throwableOnA && $"c" === 5)
+ val correctAnswer = EventTimeWatermark(nodeId, $"b", interval,
relation.where($"c" === 5))
+ .where($"b" === new java.sql.Timestamp(0) && throwableOnA)
+ comparePlans(Optimize.execute(originalQuery.analyze),
correctAnswer.analyze)
+ }
+
test("nondeterministic: can always push down filter through project with
deterministic field") {
val originalQuery = testRelation
.select($"a")
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index 273c24d00c03..ec2231bacb1f 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -2832,6 +2832,48 @@ class DataFrameSuite extends SharedSparkSession
val df = classic.Dataset.ofRows(spark, relation)
checkAnswer(df.select("b"), Row(2))
}
+
+ test("SPARK-60633: a throwable filter condition is not pushed through an
operator ahead of " +
+ "its order") {
+ def df: DataFrame = spark.range(0, 10, 1, 1).toDF()
+ // `assert_true(id > 5)` only holds for the rows the conjunct before it
keeps, and that
+ // conjunct cannot be pushed through the operator below the filter. Each
query is built
+ // inside the config loop, so it is planned under that config.
+ def generateQuery(): DataFrame = df.select($"id", explode(array($"id")) as
"x")
+ .where($"x" > 5 && assert_true($"id" > 5).isNull)
+ def aggregateQuery(): DataFrame = df.groupBy($"id").agg(max($"id") as "m")
+ .where($"m" > 5 && assert_true($"id" > 5).isNull)
+ def windowQuery(): DataFrame =
+ df.select($"id", max($"id").over(Window.partitionBy($"id")) as "m")
+ .where($"m" > 5 && assert_true($"id" > 5).isNull)
+ // `rand(0) < 0` keeps no row, so `assert_true` never runs.
+ def unionQuery(): DataFrame =
+ df.union(df).where(rand(0) < 0 && assert_true($"id" > 100).isNull)
+ // The generate only outputs the rows with `id > 5`, so the filter above
it never raises,
+ // although its only condition is written first.
+ def droppingGenerateQuery(): DataFrame =
+ df.select($"id", explode(filter(array($"id"), x => x > 5)) as "x")
+ .where(assert_true($"id" > 5).isNull)
+ // The throwable part is an aggregate result column, which the filter only
reads.
+ def aggregateAliasQuery(): DataFrame =
+ df.groupBy($"id").agg(max($"id") as "m", assert_true($"id" > 5) as "x")
+ .where($"m" > 5 && $"x".isNull)
+ val expected = (6L until 10L).map(i => Row(i, i))
+ Seq(true, false).foreach { codegen =>
+ withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegen.toString) {
+ checkAnswer(generateQuery(), expected)
+ checkAnswer(aggregateQuery(), expected)
+ checkAnswer(windowQuery(), expected)
+ checkAnswer(unionQuery(), Nil)
+ checkAnswer(droppingGenerateQuery(), expected)
+ // An interpreted aggregate computes every result column for every
group, so the
+ // throwable alias only waits for the filter when codegen defers it
past the filter.
+ if (codegen) {
+ checkAnswer(aggregateAliasQuery(), (6L until 10L).map(i => Row(i, i,
null)))
+ }
+ }
+ }
+ }
}
case class GroupByKey(a: Int, b: Int)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
index 22584231b328..17f496baa24a 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
@@ -17,8 +17,8 @@
package org.apache.spark.sql.execution.python
-import org.apache.spark.sql.{AnalysisException, IntegratedUDFTestUtils, Row}
-import org.apache.spark.sql.functions.{array, avg, col, count, transform}
+import org.apache.spark.sql.{AnalysisException, DataFrame,
IntegratedUDFTestUtils, Row}
+import org.apache.spark.sql.functions.{array, assert_true, avg, col, count,
transform}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
import org.apache.spark.sql.types.LongType
@@ -265,4 +265,27 @@ class PythonUDFSuite extends SharedSparkSession {
}
}
}
+
+ test("SPARK-60633: a throwable filter condition is not pushed below a Python
UDF " +
+ "evaluation ahead of its order") {
+ assume(shouldTestPythonUDFs)
+ val udfs = Seq(pythonTestUDF) ++ (if (shouldTestPandasUDFs) {
+ Seq(TestScalarPandasUDF(name = "pandasUDF", Some(LongType)))
+ } else {
+ Nil
+ })
+ udfs.foreach { testUDF =>
+ // `assert_true(id > 5)` only holds for the rows `f > 5` keeps. Once the
UDF is extracted
+ // into its own operator, `f > 5` cannot be pushed below it. The query
is built inside the
+ // config loop, so it is planned under that config.
+ def df(): DataFrame = spark.range(0, 10, 1, 1)
+ .select($"id", testUDF($"id") as "f")
+ .where($"f" > 5 && assert_true($"id" > 5).isNull)
+ Seq(true, false).foreach { avoid =>
+ withSQLConf(SQLConf.AVOID_DOUBLE_FILTER_EVAL.key -> avoid.toString) {
+ checkAnswer(df(), (6L until 10L).map(i => Row(i, i)))
+ }
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]