This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b98123545915 [SPARK-60633][SQL] Do not push a throwable filter
condition through an operator ahead of the conditions before it
b98123545915 is described below
commit b98123545915a84403c7fa89e8d977ee0b2f5aed
Author: Liang-Chi Hsieh <[email protected]>
AuthorDate: Sun Nov 1 14:22:02 2026 -0800
[SPARK-60633][SQL] Do not push a throwable filter condition through an
operator ahead of the conditions before it
### What changes were proposed in this pull request?
`PushPredicateThroughNonJoin` splits a filter into the conjuncts it can
push below an operator and the ones it keeps above. Apart from the `Filter`
over `Project` case fixed by SPARK-60602, these cases ignore
`Expression.throwable`:
- the generic `UnaryNode` case (`pushDownPredicate`), for every operator in
`canPushThrough`: `BatchEvalPython`, `ArrowEvalPython`, `Generate`, `Expand`,
`Sort`, `Repartition`, ...;
- `Aggregate`, `Window`, `Union` and `EventTimeWatermark`.
So a throwable conjunct (`raise_error`, `assert_true`) can be pushed below
the operator on its own, ahead of a conjunct written before it that has to stay
above.
This PR gives all of these cases, and the `Project` case, one shared split,
`splitPushable`. Every conjunct written before the first one that cannot be
pushed is pushed, as before. After that one, only conjuncts that are not
throwable are pushed. The conjuncts left above keep the order they were written
in. Before, the generic and `Window` cases moved the non-deterministic
conjuncts to the end, which could also put a throwable conjunct ahead of a
non-deterministic one written before it.
Two more rules are added:
- In the `Aggregate` case, a conjunct over a throwable alias of the
aggregate counts as throwable, since the pushed conjunct has the alias
substituted. The `Project` case already does this.
- The generic case never pushes a throwable conjunct through a `Generate`
without `outer` or a `ScriptTransformation`, even when it is written first.
These operators can output no row for an input row, so below them the conjunct
would see rows it never sees above them. The other operators of
`canPushThrough` output at least one row for each input row.
Moving a non-throwable conjunct below a throwable one is still allowed,
since that can only reduce the rows the throwable conjunct sees. A throwable
conjunct with only pushable conjuncts before it is still pushed, so the
`Aggregate` pushdown of throwable predicates from SPARK-46707 still works.
`CombineFilters`, `PushPredicateThroughJoin` and SPARK-60602 already follow
these rules.
### Why are the changes needed?
A throwable conjunct can run on rows that the `AND` short-circuit would
have kept from it, so a query that should succeed fails. For example, with a
Python UDF `f`:
```python
spark.range(0, 10, 1, 1).select("id", f("id").alias("f")) \
.where((col("f") > 5) & assert_true(col("id") > 5).isNull()).collect()
```
After `ExtractPythonUDFs`, `SparkOptimizer` runs
`PushPredicateThroughNonJoin` again. It pushes `assert_true(id > 5) IS NULL`
alone below the `BatchEvalPython` / `ArrowEvalPython` node, and the query fails
with `USER_RAISED_EXCEPTION`, whatever
`spark.sql.optimizer.avoidDoubleFilterEval` is set to. The same happens without
Python:
```scala
spark.range(10).select($"id", explode(array($"id")) as "x")
.where($"x" > 5 && assert_true($"id" > 5).isNull) // Generate
spark.range(10).select($"id", explode(filter(array($"id"), x => x > 5)) as
"x")
.where(assert_true($"id" > 5).isNull) // Generate
that drops rows
spark.range(10).groupBy($"id").agg(max($"id") as "m")
.where($"m" > 5 && assert_true($"id" > 5).isNull) // Aggregate
spark.range(10).select($"id", max($"id").over(Window.partitionBy($"id")) as
"m")
.where($"m" > 5 && assert_true($"id" > 5).isNull) // Window
df.union(df).where(rand(0) < 0 && assert_true($"id" > 100).isNull) // Union
```
### Does this PR introduce _any_ user-facing change?
Yes. Queries like the ones above failed with `USER_RAISED_EXCEPTION` and
now succeed. This affects every release that has `Expression.throwable` (4.0.0
and later).
This only covers expressions marked `throwable` (`raise_error`, which
`assert_true` becomes, `sequence` with a step, and some JSON expressions). An
expression that fails under ANSI mode, such as a division by zero, is not
marked and is pushed as before. A conjunct over a throwable alias is covered in
the `Project` and `Aggregate` cases, but not in the generic case, which does
not see the aliases below the operator.
Filters without throwable conjuncts are pushed as before. The one other
change is that conjuncts left above the operator in the generic and `Window`
cases now keep their written order, instead of the non-deterministic ones
moving to the end.
### How was this patch tested?
New tests in `FilterPushdownSuite` for `BatchEvalPython` /
`ArrowEvalPython`, `Sort`, `Generate`, `Aggregate` (including a conjunct over a
throwable alias), `Window`, `Union` and `EventTimeWatermark`. They check that a
throwable conjunct stays above, that a later non-throwable conjunct is still
pushed, that a throwable conjunct with nothing kept above before it is still
pushed, and, using `assertTopFilterConjuncts`, the written order of the
conjuncts left above.
End-to-end tests:
- `DataFrameSuite` covers `Generate` (including one that drops rows),
`Aggregate` (including a throwable alias), `Window` and `Union`, with
whole-stage codegen on and off. Each query is built inside the config loop, so
it is planned under that config.
- `PythonUDFSuite` covers a regular Python UDF and a scalar pandas UDF,
with `avoidDoubleFilterEval` on and off.
Each new test fails without the change. All tests in
`org.apache.spark.sql.catalyst.optimizer` pass, and so do `DataFrameSuite`,
`PythonUDFSuite`, `ExtractPythonUDFsSuite`, `BatchEvalPythonExecSuite`,
`GeneratorFunctionSuite` and the `FilterPushdown*` suites in `sql/core`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59944 from viirya/SPARK-60633.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>
---
.../spark/sql/catalyst/optimizer/Optimizer.scala | 104 ++++++++-----
.../catalyst/optimizer/FilterPushdownSuite.scala | 163 +++++++++++++++++++++
.../org/apache/spark/sql/DataFrameSuite.scala | 42 ++++++
.../sql/execution/python/PythonUDFSuite.scala | 25 +++-
4 files changed, 295 insertions(+), 39 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index 210ca2ce3d66..c5d54c753bca 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -2647,13 +2647,10 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// first expensive one are all pushed. After it, a throwable component
(e.g.
// `raise_error`, possibly inside a projected alias) stays above with
the expensive ones,
// so it is not evaluated on rows the components before it would have
filtered out.
- val (leading, rest) = originalAndRewrittenConditionsWithUsedAlias.span
{
- case (_, used, _) => isCheap(used)
- }
- val (cheapRest, expensiveWithUsed) = rest.partition {
- case (_, used, replaced) => isCheap(used) && !replaced.throwable
- }
- val cheapWithUsed = leading ++ cheapRest
+ val (cheapWithUsed, expensiveWithUsed) = splitPushable(
+ originalAndRewrittenConditionsWithUsedAlias)(
+ canPush = { case (_, used, _) => isCheap(used) },
+ throwable = { case (_, _, replaced) => replaced.throwable })
// Cheap filters (cases 1 & 2) pushed below the projection.
val baseChild: LogicalPlan = if (cheapWithUsed.nonEmpty) {
val combinedCheapFilter = cheapWithUsed.map(_._3).reduce(And)
@@ -2707,7 +2704,9 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// grouping keys or constants. The Aggregate operator can't reduce
distinct values of grouping
// keys so the filter won't see any new data after push down. Float/double
grouping keys are
// normalized by the Aggregate, so the pushed filter reads them normalized
wherever that could
- // change its result (see `pushBelowAggregate`).
+ // change its result (see `pushBelowAggregate`). A throwable predicate
written after one that
+ // stays up is not pushed, though (see `splitPushable`). This includes a
predicate over a
+ // throwable alias of the aggregate.
case filter @ Filter(condition, aggregate: Aggregate)
if aggregate.aggregateExpressions.forall(_.deterministic)
&& aggregate.groupingExpressions.nonEmpty =>
@@ -2726,20 +2725,15 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// pushed beneath must satisfy the following conditions:
// 1. All the expressions are part of window partitioning key. The
expressions can be compound.
// 2. Deterministic.
- // 3. Placed before any non-deterministic predicates.
+ // 3. Not throwable, if a predicate written before it stays up (see
`splitPushable`).
case filter @ Filter(condition, w: Window)
if w.partitionSpec.forall(_.isInstanceOf[AttributeReference]) =>
val partitionAttrs = AttributeSet(w.partitionSpec.flatMap(_.references))
- val (candidates, nonDeterministic) =
- splitConjunctivePredicates(condition).partition(_.deterministic)
-
- val (pushDown, rest) = candidates.partition { cond =>
- cond.references.subsetOf(partitionAttrs)
+ val (pushDown, stayUp) = splitPushableConjuncts(condition) { cond =>
+ cond.deterministic && cond.references.subsetOf(partitionAttrs)
}
- val stayUp = rest ++ nonDeterministic
-
if (pushDown.nonEmpty) {
val pushDownPredicate = pushDown.reduce(And)
val newWindow = w.copy(child = Filter(pushDownPredicate, w.child))
@@ -2750,7 +2744,7 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
case filter @ Filter(condition, union: Union) =>
// Union could change the rows, so non-deterministic predicate can't be
pushed down
- val (pushDown, stayUp) =
splitConjunctivePredicates(condition).partition(_.deterministic)
+ val (pushDown, stayUp) =
splitPushableConjuncts(condition)(_.deterministic)
if (pushDown.nonEmpty) {
val pushDownCond = pushDown.reduceLeft(And)
@@ -2780,7 +2774,7 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
}
case filter @ Filter(condition, watermark: EventTimeWatermark) =>
- val (pushDown, stayUp) = splitConjunctivePredicates(condition).partition
{ p =>
+ val (pushDown, stayUp) = splitPushableConjuncts(condition) { p =>
p.deterministic && !p.references.contains(watermark.eventTime)
}
@@ -2796,7 +2790,7 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
case filter @ Filter(_, u: UnaryNode)
if canPushThrough(u) && u.expressions.forall(_.deterministic) =>
- pushDownPredicate(filter, u.child) { predicate =>
+ pushDownPredicate(filter, u.child, keepThrowableUp = canDropRows(u)) {
predicate =>
u.withNewChildren(Seq(Filter(predicate, u.child)))
}
}
@@ -2821,10 +2815,22 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
case _ => false
}
+ /**
+ * Whether an operator of [[canPushThrough]] can output no row for an input
row. Below such an
+ * operator, a pushed condition sees the input rows it drops, which the
condition above it
+ * would never see.
+ */
+ private def canDropRows(p: UnaryNode): Boolean = p match {
+ case g: Generate => !g.outer
+ case _: ScriptTransformation => true
+ case _ => false
+ }
+
// Splits `conds`, the conjuncts of a filter on top of `aggregate`, into the
ones to push below
// it, rewritten in terms of the aggregate's child, and the ones that stay
above it. A conjunct
// can be pushed if it is deterministic and, after alias expansion, only
reads the aggregate's
- // child (that is, grouping keys and constants). Expressions above the
aggregate see its
+ // child (that is, grouping keys and constants), unless it is throwable and
a conjunct written
+ // before it stays above (see `splitPushable`). Expressions above the
aggregate see its
// float/double grouping keys normalized (see NormalizeFloatingNumbers: -0.0
becomes 0.0 and every
// NaN becomes one canonical NaN), so a pushed conjunct reads them
normalized too, wherever the
// raw value could change its result. If the form of the keys the aggregate
outputs may still
@@ -2847,9 +2853,8 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
.map(trimAliases)
}
val mayChangeKeyForm = NormalizeFloatingNumbers.mayChangeKeyForm(aggregate)
- val pushDown = Seq.newBuilder[Expression]
- val stayUp = Seq.newBuilder[Expression]
- conds.foreach { cond =>
+ // (condition, condition with aliases expanded, pushed form or None if it
cannot be pushed)
+ val candidates = conds.map { cond =>
val replaced = replaceAlias(cond, aliasMap)
val pushable = cond.deterministic && cond.references.nonEmpty &&
replaced.references.subsetOf(aggregate.child.outputSet)
@@ -2858,13 +2863,15 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
} else {
replaced
}
- if (pushable && (!mayChangeKeyForm || normalized.fastEquals(replaced))) {
- pushDown += normalized
- } else {
- stayUp += cond
- }
+ val canPush = pushable && (!mayChangeKeyForm ||
normalized.fastEquals(replaced))
+ (cond, replaced, if (canPush) Some(normalized) else None)
}
- (pushDown.result(), stayUp.result())
+ // The aliases are expanded in the pushed conjunct, so a conjunct over a
throwable alias is
+ // throwable too.
+ val (pushDown, stayUp) = splitPushable(candidates)(
+ canPush = _._3.isDefined,
+ throwable = { case (_, replaced, _) => replaced.throwable })
+ (pushDown.map(_._3.get), stayUp.map(_._1))
}
// Rewrites `e` so that it gives the same result on the raw values of
`grouping` as `e` gives on
@@ -2903,20 +2910,18 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
private def pushDownPredicate(
filter: Filter,
- grandchild: LogicalPlan)(insertFilter: Expression => LogicalPlan):
LogicalPlan = {
+ grandchild: LogicalPlan,
+ keepThrowableUp: Boolean)(insertFilter: Expression => LogicalPlan):
LogicalPlan = {
// Only push down the predicates that is deterministic and all the
referenced attributes
- // come from grandchild.
+ // come from grandchild. With `keepThrowableUp`, a throwable predicate is
not pushed even when
+ // it is written first, as the operator in between can drop rows it would
then see.
// TODO: non-deterministic predicates could be pushed through some
operators that do not change
// the rows.
- val (candidates, nonDeterministic) =
- splitConjunctivePredicates(filter.condition).partition(_.deterministic)
-
- val (pushDown, rest) = candidates.partition { cond =>
- cond.references.subsetOf(grandchild.outputSet)
+ val (pushDown, stayUp) = splitPushableConjuncts(filter.condition) { cond =>
+ cond.deterministic && !(keepThrowableUp && cond.throwable) &&
+ cond.references.subsetOf(grandchild.outputSet)
}
- val stayUp = rest ++ nonDeterministic
-
if (pushDown.nonEmpty) {
val newChild = insertFilter(pushDown.reduceLeft(And))
if (stayUp.nonEmpty) {
@@ -2929,6 +2934,29 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
}
}
+ /**
+ * Splits `conds`, the conjuncts of a filter in their written order, into
the ones to push below
+ * an operator and the ones to keep above it. `canPush` tells whether the
operator lets a
+ * conjunct through, and `throwable` whether it may throw (e.g.
`raise_error`). Every conjunct
+ * written before the first one that cannot be pushed is pushed. After it,
only the conjuncts
+ * that are not throwable are pushed, so a throwable conjunct is not
evaluated on rows the
+ * conjuncts before it would have filtered out. Moving a non-throwable
conjunct below a
+ * throwable one is fine, as it can only reduce the rows the throwable one
sees. The conjuncts
+ * to keep above are returned in the order they were written.
+ */
+ private def splitPushable[T](conds: Seq[T])(
+ canPush: T => Boolean, throwable: T => Boolean): (Seq[T], Seq[T]) = {
+ val (leading, rest) = conds.span(canPush)
+ // The head of `rest`, if any, is the first conjunct that cannot be pushed.
+ val (pushedRest, stayUp) = rest.drop(1).partition(c => canPush(c) &&
!throwable(c))
+ (leading ++ pushedRest, rest.take(1) ++ stayUp)
+ }
+
+ /** [[splitPushable]] over the conjuncts of `condition`, checking each for
`throwable`. */
+ private def splitPushableConjuncts(condition: Expression)(
+ canPush: Expression => Boolean): (Seq[Expression], Seq[Expression]) =
+ splitPushable(splitConjunctivePredicates(condition))(canPush, _.throwable)
+
/**
* Splits `project` into a stack of [[Project]]s with the expensive
conditions in
* `expensiveConds` between them, so an expensive expression only runs on
the rows the filters
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
index d10757b7219a..132db7b97887 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
@@ -821,6 +821,169 @@ class FilterPushdownSuite extends PlanTest {
assertTopFilterConjuncts(Optimize.execute(originalQuery),
Seq(nonDeterministic, unplaced))
}
+ // Raises for the rows with a <= 5, so it must only see the rows the
conditions before it kept.
+ private val throwableOnA = attrA > 5 || IsNull(RaiseError(Literal("boom")))
+
+ test("SPARK-60633: a throwable condition is not pushed through a Python UDF
evaluation " +
+ "ahead of the conditions before it") {
+ Seq(PythonEvalType.SQL_BATCHED_UDF,
PythonEvalType.SQL_SCALAR_PANDAS_UDF).foreach { evalType =>
+ withClue(s"eval type $evalType: ") {
+ val pythonUDF = PythonUDF(
+ "pythonUDF", null, IntegerType, Seq(attrA), evalType,
udfDeterministic = true)
+ val f = AttributeReference("f", IntegerType)()
+ def evalPython(child: LogicalPlan): LogicalPlan =
+ if (evalType == PythonEvalType.SQL_BATCHED_UDF) {
+ BatchEvalPython(Seq(pythonUDF), Seq(f), child)
+ } else {
+ ArrowEvalPython(Seq(pythonUDF), Seq(f), child, evalType)
+ }
+
+ // f > 5 cannot be pushed, so the throwable condition after it stays
above too. b > 1
+ // is still pushed, as that only reduces the rows the throwable
condition sees.
+ val originalQuery = Filter(f > 5 && throwableOnA && attrB > 1,
evalPython(testRelation))
+ val correctAnswer =
+ Filter(f > 5 && throwableOnA, evalPython(Filter(attrB > 1,
testRelation)))
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+ // With nothing left above before it, a throwable condition is still
pushed.
+ val throwableFirst = Filter(throwableOnA && f > 5,
evalPython(testRelation))
+ val throwableFirstAnswer = Filter(f > 5,
evalPython(Filter(throwableOnA, testRelation)))
+ comparePlans(Optimize.execute(throwableFirst), throwableFirstAnswer)
+
+ // The conditions left above keep their written order.
+ val nonDeterministic = Rand(10) > 0.5
+ val mixed = Filter(nonDeterministic && f > 5 && throwableOnA,
evalPython(testRelation))
+ assertTopFilterConjuncts(
+ Optimize.execute(mixed), Seq(nonDeterministic, f > 5, throwableOnA))
+ }
+ }
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a sort ahead
of a " +
+ "non-deterministic condition before it") {
+ val originalQuery = testRelation.orderBy($"b".asc)
+ .where(Rand(10) > 0.5 && throwableOnA && $"b" > 1)
+ .analyze
+ val Seq(nonDeterministic, throwable, _) = splitConjunctivePredicates(
+ originalQuery.asInstanceOf[Filter].condition)
+ val correctAnswer = testRelation.where($"b" > 1).orderBy($"b".asc)
+ .where(Rand(10) > 0.5 && throwableOnA)
+ .analyze
+
+ val optimized = Optimize.execute(originalQuery)
+ comparePlans(optimized, correctAnswer)
+ assertTopFilterConjuncts(optimized, Seq(nonDeterministic, throwable))
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a generate
that can drop " +
+ "rows") {
+ val throwable = $"a" > 5 || IsNull(RaiseError(Literal("boom")))
+ // The generate outputs no row for an empty array, so below it the
throwable condition would
+ // see rows it never sees above it, even though nothing is written before
it.
+ val originalQuery = testRelationWithArrayType
+ .generate(Explode($"c_arr"), alias = Some("arr"))
+ .where(throwable && $"b" > 1)
+ .analyze
+ val correctAnswer = testRelationWithArrayType.where($"b" > 1)
+ .generate(Explode($"c_arr"), alias = Some("arr"))
+ .where(throwable)
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+ // An outer generate outputs a row for every input row, so the condition
is still pushed.
+ val outerQuery = testRelationWithArrayType
+ .generate(Explode($"c_arr"), outer = true, alias = Some("arr"))
+ .where(throwable && $"b" > 1)
+ .analyze
+ val outerAnswer = testRelationWithArrayType.where(throwable && $"b" > 1)
+ .generate(Explode($"c_arr"), outer = true, alias = Some("arr"))
+ .analyze
+ comparePlans(Optimize.execute(outerQuery), outerAnswer)
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through an aggregate
ahead of the " +
+ "conditions before it") {
+ val originalQuery = testRelation.groupBy($"a")($"a", count($"b") as "c")
+ .where($"c" > 1 && throwableOnA && $"a" > 1)
+ .analyze
+ val correctAnswer = testRelation.where($"a" > 1).groupBy($"a")($"a",
count($"b") as "c")
+ .where($"c" > 1 && throwableOnA)
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+ // With nothing left above before it, a throwable condition is still
pushed.
+ val throwableFirst = testRelation.groupBy($"a")($"a", count($"b") as "c")
+ .where(throwableOnA && $"c" > 1)
+ .analyze
+ val throwableFirstAnswer = testRelation.where(throwableOnA)
+ .groupBy($"a")($"a", count($"b") as "c")
+ .where($"c" > 1)
+ .analyze
+ comparePlans(Optimize.execute(throwableFirst), throwableFirstAnswer)
+ }
+
+ test("SPARK-60633: a condition over a throwable alias is not pushed through
an aggregate " +
+ "ahead of the conditions before it") {
+ val originalQuery = testRelation
+ .groupBy($"a")($"a", count($"b") as "c", throwableOnA as "x")
+ .where($"c" > 1 && $"x" && $"a" > 1)
+ .analyze
+ // `x` is not throwable itself, but pushing `x` evaluates the throwable
expression behind it.
+ val correctAnswer = testRelation.where($"a" > 1)
+ .groupBy($"a")($"a", count($"b") as "c", throwableOnA as "x")
+ .where($"c" > 1 && $"x")
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a window
ahead of the " +
+ "conditions before it") {
+ val winExpr = windowExpr(count($"b"),
+ windowSpec($"a" :: Nil, $"b".asc :: Nil, UnspecifiedFrame))
+ val originalQuery = testRelation
+ .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+ .where($"window" > 1 && throwableOnA && $"a" > 1)
+ .analyze
+ val correctAnswer = testRelation.where($"a" > 1)
+ .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+ .where($"window" > 1 && throwableOnA)
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+
+ // The conditions left above keep their written order.
+ val withRand = testRelation
+ .window(winExpr.as("window") :: Nil, $"a" :: Nil, $"b".asc :: Nil)
+ .where(Rand(10) > 0.5 && $"window" > 1 && $"a" > 1)
+ .analyze
+ val Seq(nonDeterministic, stayUp, _) = splitConjunctivePredicates(
+ withRand.asInstanceOf[Filter].condition)
+ assertTopFilterConjuncts(Optimize.execute(withRand), Seq(nonDeterministic,
stayUp))
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a union ahead
of a " +
+ "non-deterministic condition before it") {
+ val testRelation2 = LocalRelation($"d".int, $"e".int, $"f".int)
+ val originalQuery = Union(Seq(testRelation, testRelation2))
+ .where(Rand(10) > 0.5 && throwableOnA && $"b" > 1)
+ .analyze
+ val correctAnswer = Union(Seq(testRelation.where($"b" > 1),
testRelation2.where($"e" > 1)))
+ .where(Rand(10) > 0.5 && throwableOnA)
+ .analyze
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+ }
+
+ test("SPARK-60633: a throwable condition is not pushed through a watermark
ahead of the " +
+ "conditions before it") {
+ val interval = new CalendarInterval(2, 2, 2000L)
+ val relation = LocalRelation(Seq(attrA, $"b".timestamp, attrC), Nil,
isStreaming = true)
+ val nodeId = UUID.randomUUID()
+ val originalQuery = EventTimeWatermark(nodeId, $"b", interval, relation)
+ .where($"b" === new java.sql.Timestamp(0) && throwableOnA && $"c" === 5)
+ val correctAnswer = EventTimeWatermark(nodeId, $"b", interval,
relation.where($"c" === 5))
+ .where($"b" === new java.sql.Timestamp(0) && throwableOnA)
+ comparePlans(Optimize.execute(originalQuery.analyze),
correctAnswer.analyze)
+ }
+
test("SPARK-55014: do not split the projection when configured") {
withSQLConf(SQLConf.SPLIT_PROJECTION_FOR_EXPENSIVE_FILTERS.key -> "false")
{
val originalQuery = testStringRelation
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
index d7d38f637c74..d2ba7f17ee96 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala
@@ -2949,6 +2949,48 @@ class DataFrameSuite extends SharedSparkSession
}
}
+ test("SPARK-60633: a throwable filter condition is not pushed through an
operator ahead of " +
+ "its order") {
+ def df: DataFrame = spark.range(0, 10, 1, 1).toDF()
+ // `assert_true(id > 5)` only holds for the rows the conjunct before it
keeps, and that
+ // conjunct cannot be pushed through the operator below the filter. Each
query is built
+ // inside the config loop, so it is planned under that config.
+ def generateQuery(): DataFrame = df.select($"id", explode(array($"id")) as
"x")
+ .where($"x" > 5 && assert_true($"id" > 5).isNull)
+ def aggregateQuery(): DataFrame = df.groupBy($"id").agg(max($"id") as "m")
+ .where($"m" > 5 && assert_true($"id" > 5).isNull)
+ def windowQuery(): DataFrame =
+ df.select($"id", max($"id").over(Window.partitionBy($"id")) as "m")
+ .where($"m" > 5 && assert_true($"id" > 5).isNull)
+ // `rand(0) < 0` keeps no row, so `assert_true` never runs.
+ def unionQuery(): DataFrame =
+ df.union(df).where(rand(0) < 0 && assert_true($"id" > 100).isNull)
+ // The generate only outputs the rows with `id > 5`, so the filter above
it never raises,
+ // although its only condition is written first.
+ def droppingGenerateQuery(): DataFrame =
+ df.select($"id", explode(filter(array($"id"), x => x > 5)) as "x")
+ .where(assert_true($"id" > 5).isNull)
+ // The throwable part is an aggregate result column, which the filter only
reads.
+ def aggregateAliasQuery(): DataFrame =
+ df.groupBy($"id").agg(max($"id") as "m", assert_true($"id" > 5) as "x")
+ .where($"m" > 5 && $"x".isNull)
+ val expected = (6L until 10L).map(i => Row(i, i))
+ Seq(true, false).foreach { codegen =>
+ withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegen.toString) {
+ checkAnswer(generateQuery(), expected)
+ checkAnswer(aggregateQuery(), expected)
+ checkAnswer(windowQuery(), expected)
+ checkAnswer(unionQuery(), Nil)
+ checkAnswer(droppingGenerateQuery(), expected)
+ // An interpreted aggregate computes every result column for every
group, so the
+ // throwable alias only waits for the filter when codegen defers it
past the filter.
+ if (codegen) {
+ checkAnswer(aggregateAliasQuery(), (6L until 10L).map(i => Row(i, i,
null)))
+ }
+ }
+ }
+ }
+
test("SPARK-60528: collect a nested empty struct") {
val df = spark.range(1).selectExpr("id AS i", "named_struct('a', struct())
AS s")
checkAnswer(df, Row(0L, Row(Row())))
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
index f600711f12a3..1421fce8be27 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/python/PythonUDFSuite.scala
@@ -26,7 +26,7 @@ import org.apache.spark.{SparkEnv, SparkException}
import org.apache.spark.internal.config.Python.PYTHON_UDF_PIPELINED_EXECUTION
import org.apache.spark.sql.{AnalysisException, DataFrame,
IntegratedUDFTestUtils, Row}
import org.apache.spark.sql.catalyst.InternalRow
-import org.apache.spark.sql.functions.{aggregate, array, avg, col, count, lit,
transform}
+import org.apache.spark.sql.functions.{aggregate, array, assert_true, avg,
col, count, lit, transform}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.test.SharedSparkSession
import org.apache.spark.sql.types.{LongType, StringType, StructType}
@@ -408,4 +408,27 @@ class PythonUDFSuite extends SharedSparkSession {
}
}
}
+
+ test("SPARK-60633: a throwable filter condition is not pushed below a Python
UDF " +
+ "evaluation ahead of its order") {
+ assume(shouldTestPythonUDFs)
+ val udfs = Seq(pythonTestUDF) ++ (if (shouldTestPandasUDFs) {
+ Seq(TestScalarPandasUDF(name = "pandasUDF", Some(LongType)))
+ } else {
+ Nil
+ })
+ udfs.foreach { testUDF =>
+ // `assert_true(id > 5)` only holds for the rows `f > 5` keeps. Once the
UDF is extracted
+ // into its own operator, `f > 5` cannot be pushed below it. The query
is built inside the
+ // config loop, so it is planned under that config.
+ def df(): DataFrame = spark.range(0, 10, 1, 1)
+ .select($"id", testUDF($"id") as "f")
+ .where($"f" > 5 && assert_true($"id" > 5).isNull)
+ Seq(true, false).foreach { avoid =>
+ withSQLConf(SQLConf.AVOID_DOUBLE_FILTER_EVAL.key -> avoid.toString) {
+ checkAnswer(df(), (6L until 10L).map(i => Row(i, i)))
+ }
+ }
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]