viirya opened a new pull request, #59907:
URL: https://github.com/apache/spark/pull/59907
### What changes were proposed in this pull request?
Backport of #59848 to branch-4.2.
`PushPredicateThroughNonJoin` pushes the cheap conjuncts of a `Filter` over
a `Project` with expensive (UDF, expensive regex) aliases below the projection,
and keeps the expensive ones above (SPARK-47672,
`spark.sql.optimizer.avoidDoubleFilterEval`). It did not check
`Expression.throwable`, so a cheap `raise_error` / `assert_true` conjunct could
be pushed below the projection ahead of an expensive conjunct written before it.
This PR pushes every cheap conjunct written before the first expensive one.
After it, only the cheap conjuncts that are not throwable are pushed, checking
the conjunct with the projection's aliases substituted so a throwable part
inside a projected alias counts too. It also documents the restriction in the
rule's algorithm overview.
branch-4.2 does not have the projection split from SPARK-55014, so the
split-related parts of #59848 do not apply here and are left out.
### Why are the changes needed?
A throwable condition can be evaluated on rows that the `AND` short-circuit
would have kept from it, so a query that should succeed fails. With `f` an
expensive UDF:
```scala
spark.range(0, 10)
.select($"id", f($"id") as "f")
.where($"f" > 5 && assert_true($"id" > 5).isNull)
```
The cheap `assert_true` condition is pushed below the projection, ahead of
`f > 5`, and the query fails with `USER_RAISED_EXCEPTION` whenever
`avoidDoubleFilterEval` is true.
### Does this PR introduce _any_ user-facing change?
Yes. Queries like the one above failed with `USER_RAISED_EXCEPTION` in 4.2.0
and later. They now succeed. Filters without throwable conditions are optimized
as before.
### How was this patch tested?
The `FilterPushdownSuite` and `DataFrameSuite` tests from #59848 that apply
without the projection split: the cheap pushdown path, with the throwable part
in the condition and in a projected alias, and an end-to-end test over
whole-stage codegen and `avoidDoubleFilterEval`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
