L. C. Hsieh created SPARK-60602:
-----------------------------------
Summary: Do not evaluate throwable filter conditions ahead of the
conditions before them when pushing filters through a projection
Key: SPARK-60602
URL: https://issues.apache.org/jira/browse/SPARK-60602
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
PushPredicateThroughNonJoin can evaluate a throwable filter condition
(raise_error / assert_true) on rows that the conditions written before it would
have filtered out, so a query that succeeds without the reordering fails with
USER_RAISED_EXCEPTION.
There are two paths that ignore Expression.throwable when reordering the
conjuncts of a Filter over a Project with expensive (UDF, expensive regex)
aliases:
1. The projection split added by SPARK-55014
(spark.sql.optimizer.splitProjectionForExpensiveFilters, default true) orders
condition groups by how many not-yet-computed aliases they need. With f and g
expensive UDFs:
{code}
SELECT * FROM (SELECT id, f(id) AS f, g(id) AS g FROM range(10))
WHERE f + g > 10 AND assert_true(g > 5) IS NULL
{code}
The assert_true group needs fewer aliases, so it is split below the projection
and evaluated on ids 0..5, which the AND short-circuit would have skipped.
2. The cheap/expensive partition added by SPARK-47672
(spark.sql.optimizer.avoidDoubleFilterEval, default true) pushes a cheap
conjunct below the projection ahead of an expensive conjunct written before it:
{code}
SELECT * FROM (SELECT id, f(id) AS f FROM range(10))
WHERE f > 5 AND assert_true(id > 5) IS NULL
{code}
This fails regardless of the split config, and is in 4.2.0.
Other filter pushdown rules (CombineFilters, PushPredicateThroughJoin) already
keep throwable conditions in place.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
