[
https://issues.apache.org/jira/browse/SPARK-60633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh updated SPARK-60633:
--------------------------------
Fix Version/s: 4.4.0
> Do not push a throwable filter condition through a unary operator ahead of
> the conditions written before it
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60633
> URL: https://issues.apache.org/jira/browse/SPARK-60633
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 4.3.0, 4.2.1, 4.0.5, 4.1.4, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: 5.0.0, 4.4.0, 4.2.2, 4.3.1
>
>
> The generic UnaryNode case of PushPredicateThroughNonJoin (pushDownPredicate)
> pushes every deterministic conjunct whose references are available below the
> operator, and keeps the rest above. It does not check Expression.throwable,
> so a throwable conjunct (raise_error / assert_true) can be pushed below the
> operator ahead of a conjunct written before it that stays above. The
> throwable conjunct then runs on rows the AND short-circuit would have kept
> from it, and the query fails.
> This shows up with Python UDFs: after ExtractPythonUDFs, SparkOptimizer
> reruns PushPredicateThroughNonJoin, and BatchEvalPython / ArrowEvalPython are
> in canPushThrough. For example:
> {code:python}
> from pyspark.sql.functions import col, udf, assert_true
> @udf("long")
> def f(x):
> return x
> spark.range(0, 10, 1, 1).select("id", f("id").alias("f")) \
> .where((col("f") > 5) & assert_true(col("id") > 5).isNull()).collect()
> {code}
> fails with USER_RAISED_EXCEPTION, with
> spark.sql.optimizer.avoidDoubleFilterEval either true or false and for both
> regular and pandas UDFs, because `assert_true(id > 5) IS NULL` is pushed
> alone below the ArrowEvalPython node:
> {code}
> ArrowEvalPython [f(id)], [pythonUDF0]
> +- Filter if ((id > 5)) true else isnull(raise_error(...))
> +- Range (0, 10, step=1, splits=1)
> {code}
> The same applies to the other operators in canPushThrough (Generate, Expand,
> Sort, Repartition, ...) whenever an earlier conjunct cannot be pushed through
> them.
> SPARK-60602 fixes the same problem for the Filter-over-Project case.
> CombineFilters and PushPredicateThroughJoin already keep throwable conditions
> in place.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
