[
https://issues.apache.org/jira/browse/SPARK-60392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk resolved SPARK-60392.
------------------------------
Resolution: Duplicate
Duplicate of SPARK-60332, which describes the same evaluation of a shared
subexpression before a FALSE conjunct with whole-stage codegen off, filed a day
earlier.
> Whole-stage codegen off evaluates a subexpression shared by filter conjuncts
> before the condition, so a conjunct after a FALSE one can raise
> --------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60392
> URL: https://issues.apache.org/jira/browse/SPARK-60392
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Priority: Major
>
> With whole-stage codegen disabled, FilterExec evaluates its condition through
> Predicate.create, and GeneratePredicate evaluates the common subexpressions
> of the condition before the condition itself (subexprFunctionsCode,
> GeneratePredicate.scala). So a conjunct after a FALSE one is still partly
> evaluated through a subexpression it shares with another conjunct, and an
> error it raises on the row is raised, while whole-stage codegen stops at the
> FALSE conjunct. The generated projection of an aggregate FILTER clause does
> the same.
> Reproduction with ANSI on and Parquet filter pushdown off:
> {code:sql}
> CREATE TABLE t USING parquet AS SELECT * FROM VALUES (NULL, 1, 2147483647),
> (0, 1, 0) AS v(a, b, y);
> SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND b < 0 AND y + 1 > 0 AND y + 1
> > -1;
> {code}
> With spark.sql.codegen.wholeStage=true the query returns no rows, whatever
> spark.sql.legacy.filterShortCircuitOnNullConjunct and
> spark.sql.subexpressionElimination.filterExec.enabled are: b < 0 is FALSE on
> both rows. With spark.sql.codegen.wholeStage=false it raises
> ARITHMETIC_OVERFLOW on the first row, because y + 1 is evaluated before the
> condition.
> This is independent of SPARK-60315: it also happens without a NULL conjunct,
> e.g. with b < 0 as the first conjunct. The SQL standard (9075-1, rule
> evaluation order) lets an implementation skip the inessential parts of a
> search condition and leaves it implementation-dependent whether their
> exceptions are raised, so either behavior is allowed, but the two modes
> should agree.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
