Max Gekk created SPARK-60774:
--------------------------------
Summary: Common subexpression first referenced in a conditional
branch of a predicate's conjunct is evaluated before the conjunct
Key: SPARK-60774
URL: https://issues.apache.org/jira/browse/SPARK-60774
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
When a common subexpression of a predicate's conjuncts is evaluated just before
the first conjunct that references it (the CSE branch of FilterExec.doConsume
since SPARK-56032, and GeneratePredicate since SPARK-60332), the first
reference is found with Expression.exists, which also counts an occurrence in a
conditionally evaluated part of the conjunct: a branch of CASE WHEN or IF, the
later arguments of COALESCE, or the right side of OR. So the subexpression is
evaluated before a conjunct that may never evaluate it, and an error it raises
on the row is raised although the conjunct is FALSE and And.eval would drop the
row without the error.
Reproducer (ANSI mode, both with whole-stage codegen on and off, Parquet filter
pushdown off):
{code:sql}
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (NULL, 1, 2147483647), (0,
1, 0) AS v(a, b, y);
SELECT a FROM t WHERE (CASE WHEN a > 0 THEN y + 1 ELSE 0 END) > 0 AND y + 1 > 0
AND y + 1 > -1;
{code}
y + 1 is common to the last two conjuncts, but its first occurrence is in the
THEN branch of the first conjunct, so it is evaluated before that conjunct and
raises ARITHMETIC_OVERFLOW on the first row, where the first conjunct is FALSE
(a is NULL, so the CASE gives 0). The interpreted And.eval returns no rows and
no error.
Evaluating the subexpression at its first unconditional reference instead is
not enough: the code of the conditional branch is generated through
Expression.genCode and would read the subexpression's state before it is
evaluated. The subexpression has to be left out of the elimination, or its
conditional occurrence left unsubstituted, when its first occurrence is
conditional. The first reference is computed in one place,
CodegenContext.firstReferencingExpression, shared by both generators.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
