Max Gekk created SPARK-60332:
--------------------------------
Summary: Filter with whole-stage codegen off evaluates a common
subexpression of later conjuncts before an earlier FALSE conjunct
Key: SPARK-60332
URL: https://issues.apache.org/jira/browse/SPARK-60332
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0, 4.1.3, 4.0.4, 4.3.0, 4.4.0, 5.0.0
Reporter: Max Gekk
With whole-stage codegen disabled, {{FilterExec}} evaluates its condition with
{{Predicate.create}}, which generates a {{SpecificPredicate}} with
subexpression elimination. {{GeneratePredicate}} evaluates every common
subexpression of the condition before the condition itself, so a common
subexpression of the conjuncts *after* a FALSE conjunct is evaluated on a row
that {{And.eval}} would drop at that FALSE conjunct. When the subexpression
raises an error on that row, the query fails only with whole-stage codegen off.
With whole-stage codegen on, the CSE branch of {{FilterExec.doConsume}} emits
each precompute just before the first conjunct that references it
(SPARK-56032), so it keeps the short-circuit and drops the row.
Reproducer (ANSI mode):
{code:sql}
CREATE TABLE t USING parquet AS
SELECT * FROM VALUES (NULL, 1, 2147483647, NULL), (0, 1, 0, 1) AS v(a, b, y,
x);
SET spark.sql.parquet.filterPushdown=false;
SET spark.sql.codegen.wholeStage=false;
SELECT a FROM t WHERE x IS NOT NULL AND NOT (a > 0 AND b > 0) AND y + 1 > 0 AND
y + 1 > -1;
-- [ARITHMETIC_OVERFLOW] overflow. ... SQLSTATE: 22003
-- with spark.sql.codegen.wholeStage=true: 0
{code}
On the first row {{isnotnull(x)}} is FALSE, so the interpreted {{And}} never
reaches {{y + 1}}, but the common {{y + 1}} is evaluated up front and
overflows. The results on 3264b00d897 (with SPARK-60315):
||setting||codegen on||codegen off||
|default|0|ARITHMETIC_OVERFLOW|
|spark.sql.subexpressionElimination.enabled=false|0|0|
|spark.sql.subexpressionElimination.skipForShortcutExpr=true|0|0|
The internal config {{spark.sql.subexpressionElimination.skipForShortcutExpr}}
(SPARK-42815, default false) avoids it, but it does so by excluding everything
after the first operand of an {{AND}}/{{OR}} chain from subexpression
elimination, so a filter condition gets no elimination at all past its first
conjunct. A better fix would evaluate a common subexpression lazily, just
before the first conjunct that references it, as the whole-stage codegen CSE
branch of {{FilterExec}} does, so that the interpreted path keeps both the
sharing and the short-circuit.
This is independent of SPARK-60315 (a NULL conjunct stopping the evaluation
with codegen on) and of SPARK-60329 (an {{IsNotNull}} check generated later
than its position with codegen on). The inconsistency here is in the opposite
direction: codegen off raises where codegen on drops the row.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
