[
https://issues.apache.org/jira/browse/SPARK-60332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60332:
-----------------------------------
Labels: pull-request-available (was: )
> Filter with whole-stage codegen off evaluates a common subexpression of later
> conjuncts before an earlier FALSE conjunct
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60332
> URL: https://issues.apache.org/jira/browse/SPARK-60332
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.3.0, 5.0.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> With whole-stage codegen disabled, {{FilterExec}} evaluates its condition
> with {{Predicate.create}}, which generates a {{SpecificPredicate}} with
> subexpression elimination. {{GeneratePredicate}} evaluates every common
> subexpression of the condition before the condition itself, so a common
> subexpression of the conjuncts *after* a FALSE conjunct is evaluated on a row
> that {{And.eval}} would drop at that FALSE conjunct. When the subexpression
> raises an error on that row, the query fails only with whole-stage codegen
> off.
> With whole-stage codegen on, the CSE branch of {{FilterExec.doConsume}} emits
> each precompute just before the first conjunct that references it
> (SPARK-56032), so it keeps the short-circuit and drops the row.
> Reproducer (ANSI mode):
> {code:sql}
> CREATE TABLE t USING parquet AS
> SELECT * FROM VALUES (NULL, 1, 2147483647, NULL), (0, 1, 0, 1) AS v(a, b,
> y, x);
> SET spark.sql.parquet.filterPushdown=false;
> SET spark.sql.codegen.wholeStage=false;
> SELECT a FROM t WHERE x IS NOT NULL AND NOT (a > 0 AND b > 0) AND y + 1 > 0
> AND y + 1 > -1;
> -- [ARITHMETIC_OVERFLOW] overflow. ... SQLSTATE: 22003
> -- with spark.sql.codegen.wholeStage=true: 0
> {code}
> On the first row {{isnotnull(x)}} is FALSE, so the interpreted {{And}} never
> reaches {{y + 1}}, but the common {{y + 1}} is evaluated up front and
> overflows. The results on 3264b00d897 (with SPARK-60315):
> ||setting||codegen on||codegen off||
> |default|0|ARITHMETIC_OVERFLOW|
> |spark.sql.subexpressionElimination.enabled=false|0|0|
> |spark.sql.subexpressionElimination.skipForShortcutExpr=true|0|0|
> The internal config
> {{spark.sql.subexpressionElimination.skipForShortcutExpr}} (SPARK-42815,
> default false) avoids it, but it does so by excluding everything after the
> first operand of an {{AND}}/{{OR}} chain from subexpression elimination, so a
> filter condition gets no elimination at all past its first conjunct. A better
> fix would evaluate a common subexpression lazily, just before the first
> conjunct that references it, as the whole-stage codegen CSE branch of
> {{FilterExec}} does, so that the interpreted path keeps both the sharing and
> the short-circuit.
> This is independent of SPARK-60315 (a NULL conjunct stopping the evaluation
> with codegen on) and of SPARK-60329 (an {{IsNotNull}} check generated later
> than its position with codegen on). The inconsistency here is in the opposite
> direction: codegen off raises where codegen on drops the row.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
