[
https://issues.apache.org/jira/browse/SPARK-60377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60377:
-----------------------------------
Labels: pull-request-available (was: )
> FilterExec hoists an IsNotNull check on a non-attribute expression ahead of
> an earlier conjunct under whole-stage codegen
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60377
> URL: https://issues.apache.org/jira/browse/SPARK-60377
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> Since SPARK-51356, FilterExec under whole-stage codegen generates an
> IsNotNull check on a non-attribute null-intolerant expression just before the
> first other conjunct that contains the expression, in both
> GeneratePredicateHelper.generatePredicateCode and the CSE branch of
> FilterExec.doConsume. It does so even when the check is written after that
> conjunct in the condition. If the conjunct is FALSE, And.eval never evaluates
> the later check in interpreted mode, but the generated code evaluates it
> first, so an error the check raises on the row is raised under codegen only.
> Reproduction with ANSI on and Parquet filter pushdown off:
> {code:sql}
> CREATE TABLE t USING parquet AS SELECT * FROM VALUES (1, 'x') AS v(b, s);
> SELECT b FROM t WHERE coalesce(b, cast(s AS INT)) < 0 AND cast(s AS INT) IS
> NOT NULL;
> {code}
> The conjunct coalesce(b, cast(s AS INT)) < 0 is FALSE, so the query returns
> no rows with spark.sql.codegen.wholeStage=false. With whole-stage codegen it
> raises CAST_INVALID_INPUT, because the check cast(s AS INT) IS NOT NULL is
> generated before the conjunct.
> SPARK-60315 (PR #59573) leaves such a check at its position once a conjunct
> that can be NULL has been generated, since the NULL-conjunct flag made the
> hoisted check reachable on more rows. So after SPARK-60315, with the default
> configuration, the hoisting remains before the first conjunct that can be
> NULL, including the first such conjunct itself, e.g. WHERE NOT (a > 0 AND
> coalesce(b, cast(s AS INT)) > 0) AND b < 0 AND cast(s AS INT) IS NOT NULL on
> a row (a, b, s) = (NULL, 1, 'x'), and with
> spark.sql.legacy.filterShortCircuitOnNullConjunct=true.
> This ticket is for the remaining cases: the nested IsNotNull check should be
> hoisted only when it is written before the conjunct in the condition, i.e.
> only the checks with an index below precedingNotNulls(k), so that the
> generated code follows the evaluation order of And.eval. A check written
> after the conjunct is covered by the leftover checks at the end of the
> condition.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
