[
https://issues.apache.org/jira/browse/SPARK-60377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Max Gekk updated SPARK-60377:
-----------------------------
Description:
Since SPARK-51356, FilterExec under whole-stage codegen generates an IsNotNull
check on a non-attribute null-intolerant expression just before the first other
conjunct that contains the expression, in both
GeneratePredicateHelper.generatePredicateCode and the CSE branch of
FilterExec.doConsume. It does so even when the check is written after that
conjunct in the condition. If the conjunct is FALSE, And.eval never evaluates
the later check in interpreted mode, but the generated code evaluates it first,
so an error the check raises on the row is raised under codegen only.
Reproduction with ANSI on and Parquet filter pushdown off:
{code:sql}
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (1, 'x') AS v(b, s);
SELECT b FROM t WHERE coalesce(b, cast(s AS INT)) < 0 AND cast(s AS INT) IS NOT
NULL;
{code}
The conjunct coalesce(b, cast(s AS INT)) < 0 is FALSE, so the query returns no
rows with spark.sql.codegen.wholeStage=false. With whole-stage codegen it
raises CAST_INVALID_INPUT, because the check cast(s AS INT) IS NOT NULL is
generated before the conjunct.
SPARK-60315 (PR #59573) leaves such a check at its position once a conjunct
that can be NULL has been generated, since the NULL-conjunct flag made the
hoisted check reachable on more rows. So after SPARK-60315, with the default
configuration, the hoisting remains before the first conjunct that can be NULL,
including the first such conjunct itself, e.g. WHERE NOT (a > 0 AND coalesce(b,
cast(s AS INT)) > 0) AND b < 0 AND cast(s AS INT) IS NOT NULL on a row (a, b,
s) = (NULL, 1, 'x'), and with
spark.sql.legacy.filterShortCircuitOnNullConjunct=true.
This ticket is for the remaining cases: the nested IsNotNull check should be
hoisted only when it is written before the conjunct in the condition, i.e. only
the checks with an index below precedingNotNulls(k), so that the generated code
follows the evaluation order of And.eval. A check written after the conjunct is
covered by the leftover checks at the end of the condition.
was:
Since SPARK-51356, FilterExec under whole-stage codegen generates an IsNotNull
check on a non-attribute null-intolerant expression just before the first other
conjunct that contains the expression, in both
GeneratePredicateHelper.generatePredicateCode and the CSE branch of
FilterExec.doConsume. It does so even when the check is written after that
conjunct in the condition. If the conjunct is FALSE, And.eval never evaluates
the later check in interpreted mode, but the generated code evaluates it first,
so an error the check raises on the row is raised under codegen only.
Reproduction with ANSI on and Parquet filter pushdown off:
{code:sql}
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (0, 1, 0, 1, 'x') AS v(a,
b, y, x, s);
SELECT a FROM t
WHERE NOT (a > 0 AND b > 0) AND coalesce(b, cast(s AS INT)) < 0 AND cast(s AS
INT) IS NOT NULL;
{code}
The first conjunct is TRUE and the second is FALSE, so the query returns no
rows with spark.sql.codegen.wholeStage=false. With whole-stage codegen it
raises CAST_INVALID_INPUT, because the check cast(s AS INT) IS NOT NULL is
generated before the second conjunct. This happens regardless of
spark.sql.legacy.filterShortCircuitOnNullConjunct.
SPARK-60315 (PR #59573) restricts the hoisting once a conjunct that can be NULL
has been generated, since the NULL-conjunct flag made the hoisted check
reachable on more rows. This ticket is for the remaining case: the nested
IsNotNull check should be hoisted only when it is written before the conjunct
in the condition, i.e. only the checks with an index below
precedingNotNulls(k), so that the generated code follows the evaluation order
of And.eval. The hoisting is then no longer needed for a check written after
the conjunct, which the leftover checks at the end of the condition cover.
> FilterExec hoists an IsNotNull check on a non-attribute expression ahead of
> an earlier conjunct under whole-stage codegen
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60377
> URL: https://issues.apache.org/jira/browse/SPARK-60377
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Priority: Major
>
> Since SPARK-51356, FilterExec under whole-stage codegen generates an
> IsNotNull check on a non-attribute null-intolerant expression just before the
> first other conjunct that contains the expression, in both
> GeneratePredicateHelper.generatePredicateCode and the CSE branch of
> FilterExec.doConsume. It does so even when the check is written after that
> conjunct in the condition. If the conjunct is FALSE, And.eval never evaluates
> the later check in interpreted mode, but the generated code evaluates it
> first, so an error the check raises on the row is raised under codegen only.
> Reproduction with ANSI on and Parquet filter pushdown off:
> {code:sql}
> CREATE TABLE t USING parquet AS SELECT * FROM VALUES (1, 'x') AS v(b, s);
> SELECT b FROM t WHERE coalesce(b, cast(s AS INT)) < 0 AND cast(s AS INT) IS
> NOT NULL;
> {code}
> The conjunct coalesce(b, cast(s AS INT)) < 0 is FALSE, so the query returns
> no rows with spark.sql.codegen.wholeStage=false. With whole-stage codegen it
> raises CAST_INVALID_INPUT, because the check cast(s AS INT) IS NOT NULL is
> generated before the conjunct.
> SPARK-60315 (PR #59573) leaves such a check at its position once a conjunct
> that can be NULL has been generated, since the NULL-conjunct flag made the
> hoisted check reachable on more rows. So after SPARK-60315, with the default
> configuration, the hoisting remains before the first conjunct that can be
> NULL, including the first such conjunct itself, e.g. WHERE NOT (a > 0 AND
> coalesce(b, cast(s AS INT)) > 0) AND b < 0 AND cast(s AS INT) IS NOT NULL on
> a row (a, b, s) = (NULL, 1, 'x'), and with
> spark.sql.legacy.filterShortCircuitOnNullConjunct=true.
> This ticket is for the remaining cases: the nested IsNotNull check should be
> hoisted only when it is written before the conjunct in the condition, i.e.
> only the checks with an index below precedingNotNulls(k), so that the
> generated code follows the evaluation order of And.eval. A check written
> after the conjunct is covered by the leftover checks at the end of the
> condition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
