Max Gekk created SPARK-60392:
--------------------------------
Summary: Whole-stage codegen off evaluates a subexpression shared
by filter conjuncts before the condition, so a conjunct after a FALSE one can
raise
Key: SPARK-60392
URL: https://issues.apache.org/jira/browse/SPARK-60392
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
With whole-stage codegen disabled, FilterExec evaluates its condition through
Predicate.create, and GeneratePredicate evaluates the common subexpressions of
the condition before the condition itself (subexprFunctionsCode,
GeneratePredicate.scala). So a conjunct after a FALSE one is still partly
evaluated through a subexpression it shares with another conjunct, and an error
it raises on the row is raised, while whole-stage codegen stops at the FALSE
conjunct. The generated projection of an aggregate FILTER clause does the same.
Reproduction with ANSI on and Parquet filter pushdown off:
{code:sql}
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (NULL, 1, 2147483647), (0,
1, 0) AS v(a, b, y);
SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND b < 0 AND y + 1 > 0 AND y + 1 >
-1;
{code}
With spark.sql.codegen.wholeStage=true the query returns no rows, whatever
spark.sql.legacy.filterShortCircuitOnNullConjunct and
spark.sql.subexpressionElimination.filterExec.enabled are: b < 0 is FALSE on
both rows. With spark.sql.codegen.wholeStage=false it raises
ARITHMETIC_OVERFLOW on the first row, because y + 1 is evaluated before the
condition.
This is independent of SPARK-60315: it also happens without a NULL conjunct,
e.g. with b < 0 as the first conjunct. The SQL standard (9075-1, rule
evaluation order) lets an implementation skip the inessential parts of a search
condition and leaves it implementation-dependent whether their exceptions are
raised, so either behavior is allowed, but the two modes should agree.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
