Max Gekk created SPARK-60393:
--------------------------------
Summary: Whole-stage codegen evaluates the common subexpressions
of aggregate update expressions before the FILTER clause
Key: SPARK-60393
URL: https://issues.apache.org/jira/browse/SPARK-60393
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
AggregateCodegenSupport eliminates the common subexpressions of the update
expressions of the aggregate functions without their FILTER clauses and
evaluates them before the do { ... } while (false) block of each FILTER
(AggregateCodegenSupport.scala, the common sub-expressions block before
codeToEvalAggFuncs; the with-keys paths of HashAggregateExec do the same). With
whole-stage codegen disabled, AggregationIterator evaluates If(filter,
updateExpr, buffer), so the update expressions are not evaluated on a row that
the FILTER rejects.
Reproduction with ANSI on:
{code:sql}
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (NULL, 1, 2147483647), (0,
1, 0) AS v(a, b, y);
SELECT avg(y + 1) FILTER (WHERE b < 0) FROM t;
{code}
Average uses y + 1 in two update expressions, so it is a common subexpression.
With spark.sql.codegen.wholeStage=true the query raises ARITHMETIC_OVERFLOW on
the first row although b < 0 is FALSE on it; with
spark.sql.codegen.wholeStage=false it returns NULL.
spark.sql.subexpressionElimination.enabled=false does not turn this off.
Pre-existing on master, found while reviewing SPARK-60315, which fixes the
evaluation of the FILTER condition itself and does not touch the update
expressions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
