[
https://issues.apache.org/jira/browse/SPARK-60393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60393:
-----------------------------------
Labels: pull-request-available (was: )
> Whole-stage codegen evaluates the common subexpressions of aggregate update
> expressions before the FILTER clause
> ----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60393
> URL: https://issues.apache.org/jira/browse/SPARK-60393
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Max Gekk
> Priority: Major
> Labels: pull-request-available
>
> AggregateCodegenSupport eliminates the common subexpressions of the update
> expressions of the aggregate functions without their FILTER clauses and
> evaluates them before the do { ... } while (false) block of each FILTER
> (AggregateCodegenSupport.scala, the common sub-expressions block before
> codeToEvalAggFuncs; the with-keys paths of HashAggregateExec do the same).
> With whole-stage codegen disabled, AggregationIterator evaluates If(filter,
> updateExpr, buffer), so the update expressions are not evaluated on a row
> that the FILTER rejects.
> Reproduction with ANSI on:
> {code:sql}
> CREATE TABLE t USING parquet AS SELECT * FROM VALUES (NULL, 1, 2147483647),
> (0, 1, 0) AS v(a, b, y);
> SELECT avg(y + 1) FILTER (WHERE b < 0) FROM t;
> {code}
> Average uses y + 1 in two update expressions, so it is a common
> subexpression. With spark.sql.codegen.wholeStage=true the query raises
> ARITHMETIC_OVERFLOW on the first row although b < 0 is FALSE on it; with
> spark.sql.codegen.wholeStage=false it returns NULL.
> spark.sql.subexpressionElimination.enabled=false does not turn this off.
> Pre-existing on master, found while reviewing SPARK-60315, which fixes the
> evaluation of the FILTER condition itself and does not touch the update
> expressions.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
