Max Gekk created SPARK-61197:
--------------------------------
Summary: Evaluate the shared subexpressions of a common AND group
of a predicate after the group's earlier operands
Key: SPARK-61197
URL: https://issues.apache.org/jira/browse/SPARK-61197
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
With whole-stage codegen off, `GeneratePredicate` (SPARK-60332) evaluates a
common subexpression of a predicate's conjuncts just before the first conjunct
that references it, but it keeps an AND of the top-level chain whole when that
AND is itself a common subexpression, for example when the same guard is
repeated as an IF condition. A shared subexpression first referenced inside
such a group then has its function called before the group, so it runs before
the group's first operand is checked.
For example, with ANSI on and whole-stage codegen off, on a row with b = 1 and
y = 2147483647:
SELECT a FROM t WHERE (b < 0 AND y + 1 > 0) AND IF(b < 0 AND y + 1 > 0, a > 0,
a < 0) AND y + 1 < 100
fails with ARITHMETIC_OVERFLOW, as it does without SPARK-60332, while with
whole-stage codegen on `FilterExec` splits every AND, `b < 0` drops the row and
the query returns no rows, as `And.eval` does.
Possible approaches: split such a group like the other ANDs of the chain and
set its state from its operands' checks for its other references, or always
split the chain as `FilterExec` does, at the cost of evaluating the group's AND
again for its other references.
Raised in the review of https://github.com/apache/spark/pull/59984
(https://github.com/apache/spark/pull/59984#discussion_r4505510516).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
