Max Gekk created SPARK-60394:
--------------------------------
Summary: Compute the order of IS NOT NULL checks and conjuncts
once for both FilterExec code generation paths
Key: SPARK-60394
URL: https://issues.apache.org/jira/browse/SPARK-60394
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
Whole-stage codegen of a filter condition interleaves the IsNotNull checks
(notNullPreds) and the other conjuncts (otherPreds) in two places that mirror
each other by hand: GeneratePredicateHelper.generatePredicateCode, also used by
the aggregate FILTER clause, and the CSE branch of FilterExec.doConsume, which
emits the CSE state precomputes between the checks and each conjunct. The NOTE
in generatePredicateCode requires any change to one to be mirrored in the other.
SPARK-60315 adds two rules to both copies: the checks that precede a conjunct
in the condition are generated before it once a conjunct that can be NULL has
been generated (precedingNotNullChecks), and a check on an expression written
after the conjunct that contains it is then no longer hoisted before it. A
mismatch between the copies shows only for filters with a non-cheap common
subexpression, which is why the SPARK-60315 tests add conjuncts such as b * 2 >
0 AND b * 2 > -1 and count multiplyExact in the generated code. Earlier
CSE-only bugs came from the same split, e.g. the one fixed by a4399f0c6e7
(SPARK-56032 follow-up), which added the NOTE.
Compute the order once, as a sequence of checks and conjuncts (with, per
conjunct, the CSE states to precompute before it) that both emitters consume,
so that SPARK-60329 and SPARK-60377, which change the placement of the checks
again, need one edit each.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
