cloud-fan commented on PR #59573:
URL: https://github.com/apache/spark/pull/59573#issuecomment-6257890434
Thanks for the detailed write-up and the cross-engine comparison. The stop
rule now matches And.eval, but I think the flag interacts badly with the
deferred IsNotNull checks. Both codegen branches emit an IsNotNull just before
the first conjunct that references its column, or after all conjuncts. With
codegen off, the condition is evaluated in order, and the IsNotNull predicates
that InferFiltersFromConstraints prepends come first. Take `WHERE NOT (a > 0
AND b > 0) AND y + 1 > 0 AND x > 0` on a row with a NULL, b = 1, y =
Int.MaxValue, x NULL. Codegen off drops the row at isnotnull(x). With this PR,
codegen sets the flag, evaluates y + 1, and fails with ARITHMETIC_OVERFLOW.
Before the PR, both modes returned without an error. A user-written guard
written first has the same problem: in `WHERE x IS NOT NULL AND NOT (a > 0 AND
b > 0) AND y + 1 > 0`, isnotnull(x) is emitted after all the other conjuncts.
One way to fix it: once a flag is in play, emit each pending attribute-only
IsNot
Null check just before the first later conjunct it precedes in the condition.
A check written after the overflowing conjunct, as in `... AND y + 1 > 0 AND x
IS NOT NULL`, then stays where it is, and that query still raises in both
modes. Hoisting all pending checks right after the NULL conjunct would drop
that row with codegen on and bring back the original inconsistency for that
shape. What do you think?
<!-- SPARK_DEV_REVIEW_REPLY
{"feedback_id":"conversation:6233128726","thread_id":"conversation:6233128726","verdict_sha256":"8416652c1630a56960b29dad840c9dd175c381d6716e5c914f06da2391389792"}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
