Ganesha S created SPARK-58211:
---------------------------------
Summary: [SQL] Subexpression elimination evaluates short-circuited
AND/OR operands, causing spurious errors
Key: SPARK-58211
URL: https://issues.apache.org/jira/browse/SPARK-58211
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Ganesha S
Subexpression elimination (SEE) can evaluate a common subexpression that lives
in a short-circuited operand of an AND/OR, defeating the short-circuit
semantics. When evaluating that operand would raise an exception (e.g.
NullPointerException, DIVIDE_BY_ZERO), the query fails even though, per SQL
semantics, the operand should never have been evaluated.
This is a correctness bug: a query that should succeed instead throws at
runtime.
*How to reproduce*
With default configs (`spark.sql.ansi.enabled=true`,
`spark.sql.subexpressionElimination.enabled=true`):
{code:java}
SELECT id != 0 AND 1 / id > 0 AND 1 / id < 1 FROM range(0, 1, 1, 1); {code}
For id = 0, id != 0 is false, so short-circuit evaluation should stop and never
compute 1 / id. Instead the query fails with:
[DIVIDE_BY_ZERO] Division by zero. ... SQLSTATE: 22012
The two-operand form `SELECT id != 0 AND 1 / id > 0 FROM range(0, 1, 1, 1)`
works correctly; the failure appears once there are three or more AND operands
and a common subexpression (1 / id) is shared across the short-circuited
operands.
Disabling subexpression elimination
(`spark.sql.subexpressionElimination.enabled=false`) avoids the error,
confirming SEE as the cause.
*Root cause*
`EquivalentExpressions.skipForShortcut` strips only a single And/Or operand:
{code:java}
expr match {
case and: And => and.left
case or: Or => or.left
case other => other
}
{code}
A chained predicate a AND b AND c is parsed left-deep as And(And(a, b), c).
Peeling one level yields And(a, b), whose children [a, b] are then recursed
into as if both are always evaluated. But under short-circuit semantics only
the first operand (a) is always evaluated; b and c are conditional. A
subexpression shared with b or c is therefore hoisted and evaluated eagerly.
By contrast, If/CaseWhen model this correctly via the ConditionalExpression
trait (recursing only alwaysEvaluatedInputs, with branchGroups for safe
cross-branch common subexpressions). And/Or do not.
*Note:* `spark.sql.subexpressionElimination.skipForShortcutExpr=true` (added in
SPARK-42815) only partially mitigates this; it handles simple cases, but still
fails when the shared subexpression is inside a non-first operand of a 3+-way
AND, because it too peels only one level.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
