[
https://issues.apache.org/jira/browse/SPARK-58211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-58211.
---------------------------------
Fix Version/s: 4.3.0
Resolution: Fixed
Issue resolved by pull request 57368
[https://github.com/apache/spark/pull/57368]
> [SQL] Subexpression elimination evaluates short-circuited AND/OR operands,
> causing spurious errors
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-58211
> URL: https://issues.apache.org/jira/browse/SPARK-58211
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0
> Reporter: Ganesha S
> Assignee: Ganesha S
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.3.0
>
>
> Subexpression elimination (SEE) can evaluate a common subexpression that
> lives in a short-circuited operand of an AND/OR, defeating the short-circuit
> semantics. When evaluating that operand would raise an exception (e.g.
> NullPointerException, DIVIDE_BY_ZERO), the query fails even though, per SQL
> semantics, the operand should never have been evaluated.
> This is a correctness bug: a query that should succeed instead throws at
> runtime.
> *How to reproduce*
> With default configs (`spark.sql.ansi.enabled=true`,
> `spark.sql.subexpressionElimination.enabled=true`):
> {code:java}
> SELECT id != 0 AND 1 / id > 0 AND 1 / id < 1 FROM range(0, 1, 1, 1); {code}
> For id = 0, id != 0 is false, so short-circuit evaluation should stop and
> never compute 1 / id. Instead the query fails with:
>
> The two-operand form `SELECT id != 0 AND 1 / id > 0 FROM range(0, 1, 1, 1)`
> works correctly; the failure appears once there are three or more AND
> operands and a common subexpression (1 / id) is shared across the
> short-circuited operands.
> Disabling subexpression elimination
> (`spark.sql.subexpressionElimination.enabled=false`) avoids the error,
> confirming SEE as the cause.
> *Root cause*
> `EquivalentExpressions.skipForShortcut` strips only a single And/Or operand:
>
> {code:java}
> expr match {
> case and: And => and.left
> case or: Or => or.left
> case other => other
> }
> {code}
> A chained predicate a AND b AND c is parsed left-deep as And(And(a, b), c).
> Peeling one level yields And(a, b), whose children [a, b] are then recursed
> into as if both are always evaluated. But under short-circuit semantics only
> the first operand (a) is always evaluated; b and c are conditional. A
> subexpression shared with b or c is therefore hoisted and evaluated eagerly.
> By contrast, If/CaseWhen model this correctly via the ConditionalExpression
> trait (recursing only alwaysEvaluatedInputs, with branchGroups for safe
> cross-branch common subexpressions). And/Or do not.
> *Note:* `spark.sql.subexpressionElimination.skipForShortcutExpr=true` (added
> in SPARK-42815) only partially mitigates this; it handles simple cases, but
> still fails when the shared subexpression is inside a non-first operand of a
> 3+-way AND, because it too peels only one level.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
