SEPURI-SAI-KRISHNA opened a new pull request, #59668:
URL: https://github.com/apache/spark/pull/59668
### What changes were proposed in this pull request?
`In` evaluates `value` and then the list elements until one of them matches,
in
`eval` and in generated code alike. An element past the match is never
evaluated,
and no element is evaluated at all when `value` is NULL.
`EquivalentExpressions.skipForShortcut` already peels `And`/`Or` down to the
single operand that is always evaluated, so that a subexpression shared by
short-circuited operands is not hoisted out and evaluated eagerly. `In` was
not
peeled, so recursing into its children counted the conditional list elements
as
always evaluated.
This PR peels `In` down to `value`, next to the existing `And`/`Or` cases,
and
updates the config doc to list `IN`. The `In` expression itself is still
eliminated as a whole when it repeats, because it carries its own
short-circuit;
only its list elements stop being candidates.
The scope is deliberately the same as the existing cases: the peel stays
behind
`spark.sql.subexpressionElimination.skipForShortcutExpr`, which is `false` by
default, so `IN` now behaves exactly like `AND`/`OR` under that config rather
than differently from them. Whether that default should change is a separate
question that applies equally to `AND`/`OR`.
### Why are the changes needed?
A query that must return rows fails instead. In ANSI mode:
```sql
CREATE OR REPLACE TEMP VIEW t AS
SELECT cast(id as int) AS b, cast(2147483647 - id as int) AS y FROM
range(1);
SELECT b IN (b, y + 1), b IN (b, y + 1, 7) FROM t;
```
`b` matches the first element of both lists, so `y + 1` is never needed, but
the
query fails with `[ARITHMETIC_OVERFLOW]`. Either `IN` on its own returns
`true`,
and so does the query with `spark.sql.subexpressionElimination.enabled` set
to
`false`, so the error comes from the elimination and not from the query. It
fails
with whole-stage codegen both on and off.
`In` short-circuits the same way `And` and `Or` do, and is already listed
alongside them as conditionally evaluated in `OptimizeCsvJsonExprs`.
### Does this PR introduce _any_ user-facing change?
Yes, with `spark.sql.subexpressionElimination.skipForShortcutExpr` set to
`true`.
A query whose `IN` list element is a common subexpression no longer
evaluates that
element before the comparison, so it no longer raises an error that the
short-circuit should have avoided. The query above returns `true, true`
instead of
failing. Nothing changes under the default configuration, and no result that
was
already correct changes: the elimination only stops hoisting an element,
which the
short-circuit may then skip.
### How was this patch tested?
New tests, both of which fail without the change:
- `SubexpressionEliminationSuite`: a unit test over `EquivalentExpressions`
covering a subexpression in a conditional element in each position, `value`
staying eligible, the peel reaching `value` through an `And` chain and an
`In`
inside an `And`, two `In`s sharing an element (the reported shape), and a
repeated `In` still being eliminated as a whole.
- `SQLQuerySuite`: an end-to-end test next to the existing `SPARK-58211` and
`SPARK-59579` tests, in the same shape, which raises `[DIVIDE_BY_ZERO]`
without
the change.
Ran `SubexpressionEliminationSuite`, `PredicateSuite` and `OptimizeInSuite`
(78 tests) and the `SPARK-58211`, `SPARK-59579` and `SPARK-60418` tests of
`SQLQuerySuite`, all passing.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
