dongjoon-hyun opened a new pull request, #60162:
URL: https://github.com/apache/spark/pull/60162
### What changes were proposed in this pull request?
`InferFiltersFromConstraints` turns data constraints into filters. This PR
makes it skip a constraint when evaluating that constraint as a filter can
fail. A constraint becomes a filter only if
`ExprUtils.canEvaluateUnconditionally` accepts it, with these parts treated as
safe so that the useful inference stays:
- Constant parts count as the value they evaluate to, because
`ConstantFolding` folds them right after this rule. For example, `b.d >=
date_sub(a.dt, 7)` with `a.dt = DATE'2026-06-10'` (SPARK-57437) still becomes
`b.d >= DATE'2026-06-03'`. A constant that fails to evaluate does not become a
filter.
- Widening numeric casts, which type coercion adds to join keys.
- Uncorrelated subqueries, which run once whichever rows use them. This
keeps the semi-joins inferred for TPC-H q18 and q20 and TPC-DS q14 and q23b.
`Filter.constraints` does not change. Constraints are still facts about the
output rows, and `PruneFilters` and `EliminateOuterJoin` rely on them. Only the
two places that turn constraints into filters change, and `IsNotNull` is still
inferred from a predicate that can fail.
`spark.sql.legacy.inferFailableFiltersFromConstraints` restores the previous
behavior, and the migration guide describes the change.
Two alternatives were measured and rejected:
- Moving the inferred filters after the original condition changes 343 plan
stability golden files and does not help joins, because there the filter goes
to a relation that has no such condition.
- Keeping only the filters that can fail after the original condition, on
the `Filter` path, still fails the query on a partition column, and still
evaluates a UDF twice per row.
### Why are the changes needed?
A constraint only says what holds on the rows an operator outputs. As a
filter, it also runs on rows and values that the query does not evaluate it on.
So this rule can add an expression that is not in the query, and fail a query
that does not fail in any evaluation order of its own conjuncts.
```sql
-- t has the rows (0, 6) and (6, 6).
SELECT * FROM t WHERE x = y AND 10 / y > 0;
```
On `(0, 6)`, `x = y` is false and `10 / y` is about `1.67`, whichever is
evaluated first. The optimizer adds `10 / x > 0` ahead of both, and the query
fails with `DIVIDE_BY_ZERO`.
A join is worse, because the new filter goes to the other relation, where
nothing guards it.
```sql
-- 'abc' exists only in l.s.
SELECT * FROM l JOIN r ON l.s = r.s WHERE CAST(r.s AS INT) > 0;
```
This query fails with `CAST_INVALID_INPUT`, although no row of `r` has a
value that fails.
| Query | Data | Before | After |
|---|---|---|---|
| `x = y AND 10 / y > 0` | `(0, 6), (6, 6)` | `DIVIDE_BY_ZERO` | `(6, 6)` |
| `id > 5 AND x = y AND 10 / (id - y) > 0` | `(0, 0, 0), (6, 0, 0)` |
`DIVIDE_BY_ZERO` | `(6, 0, 0)` |
| `p = y AND 10 / y > 0`, where `p` is a partition column | `(0, 6), (6, 6)`
| `DIVIDE_BY_ZERO` | `(6, 6)` |
| `s1 = s2 AND CAST(s2 AS INT) > 0` | `('abc', '1'), ('1', '1')` |
`CAST_INVALID_INPUT` | `('1', '1')` |
| `l.x = r.x WHERE 10 / r.x > 0` | `l.x`: 0, 6; `r.x`: 6 | `DIVIDE_BY_ZERO`
| `(6, 6)` |
| `a LEFT JOIN b ON a.k = b.k AND 10 / (b.id - a.y) > 0 WHERE a.y = 0` |
`a`: `(1, 0)`; `b`: `(1, 6), (99, 0)` | `DIVIDE_BY_ZERO` | `(1, 0, 1, 6)` |
| `id > 5 AND x = y AND IF(id - y = 0, raise_error('boom'), true)` | `(0, 0,
0), (6, 0, 0)` | `USER_RAISED_EXCEPTION` | `(6, 0, 0)` |
The attribute equality path dates back to SPARK-13789 and SPARK-13871 in
Spark 2.0. It became reachable with default settings when ANSI mode became the
default in [v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0)
(2025-05-19). Without ANSI mode, `raise_error`, `assert_true` and UDFs still
reach it. SPARK-57437, in
[v4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27),
extended the problem to literal bindings: the `LEFT JOIN` query in the table
passes on `branch-4.1` and fails on `master`.
Other optimizer rules already guard against this:
- `InferFiltersFromGenerate` generates predicates only over attributes
(SPARK-37392), because the input "may fail to be evaluated under ANSI mode".
- `PushPredicateThroughJoin` (SPARK-46707) and `PushPredicateThroughNonJoin`
(SPARK-60602, SPARK-60633) hold back `throwable` conditions. An inferred copy
goes around that: over a projection with `f(id) AS f`, `f > 5 AND x = y AND
assert_true(id - y > 5) IS NULL` fails, because the copy is placed first and is
pushed below the projection.
Inferring a UDF also evaluates it twice per row: `x = y AND f(y) >= 0` calls
`f` 20 times on 10 rows, and 10 times after this PR.
### Does this PR introduce _any_ user-facing change?
Yes, compared to the released versions.
- The queries above return rows instead of failing.
- A UDF in such a condition is no longer evaluated twice per row.
- A filter that can fail no longer reaches the other side of a join, a data
source, or partition pruning, so some queries can read more data. In `l JOIN r
ON l.s = r.s WHERE r.s LIKE 'a%'`, `l` now gets only `l.s IS NOT NULL`. The
same applies to `upper(s) = 'A'`, `i + 1 > 5`, `abs(i) > 5`, and narrowing or
string casts.
- Plain comparisons, `IN`, `BETWEEN`, widening numeric casts, constants,
uncorrelated subqueries, and `IS NOT NULL` are still inferred. No TPC-DS or
TPC-H plan changes.
Set `spark.sql.legacy.inferFailableFiltersFromConstraints` to `true` to
restore the previous behavior.
### How was this patch tested?
Added four tests to `InferFiltersFromConstraintsSuite` and one to
`DataFrameSuite`. They cover the `Filter` path, the join path, the legacy
configuration, uncorrelated subqueries, and an `Or` whose canonical form is
`MultiCommutativeOp`, which is why the rule filters the original expressions
rather than calling `ExpressionSet.filter`. The test "filter lower data type"
now expects no narrowing cast to be inferred, as its title says.
Constant folding and widening casts are covered by existing tests. Disabling
each path or allowance in turn fails at least one test.
The plan stability golden files do not change when regenerated with ANSI
mode on and off:
```bash
SPARK_GENERATE_GOLDEN_FILES=1 build/sbt "sql/testOnly *PlanStability*Suite"
SPARK_GENERATE_GOLDEN_FILES=1 SPARK_ANSI_SQL_MODE=false build/sbt
"sql/testOnly *PlanStability*Suite"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
