This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 23bf4c48a84d [SPARK-60983][SQL] Fix query errors by
ConstantPropagation when a constant is substituted into an expression that can
fail
23bf4c48a84d is described below
commit 23bf4c48a84df85943176322c9ba2c27ca3c33a5
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 18:37:50 2026 -0800
[SPARK-60983][SQL] Fix query errors by ConstantPropagation when a constant
is substituted into an expression that can fail
### What changes were proposed in this pull request?
`ConstantPropagation` replaces an attribute with the literal of an equality
conjunct, such as `y = 0`, in the other comparisons of the same `AND`. The
literal is the value of the attribute only on the rows that pass the whole
`AND`. However, the rewritten conjuncts also run on other rows, and a part that
becomes constant is folded by `ConstantFolding` before any row is read. So an
expression that can fail, such as a division, an ANSI cast or an array index,
can raise an error that the [...]
- `10 / (id - y) > 0 AND y = 0` becomes `10 / (id - 0) > 0 AND y = 0`,
which divides by zero on a row with `id = 0` and `y = 1`.
- `y = 0 AND id > 10 / y` becomes `y = 0 AND id > 10 / 0`, which fails
during the optimization, even on an empty table.
This PR replaces an attribute only where its value cannot change whether an
expression fails:
- under operators that cannot fail on their own, such as comparisons,
`AND`, `OR`, `NOT`, `IS NULL` and `COALESCE`. This is the allowlist of
`ExprUtils.canEvaluateUnconditionally`, which is now also exposed as
`ExprUtils.isTotalOperator`.
- in a subexpression whose attributes are all replaced, when it folds to a
literal without an error. For example, `y = 5 AND x = y + 1` still becomes `y =
5 AND x = 6`.
Otherwise, the expression keeps its attributes. The comparisons in the
pulled up condition of a correlated `IN` or `EXISTS` subquery are rewritten
within the same boundary.
A conjunct that keeps an attribute can still run on rows where the equality
does not hold, while a guard over the same attribute may become a constant,
such as `d <> 0` in `d <> 0 AND n / d > 1 AND d = 5`. So when a conjunct that
can fail keeps a propagated attribute, this PR also moves the equalities ahead
of the other conjuncts. An equality between an attribute and a literal cannot
fail, so running it first raises no error.
### Why are the changes needed?
The queries below cannot raise an error in any order of their conjuncts,
but fail with `DIVIDE_BY_ZERO`.
```sql
CREATE TABLE t1(id BIGINT, y BIGINT) USING parquet;
INSERT INTO t1 VALUES (0, 1), (6, 0);
-- It should return (6, 0), but fails with DIVIDE_BY_ZERO.
SELECT * FROM t1 WHERE 10 / (id - y) > 0 AND y = 0;
CREATE TABLE t2(id BIGINT, y BIGINT) USING parquet;
-- It should return no rows, but fails with DIVIDE_BY_ZERO, with or without
rows in `t2`.
SELECT * FROM t2 WHERE y = 0 AND id > 10 / y;
```
The same happens with other expressions that can fail. For example, `s =
'abc' AND CAST(s AS INT) > 0` fails with `CAST_INVALID_INPUT` on a table
without `'abc'`, and `arr[i] > 0 AND i = 5` fails with `INVALID_ARRAY_INDEX` on
a row with a shorter array and `i = 1`. A UDF or `assert_true` also fails like
this with ANSI mode off.
`ConstantPropagation` has substituted into all comparisons since
SPARK-42500 (#42038), and ANSI mode has been on by default since SPARK-44444
(#46013). Both are in [v4.0.0](https://github.com/apache/spark/tree/v4.0.0)
(2025-05-19). Before that, it only substituted into `=` and `<=>` (SPARK-20758,
#17993, [v2.3.0](https://github.com/apache/spark/tree/v2.3.0) (2018-02-22)).
This PR should be merged after #60162 (SPARK-60985). Without it,
`InferFiltersFromConstraints` infers the expression that this PR no longer
rewrites, such as `10 / (id - 0) > 0` from `10 / (id - y) > 0 AND y = 0`, and
puts it ahead of the original conditions. Then a query that works today, such
as `SELECT * FROM t WHERE id > 5 AND y = 0 AND 10 / (id - y) > 0` on rows `(0,
0)` and `(6, 0)`, would fail with `DIVIDE_BY_ZERO`.
### Does this PR introduce _any_ user-facing change?
Yes. The queries above, which fail with an error, now return their results.
Some comparisons keep an attribute instead of a literal, for example `y = 5
AND x / y > 1`, `y = 5 AND CASE WHEN y > 0 THEN x ELSE z END > 1`, and `r.d +
l.b > 10` in a correlated subquery with `l.b = 5.0` outside. The results are
the same, but some plans become less efficient. For example, the CASE above no
longer pushes `x > 1` down to the scan, and `l.b = 5.0 AND l.a NOT IN (SELECT c
FROM r WHERE r.d + l.b > 10)` is planned as a broadcast nested loop join
instead of a null-aware br [...]
### How was this patch tested?
- Added `SPARK-60983: Do not propagate a constant into an expression that
can fail` to `ConstantPropagationSuite`. It covers an expression that can fail
and does not become constant, one that becomes constant but fails to fold, one
that folds, a comparison whose other side can fail, a guard that becomes a
constant, after which the equality goes first, and the condition of an `IF`,
which is not rewritten. Without this change, the test fails.
- Added `SPARK-60983: Propagate a constant into a comparison with a
correlated scalar subquery` to `SubquerySuite`. It fails with
`NoSuchElementException` if the columns in the condition of the subquery are
looked up as constants.
- The existing `SPARK-32290: SingleColumn Null Aware Anti Join Optimize` in
`SubquerySuite` covers the comparisons in the condition of a correlated
subquery. It fails if they are not rewritten.
- Ran all suites in `org.apache.spark.sql.catalyst.optimizer`,
`ExprUtilsSuite`, the TPC-DS and TPC-H `PlanStabilitySuite`s with ANSI mode on
and off, `SQLQueryTestSuite`, `SQLQuerySuite`, `DataFrameSuite`, `JoinSuite`,
`SubquerySuite`, `ExplainSuite`, `CachedTableSuite`,
`DynamicPartitionPruningV1SuiteAEOff`, `ParquetV1FilterSuite`,
`ParquetV2FilterSuite`, `OrcV1FilterSuite`, `FileSourceStrategySuite`,
`PruneFileSourcePartitionsSuite` and the DSv2 strategy and SQL suites. All pass
an [...]
- Ran the queries above and the variants with a cast, an array index, a
UDF, `assert_true`, `HAVING`, a window, a view, `<=>`, a guard over the
propagated column and a correlated scalar subquery before and after this
change, together with the change of #60162. They no longer fail, and `y = 0 AND
id > 10 / y` still fails on a row with `y = 0`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60163 from dongjoon-hyun/SPARK-60983.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/catalyst/expressions/ExprUtils.scala | 15 ++--
.../spark/sql/catalyst/optimizer/expressions.scala | 80 +++++++++++++++++++---
.../optimizer/ConstantPropagationSuite.scala | 43 +++++++++++-
.../scala/org/apache/spark/sql/SubquerySuite.scala | 12 ++++
4 files changed, 136 insertions(+), 14 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala
index c88825d41245..d8e7a3e1a6d1 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala
@@ -268,11 +268,18 @@ object ExprUtils extends EvalHelper with QueryErrorsBase {
private def canEvaluateUnconditionallyInternal(e: Expression): Boolean = e
match {
case _: AttributeReference | _: Literal => true
- case _: GetStructField | _: GetArrayStructFields | _: GetMapValue =>
- e.children.forall(canEvaluateUnconditionallyInternal)
+ case _ if isTotalOperator(e) =>
e.children.forall(canEvaluateUnconditionallyInternal)
+ case _ => false
+ }
+
+ /**
+ * Returns true if `e` itself cannot raise an error, whatever the values of
its children. See
+ * [[canEvaluateUnconditionally]] for the whitelist.
+ */
+ def isTotalOperator(e: Expression): Boolean = e match {
+ case _: GetStructField | _: GetArrayStructFields | _: GetMapValue => true
case _: And | _: Or | _: Not | _: BinaryComparison | _: IsNull | _:
IsNotNull |
- _: IsNaN | _: NullIf | _: Coalesce | _: In | _: InSet =>
- e.children.forall(canEvaluateUnconditionallyInternal)
+ _: IsNaN | _: NullIf | _: Coalesce | _: In | _: InSet => true
case _ => false
}
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 842c8e6fed07..9bcc5cde01d4 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -141,7 +141,7 @@ object ConstantFolding extends Rule[LogicalPlan] {
* - Using this mapping, replace occurrence of the attributes with the
corresponding constant values
* in the AND node.
*/
-object ConstantPropagation extends Rule[LogicalPlan] {
+object ConstantPropagation extends Rule[LogicalPlan] with PredicateHelper {
def apply(plan: LogicalPlan): LogicalPlan = plan.transformUpWithPruning(
_.containsAllPatterns(LITERAL, FILTER, BINARY_COMPARISON), ruleId) {
case f: Filter =>
@@ -195,8 +195,10 @@ object ConstantPropagation extends Rule[LogicalPlan] {
traverse(a.right, replaceChildren = false, nullIsFalse)
val equalityPredicates = equalityPredicatesLeft ++
equalityPredicatesRight
val newSelf = if (equalityPredicates.nonEmpty && replaceChildren) {
- Some(And(replaceConstants(newLeft.getOrElse(a.left),
equalityPredicates),
- replaceConstants(newRight.getOrElse(a.right), equalityPredicates)))
+ Some(moveEqualitiesFirst(
+ And(replaceConstants(newLeft.getOrElse(a.left),
equalityPredicates),
+ replaceConstants(newRight.getOrElse(a.right),
equalityPredicates)),
+ equalityPredicates))
} else {
if (newLeft.isDefined || newRight.isDefined) {
Some(And(newLeft.getOrElse(a.left), newRight.getOrElse(a.right)))
@@ -230,6 +232,16 @@ object ConstantPropagation extends Rule[LogicalPlan] {
private def safeToReplace(ar: AttributeReference, nullIsFalse: Boolean) =
!ar.nullable || nullIsFalse
+ /**
+ * Replaces attributes in the comparisons of `condition` with their constant
values.
+ *
+ * The constant is the value of an attribute only on the rows that pass the
whole condition.
+ * The rewritten condition also runs on other rows, e.g. a row with `y = 1`
for
+ * `10 / (id - y) > 0 AND y = 0`, and a part of it that becomes constant is
folded with no row
+ * at all, e.g. `10 / y` for `y = 0 AND id > 10 / y`. So that this raises no
error the original
+ * condition cannot raise, an expression that can fail keeps its attributes,
unless all of them
+ * are replaced and the result evaluates without an error.
+ */
private def replaceConstants(
condition: Expression,
equalityPredicates: AttributeMap[(Literal, BinaryComparison)]):
Expression = {
@@ -237,14 +249,64 @@ object ConstantPropagation extends Rule[LogicalPlan] {
val predicates = equalityPredicates.values.map(_._2).toSet
def replaceInComparison(b: BinaryComparison): Expression = {
lazy val collationSafeReplacement = isSameCollationAttrRefComparison(b)
- b transform {
- case a: AttributeReference
- if isBinaryStable(a.dataType) || collationSafeReplacement =>
- constantsMap.getOrElse(a, a)
+ def canReplace(a: Attribute): Boolean =
+ constantsMap.contains(a) && (isBinaryStable(a.dataType) ||
collationSafeReplacement)
+ def replace(e: Expression): Expression = e match {
+ case a: AttributeReference if canReplace(a) => constantsMap(a)
+ case _ if ExprUtils.isTotalOperator(e) => e.mapChildren(replace)
+ case _ if e.references.nonEmpty && e.references.forall(canReplace) =>
+ fold(e.transform { case a: AttributeReference if canReplace(a) =>
constantsMap(a) })
+ .getOrElse(e)
+ case _ => e
}
+ b.mapChildren(replace)
+ }
+ def replaceInCondition(e: Expression): Expression = e match {
+ case b: BinaryComparison if predicates.contains(b) => b
+ case b: BinaryComparison => replaceInComparison(b)
+ case _ if ExprUtils.isTotalOperator(e) =>
e.mapChildren(replaceInCondition)
+ // The children of these include the comparisons of a pulled up
correlated condition.
+ case _: InSubquery | _: ListQuery | _: Exists =>
e.mapChildren(replaceInCondition)
+ case _ => e
+ }
+ replaceInCondition(condition)
+ }
+
+ /** Returns the literal that `e` folds to, or None if it is not constant or
fails to fold. */
+ private def fold(e: Expression): Option[Literal] = {
+ if (e.foldable) {
+ try {
+ Some(ConstantFolding.constantFolding(e)).collect { case l: Literal =>
l }
+ } catch {
+ case NonFatal(_) => None
+ }
+ } else {
+ None
}
- condition.transformWithPruning(_.containsPattern(BINARY_COMPARISON)) {
- case b: BinaryComparison if !predicates.contains(b) =>
replaceInComparison(b)
+ }
+
+ /**
+ * Moves the equalities ahead of the other conjuncts of `condition` if one
that can fail still
+ * refers to a propagated attribute. Such a conjunct also runs on rows where
an equality does
+ * not hold, with the value of the row, while a guard over the same
attribute may be folded
+ * away, e.g. `n / d > 1` for `d <> 0 AND n / d > 1 AND d = 5`. An equality
between an
+ * attribute and a literal cannot fail, so running it first raises no error.
+ */
+ private def moveEqualitiesFirst(
+ condition: Expression,
+ equalityPredicates: AttributeMap[(Literal, BinaryComparison)]):
Expression = {
+ val predicates = equalityPredicates.values.map(_._2).toSet[Expression]
+ val propagated = AttributeSet(equalityPredicates.keys)
+ val conjuncts = splitConjunctivePredicates(condition)
+ val (equalities, others) = conjuncts.partition(predicates.contains)
+ val reordered = equalities ++ others
+ val keepsFailable = others.exists { c =>
+ c.references.intersect(propagated).nonEmpty &&
!ExprUtils.canEvaluateUnconditionally(c)
+ }
+ if (keepsFailable && reordered != conjuncts) {
+ reordered.reduce(And)
+ } else {
+ condition
}
}
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
index 106af71a9d65..03fbdfed1e9a 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
@@ -22,8 +22,10 @@ import org.apache.spark.sql.catalyst.dsl.expressions._
import org.apache.spark.sql.catalyst.dsl.plans._
import org.apache.spark.sql.catalyst.expressions._
import org.apache.spark.sql.catalyst.plans.PlanTest
-import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LocalRelation,
LogicalPlan}
import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.DoubleType
/**
* Unit tests for constant propagation in expressions.
@@ -214,4 +216,43 @@ class ConstantPropagationSuite extends PlanTest {
Optimize.execute(testRelation.where(Not(columnA === 1 && columnA ===
columnA + 2)).analyze),
testRelation.where(Not(columnA === 1) || Not(columnA === columnA +
2)).analyze)
}
+
+ test("SPARK-60983: Do not propagate a constant into an expression that can
fail") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ def divide(e: Expression): Expression = Divide(Literal(10.0),
e.cast(DoubleType))
+
+ // The conjunct can also run on rows where `b` is not 0. With `b`
replaced, it would divide by
+ // zero on such a row with `c` = 0, where `c - b` is not 0.
+ val notConstant =
+ testRelation.where(divide(columnC - columnB) > Literal(0.0) && columnB
=== 0).analyze
+ comparePlans(Optimize.execute(notConstant), notConstant)
+
+ // It can become constant, but keeps the attribute if it fails: no row
may reach it.
+ val failsToFold = testRelation.where(columnB === 0 && columnA >
divide(columnB)).analyze
+ comparePlans(Optimize.execute(failsToFold), failsToFold)
+
+ comparePlans(
+ Optimize.execute(testRelation.where(columnB === 2 && columnA >
divide(columnB)).analyze),
+ testRelation.where(columnB === 2 && columnA > Literal(5.0)).analyze)
+
+ // An expression that fails on its own does not stop the other side of
the comparison.
+ comparePlans(
+ Optimize.execute(testRelation.where(columnB === 2 && divide(columnC) >
columnB).analyze),
+ testRelation.where(columnB === 2 && divide(columnC) >
Literal(2.0)).analyze)
+
+ // The guard `b <> 0` becomes TRUE, so the equality goes ahead of the
conjunct that keeps `b`.
+ val guarded = Optimize.execute(testRelation
+ .where(columnB =!= 0 && divide(columnC - columnB) > Literal(0.0) &&
columnB === 5).analyze)
+ comparePlans(guarded,
+ testRelation.where(columnB === 5 && divide(columnC - columnB) >
Literal(0.0)).analyze)
+ val Filter(guardedCondition, _) = guarded
+ assert(splitConjunctivePredicates(guardedCondition).head.sql == "(b =
5)")
+
+ // The condition of an IF decides whether the branch that can fail runs.
+ val conditional = testRelation
+ .where(If(columnC > columnB, Literal(true), divide(columnC) >
Literal(0.0)) &&
+ columnB === 0).analyze
+ comparePlans(Optimize.execute(conditional), conditional)
+ }
+ }
}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
index 5e58dce3c647..6dad2a3bf214 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
@@ -3389,4 +3389,16 @@ class SubquerySuite extends SharedSparkSession
}
}
}
+
+ test("SPARK-60983: Propagate a constant into a comparison with a correlated
scalar subquery") {
+ withTempView("t1", "t2") {
+ Seq((1, 2), (2, 3)).toDF("a", "b").createOrReplaceTempView("t1")
+ Seq((1, 2), (1, 1), (2, 3)).toDF("a", "b").createOrReplaceTempView("t2")
+ // The constant of `t1.a` is not looked up for `t2.a` in the condition
of the subquery.
+ checkAnswer(
+ sql("SELECT * FROM t1 WHERE t1.a = 1 AND " +
+ "t1.b = (SELECT max(t2.b) FROM t2 WHERE t2.a = t1.a)"),
+ Row(1, 2))
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]