This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 4e1c11fa2025 [SPARK-61033][SQL] Fix query errors by
ConstantPropagation when a rewritten operand of AND, OR, COALESCE or IN guards
an expression that can fail
4e1c11fa2025 is described below
commit 4e1c11fa202522c1ed4c9dd3d4fdd6b64638356a
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 22:46:40 2026 -0800
[SPARK-61033][SQL] Fix query errors by ConstantPropagation when a rewritten
operand of AND, OR, COALESCE or IN guards an expression that can fail
### What changes were proposed in this pull request?
`ConstantPropagation` replaces an attribute with the literal of an equality
conjunct, such as `y = 5`, and rewrites each operand of `AND`, `OR`, `COALESCE`
and `IN` on its own. These operators short-circuit, so an operand decides
whether the operands after it run. The literal is the value of the attribute
only on the rows that pass the whole condition. On the other rows, a rewritten
operand can take another value and run an operand after it that can fail:
- `(y < 3 OR 10 / x > 1) AND y = 5`: `y < 3` becomes `5 < 3` and is folded
away, so `10 / x > 1` runs on a row with `y = 1` and `x = 0`.
- `coalesce(m[k], 10 / x) > 0 AND k = 'a'`: `m[k]` becomes `m['a']`, which
is NULL on a row with `k = 'b'` and `m = map('b', 1.0)`.
- `y > 0 AND 10 / x > 0 AND y = 1`: the conjunct `y > 0` becomes TRUE and
is removed, so `10 / x > 0` runs on a row with `y = -1` and `x = 0`.
When the rewrite removes the last reference to the propagated column,
predicate pushdown can also move the conjunct below an operator. In `HAVING s =
5 AND (s < 3 OR 10 / g > 1)`, `10 / g > 1` runs below the `Aggregate` on every
input row.
This PR changes `ConstantPropagation` as follows:
- An operand of `AND`, `OR`, `COALESCE` or `IN` in a conjunct is rewritten
only if none of the operands after it, as rewritten, can fail
(`ExprUtils.canEvaluateUnconditionally`). The conjunct then keeps the
propagated attribute, so it is not pushed below the operator that produces the
attribute. Since SPARK-60983 (#60163), the equalities move ahead of such a
conjunct. For example, `(y > 3 OR 10 / x > 1) AND y = 5` keeps its `OR`, which
used to become TRUE, and runs it after `y = 5`.
- The conjuncts of the condition are still rewritten, but the equalities
also move ahead of the other conjuncts when a conjunct that can fail comes
after a rewritten conjunct. Then such a conjunct only runs on rows where the
equalities hold. An equality between an attribute and a literal cannot fail, so
running it first raises no error.
### Why are the changes needed?
The queries below return their results when they are evaluated as written,
but fail with `DIVIDE_BY_ZERO` under ANSI mode.
```sql
CREATE TABLE t1(y BIGINT, x BIGINT) USING parquet;
INSERT INTO t1 VALUES (1, 0), (5, 20);
-- It should return no rows, but fails with DIVIDE_BY_ZERO.
SELECT * FROM t1 WHERE (y < 3 OR 10 / x > 1) AND y = 5;
CREATE TABLE t2(y BIGINT, x BIGINT) USING parquet;
INSERT INTO t2 VALUES (-1, 0), (1, 5);
-- It should return (1, 5), but fails with DIVIDE_BY_ZERO.
SELECT * FROM t2 WHERE y > 0 AND 10 / x > 0 AND y = 1;
CREATE TABLE t3(g BIGINT, x BIGINT) USING parquet;
INSERT INTO t3 VALUES (0, 1);
-- It should return no rows, but fails with DIVIDE_BY_ZERO.
SELECT g, sum(x) s FROM t3 GROUP BY g HAVING s = 5 AND (s < 3 OR 10 / g >
1);
```
`ConstantPropagation` has rewritten the operands of `OR` since SPARK-20758
(#17993, [v2.3.0](https://github.com/apache/spark/tree/v2.3.0) (2018-02-22)),
first only in `=` and `<=>`. SPARK-42500 (#42038) extended it to all
comparisons, and SPARK-44444 (#46013) turned on ANSI mode by default. Both are
in [v4.0.0](https://github.com/apache/spark/tree/v4.0.0) (2025-05-19).
### Does this PR introduce _any_ user-facing change?
Yes. The queries above, which fail with an error, now return their results.
Some plans keep an operand that was rewritten before. For example, `y = 5
AND (y > 3 OR 10 / x > 1)` keeps the `OR` with `y = 5` ahead of it, instead of
becoming `y = 5`, and `y = 5 AND coalesce(y, 10 / x) > 3` keeps the `COALESCE`.
The results are the same. When a conjunct that can fail comes after a rewritten
conjunct, the equalities move ahead of the other conjuncts in the plan.
### How was this patch tested?
- Added `SPARK-61033: Do not propagate a constant into an operand that
guards an expression that can fail` to `ConstantPropagationSuite`. It covers an
`OR` with a nested `AND` in a condition, a `COALESCE` in a comparison, and a
conjunct before a conjunct that can fail, and checks that the equality goes
first. Each part fails if its change is reverted.
- Ran all suites in `org.apache.spark.sql.catalyst.optimizer`,
`SubquerySuite`, and the TPC-DS and TPC-H `PlanStabilitySuite`s with ANSI mode
on and off. All pass, and no golden files change.
- Ran the queries above and the variants with a nested `AND`, an `IN` in a
comparison and a `COALESCE` over a map lookup. They no longer fail, and the
queries of #60163 return the same results.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60190 from dongjoon-hyun/SPARK-61033-master.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 3c58e92ff6f1a9c11ce92fe0ddc12363200d32e6)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/catalyst/optimizer/expressions.scala | 51 +++++++++++++++++-----
.../optimizer/ConstantPropagationSuite.scala | 33 +++++++++++++-
2 files changed, 72 insertions(+), 12 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 9bcc5cde01d4..6da3b7c69ac9 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -195,10 +195,9 @@ object ConstantPropagation extends Rule[LogicalPlan] with
PredicateHelper {
traverse(a.right, replaceChildren = false, nullIsFalse)
val equalityPredicates = equalityPredicatesLeft ++
equalityPredicatesRight
val newSelf = if (equalityPredicates.nonEmpty && replaceChildren) {
+ val unreplaced = And(newLeft.getOrElse(a.left),
newRight.getOrElse(a.right))
Some(moveEqualitiesFirst(
- And(replaceConstants(newLeft.getOrElse(a.left),
equalityPredicates),
- replaceConstants(newRight.getOrElse(a.right),
equalityPredicates)),
- equalityPredicates))
+ unreplaced, replaceConstants(unreplaced, equalityPredicates),
equalityPredicates))
} else {
if (newLeft.isDefined || newRight.isDefined) {
Some(And(newLeft.getOrElse(a.left), newRight.getOrElse(a.right)))
@@ -240,7 +239,10 @@ object ConstantPropagation extends Rule[LogicalPlan] with
PredicateHelper {
* `10 / (id - y) > 0 AND y = 0`, and a part of it that becomes constant is
folded with no row
* at all, e.g. `10 / y` for `y = 0 AND id > 10 / y`. So that this raises no
error the original
* condition cannot raise, an expression that can fail keeps its attributes,
unless all of them
- * are replaced and the result evaluates without an error.
+ * are replaced and the result evaluates without an error. For the same
reason, an operand of an
+ * AND, OR, COALESCE or IN in a conjunct is rewritten only if no operand
after it can fail, as it
+ * decides whether they run, e.g. `y < 3` in `(y < 3 OR 10 / x > 1) AND y =
5`. The conjuncts of
+ * `condition` itself are rewritten regardless, and [[moveEqualitiesFirst]]
orders them.
*/
private def replaceConstants(
condition: Expression,
@@ -253,6 +255,7 @@ object ConstantPropagation extends Rule[LogicalPlan] with
PredicateHelper {
constantsMap.contains(a) && (isBinaryStable(a.dataType) ||
collationSafeReplacement)
def replace(e: Expression): Expression = e match {
case a: AttributeReference if canReplace(a) => constantsMap(a)
+ case _: And | _: Or | _: Coalesce | _: In => replaceOperands(e,
replace)
case _ if ExprUtils.isTotalOperator(e) => e.mapChildren(replace)
case _ if e.references.nonEmpty && e.references.forall(canReplace) =>
fold(e.transform { case a: AttributeReference if canReplace(a) =>
constantsMap(a) })
@@ -264,12 +267,31 @@ object ConstantPropagation extends Rule[LogicalPlan] with
PredicateHelper {
def replaceInCondition(e: Expression): Expression = e match {
case b: BinaryComparison if predicates.contains(b) => b
case b: BinaryComparison => replaceInComparison(b)
+ case _: And | _: Or | _: Coalesce | _: In => replaceOperands(e,
replaceInCondition)
case _ if ExprUtils.isTotalOperator(e) =>
e.mapChildren(replaceInCondition)
// The children of these include the comparisons of a pulled up
correlated condition.
case _: InSubquery | _: ListQuery | _: Exists =>
e.mapChildren(replaceInCondition)
case _ => e
}
- replaceInCondition(condition)
+ def replaceInConjuncts(e: Expression): Expression = e match {
+ case a: And => a.mapChildren(replaceInConjuncts)
+ case _ => replaceInCondition(e)
+ }
+ replaceInConjuncts(condition)
+ }
+
+ /**
+ * Rewrites the operands of `e`, which short-circuits, with `f`. An operand
decides whether the
+ * ones after it run, so it is rewritten only if none of those, as
rewritten, can fail.
+ */
+ private def replaceOperands(e: Expression, f: Expression => Expression):
Expression = {
+ val (newChildren, _) = e.children.foldRight((List.empty[Expression],
false)) {
+ case (child, (laterChildren, laterCanFail)) =>
+ val newChild = if (laterCanFail) child else f(child)
+ (newChild :: laterChildren,
+ laterCanFail || !ExprUtils.canEvaluateUnconditionally(newChild))
+ }
+ e.withNewChildren(newChildren)
}
/** Returns the literal that `e` folds to, or None if it is not constant or
fails to fold. */
@@ -286,13 +308,16 @@ object ConstantPropagation extends Rule[LogicalPlan] with
PredicateHelper {
}
/**
- * Moves the equalities ahead of the other conjuncts of `condition` if one
that can fail still
- * refers to a propagated attribute. Such a conjunct also runs on rows where
an equality does
- * not hold, with the value of the row, while a guard over the same
attribute may be folded
- * away, e.g. `n / d > 1` for `d <> 0 AND n / d > 1 AND d = 5`. An equality
between an
- * attribute and a literal cannot fail, so running it first raises no error.
+ * Moves the equalities ahead of the other conjuncts of `condition`, the
rewritten `original`, if
+ * one that can fail still refers to a propagated attribute. Such a conjunct
also runs on rows
+ * where an equality does not hold, with the value of the row, while a guard
over the same
+ * attribute may be folded away, e.g. `n / d > 1` for `d <> 0 AND n / d > 1
AND d = 5`. The
+ * same holds for a conjunct that can fail after one that is rewritten, as
that one decides
+ * whether it runs, e.g. `10 / x > 0` for `y > 0 AND 10 / x > 0 AND y = 1`.
An equality between
+ * an attribute and a literal cannot fail, so running it first raises no
error.
*/
private def moveEqualitiesFirst(
+ original: Expression,
condition: Expression,
equalityPredicates: AttributeMap[(Literal, BinaryComparison)]):
Expression = {
val predicates = equalityPredicates.values.map(_._2).toSet[Expression]
@@ -303,7 +328,11 @@ object ConstantPropagation extends Rule[LogicalPlan] with
PredicateHelper {
val keepsFailable = others.exists { c =>
c.references.intersect(propagated).nonEmpty &&
!ExprUtils.canEvaluateUnconditionally(c)
}
- if (keepsFailable && reordered != conjuncts) {
+ val firstRewritten = splitConjunctivePredicates(original).zip(conjuncts)
+ .indexWhere { case (before, after) => !before.fastEquals(after) }
+ val unmasksFailable = firstRewritten >= 0 &&
+ conjuncts.drop(firstRewritten +
1).exists(!ExprUtils.canEvaluateUnconditionally(_))
+ if ((keepsFailable || unmasksFailable) && reordered != conjuncts) {
reordered.reduce(And)
} else {
condition
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
index 03fbdfed1e9a..1dd1641fec47 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
@@ -25,7 +25,7 @@ import org.apache.spark.sql.catalyst.plans.PlanTest
import org.apache.spark.sql.catalyst.plans.logical.{Filter, LocalRelation,
LogicalPlan}
import org.apache.spark.sql.catalyst.rules.RuleExecutor
import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.DoubleType
+import org.apache.spark.sql.types.{DoubleType, StringType}
/**
* Unit tests for constant propagation in expressions.
@@ -255,4 +255,35 @@ class ConstantPropagationSuite extends PlanTest {
comparePlans(Optimize.execute(conditional), conditional)
}
}
+
+ test("SPARK-61033: Do not propagate a constant into an operand that guards
an expression " +
+ "that can fail") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ def divide(e: Expression): Expression = Divide(Literal(10.0),
e.cast(DoubleType))
+ def first(plan: LogicalPlan): String = plan match {
+ case Filter(condition, _) =>
splitConjunctivePredicates(condition).head.sql
+ }
+
+ // `a < 3` and `a > 3` decide whether the division runs. With `a`
replaced, it would run on a
+ // row with `a` = 1 and `c` = 0. The conjunct keeps `a`, so the equality
goes first.
+ val nested = testRelation.where(
+ (columnA < 3 || (columnA > 3 && divide(columnC) > Literal(1.0))) &&
columnA === 5).analyze
+ val optimizedNested = Optimize.execute(nested)
+ comparePlans(optimizedNested, nested)
+ assert(first(optimizedNested) == "(a = 5)")
+
+ // With `k` replaced, the map lookup can be NULL on a row with another
`k`.
+ val maps = LocalRelation($"k".string, Symbol("m").map(StringType,
DoubleType), $"x".int)
+ val inComparison = maps.where(Coalesce(Seq(GetMapValue($"m", $"k"),
divide($"x"))) >
+ Literal(0.0) && $"k" === "a").analyze
+ comparePlans(Optimize.execute(inComparison), inComparison)
+
+ // A conjunct guards the ones after it as well. It is rewritten, and the
equality goes first.
+ val conjunct = Optimize.execute(testRelation
+ .where(columnA > 0 && divide(columnC) > Literal(0.0) && columnA ===
1).analyze)
+ comparePlans(conjunct,
+ testRelation.where(columnA === 1 && divide(columnC) >
Literal(0.0)).analyze)
+ assert(first(conjunct) == "(a = 1)")
+ }
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]