This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 3c58e92ff6f1 [SPARK-61033][SQL] Fix query errors by 
ConstantPropagation when a rewritten operand of AND, OR, COALESCE or IN guards 
an expression that can fail
3c58e92ff6f1 is described below

commit 3c58e92ff6f1a9c11ce92fe0ddc12363200d32e6
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 22:46:40 2026 -0800

    [SPARK-61033][SQL] Fix query errors by ConstantPropagation when a rewritten 
operand of AND, OR, COALESCE or IN guards an expression that can fail
    
    ### What changes were proposed in this pull request?
    
    `ConstantPropagation` replaces an attribute with the literal of an equality 
conjunct, such as `y = 5`, and rewrites each operand of `AND`, `OR`, `COALESCE` 
and `IN` on its own. These operators short-circuit, so an operand decides 
whether the operands after it run. The literal is the value of the attribute 
only on the rows that pass the whole condition. On the other rows, a rewritten 
operand can take another value and run an operand after it that can fail:
    
    - `(y < 3 OR 10 / x > 1) AND y = 5`: `y < 3` becomes `5 < 3` and is folded 
away, so `10 / x > 1` runs on a row with `y = 1` and `x = 0`.
    - `coalesce(m[k], 10 / x) > 0 AND k = 'a'`: `m[k]` becomes `m['a']`, which 
is NULL on a row with `k = 'b'` and `m = map('b', 1.0)`.
    - `y > 0 AND 10 / x > 0 AND y = 1`: the conjunct `y > 0` becomes TRUE and 
is removed, so `10 / x > 0` runs on a row with `y = -1` and `x = 0`.
    
    When the rewrite removes the last reference to the propagated column, 
predicate pushdown can also move the conjunct below an operator. In `HAVING s = 
5 AND (s < 3 OR 10 / g > 1)`, `10 / g > 1` runs below the `Aggregate` on every 
input row.
    
    This PR changes `ConstantPropagation` as follows:
    
    - An operand of `AND`, `OR`, `COALESCE` or `IN` in a conjunct is rewritten 
only if none of the operands after it, as rewritten, can fail 
(`ExprUtils.canEvaluateUnconditionally`). The conjunct then keeps the 
propagated attribute, so it is not pushed below the operator that produces the 
attribute. Since SPARK-60983 (#60163), the equalities move ahead of such a 
conjunct. For example, `(y > 3 OR 10 / x > 1) AND y = 5` keeps its `OR`, which 
used to become TRUE, and runs it after `y = 5`.
    - The conjuncts of the condition are still rewritten, but the equalities 
also move ahead of the other conjuncts when a conjunct that can fail comes 
after a rewritten conjunct. Then such a conjunct only runs on rows where the 
equalities hold. An equality between an attribute and a literal cannot fail, so 
running it first raises no error.
    
    ### Why are the changes needed?
    
    The queries below return their results when they are evaluated as written, 
but fail with `DIVIDE_BY_ZERO` under ANSI mode.
    
    ```sql
    CREATE TABLE t1(y BIGINT, x BIGINT) USING parquet;
    INSERT INTO t1 VALUES (1, 0), (5, 20);
    -- It should return no rows, but fails with DIVIDE_BY_ZERO.
    SELECT * FROM t1 WHERE (y < 3 OR 10 / x > 1) AND y = 5;
    
    CREATE TABLE t2(y BIGINT, x BIGINT) USING parquet;
    INSERT INTO t2 VALUES (-1, 0), (1, 5);
    -- It should return (1, 5), but fails with DIVIDE_BY_ZERO.
    SELECT * FROM t2 WHERE y > 0 AND 10 / x > 0 AND y = 1;
    
    CREATE TABLE t3(g BIGINT, x BIGINT) USING parquet;
    INSERT INTO t3 VALUES (0, 1);
    -- It should return no rows, but fails with DIVIDE_BY_ZERO.
    SELECT g, sum(x) s FROM t3 GROUP BY g HAVING s = 5 AND (s < 3 OR 10 / g > 
1);
    ```
    
    `ConstantPropagation` has rewritten the operands of `OR` since SPARK-20758 
(#17993, [v2.3.0](https://github.com/apache/spark/tree/v2.3.0) (2018-02-22)), 
first only in `=` and `<=>`. SPARK-42500 (#42038) extended it to all 
comparisons, and SPARK-44444 (#46013) turned on ANSI mode by default. Both are 
in [v4.0.0](https://github.com/apache/spark/tree/v4.0.0) (2025-05-19).
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The queries above, which fail with an error, now return their results.
    
    Some plans keep an operand that was rewritten before. For example, `y = 5 
AND (y > 3 OR 10 / x > 1)` keeps the `OR` with `y = 5` ahead of it, instead of 
becoming `y = 5`, and `y = 5 AND coalesce(y, 10 / x) > 3` keeps the `COALESCE`. 
The results are the same. When a conjunct that can fail comes after a rewritten 
conjunct, the equalities move ahead of the other conjuncts in the plan.
    
    ### How was this patch tested?
    
    - Added `SPARK-61033: Do not propagate a constant into an operand that 
guards an expression that can fail` to `ConstantPropagationSuite`. It covers an 
`OR` with a nested `AND` in a condition, a `COALESCE` in a comparison, and a 
conjunct before a conjunct that can fail, and checks that the equality goes 
first. Each part fails if its change is reverted.
    - Ran all suites in `org.apache.spark.sql.catalyst.optimizer`, 
`SubquerySuite`, and the TPC-DS and TPC-H `PlanStabilitySuite`s with ANSI mode 
on and off. All pass, and no golden files change.
    - Ran the queries above and the variants with a nested `AND`, an `IN` in a 
comparison and a `COALESCE` over a map lookup. They no longer fail, and the 
queries of #60163 return the same results.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60190 from dongjoon-hyun/SPARK-61033-master.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/catalyst/optimizer/expressions.scala | 51 +++++++++++++++++-----
 .../optimizer/ConstantPropagationSuite.scala       | 33 +++++++++++++-
 2 files changed, 72 insertions(+), 12 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 9bcc5cde01d4..6da3b7c69ac9 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -195,10 +195,9 @@ object ConstantPropagation extends Rule[LogicalPlan] with 
PredicateHelper {
           traverse(a.right, replaceChildren = false, nullIsFalse)
         val equalityPredicates = equalityPredicatesLeft ++ 
equalityPredicatesRight
         val newSelf = if (equalityPredicates.nonEmpty && replaceChildren) {
+          val unreplaced = And(newLeft.getOrElse(a.left), 
newRight.getOrElse(a.right))
           Some(moveEqualitiesFirst(
-            And(replaceConstants(newLeft.getOrElse(a.left), 
equalityPredicates),
-              replaceConstants(newRight.getOrElse(a.right), 
equalityPredicates)),
-            equalityPredicates))
+            unreplaced, replaceConstants(unreplaced, equalityPredicates), 
equalityPredicates))
         } else {
           if (newLeft.isDefined || newRight.isDefined) {
             Some(And(newLeft.getOrElse(a.left), newRight.getOrElse(a.right)))
@@ -240,7 +239,10 @@ object ConstantPropagation extends Rule[LogicalPlan] with 
PredicateHelper {
    * `10 / (id - y) > 0 AND y = 0`, and a part of it that becomes constant is 
folded with no row
    * at all, e.g. `10 / y` for `y = 0 AND id > 10 / y`. So that this raises no 
error the original
    * condition cannot raise, an expression that can fail keeps its attributes, 
unless all of them
-   * are replaced and the result evaluates without an error.
+   * are replaced and the result evaluates without an error. For the same 
reason, an operand of an
+   * AND, OR, COALESCE or IN in a conjunct is rewritten only if no operand 
after it can fail, as it
+   * decides whether they run, e.g. `y < 3` in `(y < 3 OR 10 / x > 1) AND y = 
5`. The conjuncts of
+   * `condition` itself are rewritten regardless, and [[moveEqualitiesFirst]] 
orders them.
    */
   private def replaceConstants(
       condition: Expression,
@@ -253,6 +255,7 @@ object ConstantPropagation extends Rule[LogicalPlan] with 
PredicateHelper {
         constantsMap.contains(a) && (isBinaryStable(a.dataType) || 
collationSafeReplacement)
       def replace(e: Expression): Expression = e match {
         case a: AttributeReference if canReplace(a) => constantsMap(a)
+        case _: And | _: Or | _: Coalesce | _: In => replaceOperands(e, 
replace)
         case _ if ExprUtils.isTotalOperator(e) => e.mapChildren(replace)
         case _ if e.references.nonEmpty && e.references.forall(canReplace) =>
           fold(e.transform { case a: AttributeReference if canReplace(a) => 
constantsMap(a) })
@@ -264,12 +267,31 @@ object ConstantPropagation extends Rule[LogicalPlan] with 
PredicateHelper {
     def replaceInCondition(e: Expression): Expression = e match {
       case b: BinaryComparison if predicates.contains(b) => b
       case b: BinaryComparison => replaceInComparison(b)
+      case _: And | _: Or | _: Coalesce | _: In => replaceOperands(e, 
replaceInCondition)
       case _ if ExprUtils.isTotalOperator(e) => 
e.mapChildren(replaceInCondition)
       // The children of these include the comparisons of a pulled up 
correlated condition.
       case _: InSubquery | _: ListQuery | _: Exists => 
e.mapChildren(replaceInCondition)
       case _ => e
     }
-    replaceInCondition(condition)
+    def replaceInConjuncts(e: Expression): Expression = e match {
+      case a: And => a.mapChildren(replaceInConjuncts)
+      case _ => replaceInCondition(e)
+    }
+    replaceInConjuncts(condition)
+  }
+
+  /**
+   * Rewrites the operands of `e`, which short-circuits, with `f`. An operand 
decides whether the
+   * ones after it run, so it is rewritten only if none of those, as 
rewritten, can fail.
+   */
+  private def replaceOperands(e: Expression, f: Expression => Expression): 
Expression = {
+    val (newChildren, _) = e.children.foldRight((List.empty[Expression], 
false)) {
+      case (child, (laterChildren, laterCanFail)) =>
+        val newChild = if (laterCanFail) child else f(child)
+        (newChild :: laterChildren,
+          laterCanFail || !ExprUtils.canEvaluateUnconditionally(newChild))
+    }
+    e.withNewChildren(newChildren)
   }
 
   /** Returns the literal that `e` folds to, or None if it is not constant or 
fails to fold. */
@@ -286,13 +308,16 @@ object ConstantPropagation extends Rule[LogicalPlan] with 
PredicateHelper {
   }
 
   /**
-   * Moves the equalities ahead of the other conjuncts of `condition` if one 
that can fail still
-   * refers to a propagated attribute. Such a conjunct also runs on rows where 
an equality does
-   * not hold, with the value of the row, while a guard over the same 
attribute may be folded
-   * away, e.g. `n / d > 1` for `d <> 0 AND n / d > 1 AND d = 5`. An equality 
between an
-   * attribute and a literal cannot fail, so running it first raises no error.
+   * Moves the equalities ahead of the other conjuncts of `condition`, the 
rewritten `original`, if
+   * one that can fail still refers to a propagated attribute. Such a conjunct 
also runs on rows
+   * where an equality does not hold, with the value of the row, while a guard 
over the same
+   * attribute may be folded away, e.g. `n / d > 1` for `d <> 0 AND n / d > 1 
AND d = 5`. The
+   * same holds for a conjunct that can fail after one that is rewritten, as 
that one decides
+   * whether it runs, e.g. `10 / x > 0` for `y > 0 AND 10 / x > 0 AND y = 1`. 
An equality between
+   * an attribute and a literal cannot fail, so running it first raises no 
error.
    */
   private def moveEqualitiesFirst(
+      original: Expression,
       condition: Expression,
       equalityPredicates: AttributeMap[(Literal, BinaryComparison)]): 
Expression = {
     val predicates = equalityPredicates.values.map(_._2).toSet[Expression]
@@ -303,7 +328,11 @@ object ConstantPropagation extends Rule[LogicalPlan] with 
PredicateHelper {
     val keepsFailable = others.exists { c =>
       c.references.intersect(propagated).nonEmpty && 
!ExprUtils.canEvaluateUnconditionally(c)
     }
-    if (keepsFailable && reordered != conjuncts) {
+    val firstRewritten = splitConjunctivePredicates(original).zip(conjuncts)
+      .indexWhere { case (before, after) => !before.fastEquals(after) }
+    val unmasksFailable = firstRewritten >= 0 &&
+      conjuncts.drop(firstRewritten + 
1).exists(!ExprUtils.canEvaluateUnconditionally(_))
+    if ((keepsFailable || unmasksFailable) && reordered != conjuncts) {
       reordered.reduce(And)
     } else {
       condition
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
index 03fbdfed1e9a..1dd1641fec47 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
@@ -25,7 +25,7 @@ import org.apache.spark.sql.catalyst.plans.PlanTest
 import org.apache.spark.sql.catalyst.plans.logical.{Filter, LocalRelation, 
LogicalPlan}
 import org.apache.spark.sql.catalyst.rules.RuleExecutor
 import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.DoubleType
+import org.apache.spark.sql.types.{DoubleType, StringType}
 
 /**
  * Unit tests for constant propagation in expressions.
@@ -255,4 +255,35 @@ class ConstantPropagationSuite extends PlanTest {
       comparePlans(Optimize.execute(conditional), conditional)
     }
   }
+
+  test("SPARK-61033: Do not propagate a constant into an operand that guards 
an expression " +
+      "that can fail") {
+    withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+      def divide(e: Expression): Expression = Divide(Literal(10.0), 
e.cast(DoubleType))
+      def first(plan: LogicalPlan): String = plan match {
+        case Filter(condition, _) => 
splitConjunctivePredicates(condition).head.sql
+      }
+
+      // `a < 3` and `a > 3` decide whether the division runs. With `a` 
replaced, it would run on a
+      // row with `a` = 1 and `c` = 0. The conjunct keeps `a`, so the equality 
goes first.
+      val nested = testRelation.where(
+        (columnA < 3 || (columnA > 3 && divide(columnC) > Literal(1.0))) && 
columnA === 5).analyze
+      val optimizedNested = Optimize.execute(nested)
+      comparePlans(optimizedNested, nested)
+      assert(first(optimizedNested) == "(a = 5)")
+
+      // With `k` replaced, the map lookup can be NULL on a row with another 
`k`.
+      val maps = LocalRelation($"k".string, Symbol("m").map(StringType, 
DoubleType), $"x".int)
+      val inComparison = maps.where(Coalesce(Seq(GetMapValue($"m", $"k"), 
divide($"x"))) >
+        Literal(0.0) && $"k" === "a").analyze
+      comparePlans(Optimize.execute(inComparison), inComparison)
+
+      // A conjunct guards the ones after it as well. It is rewritten, and the 
equality goes first.
+      val conjunct = Optimize.execute(testRelation
+        .where(columnA > 0 && divide(columnC) > Literal(0.0) && columnA === 
1).analyze)
+      comparePlans(conjunct,
+        testRelation.where(columnA === 1 && divide(columnC) > 
Literal(0.0)).analyze)
+      assert(first(conjunct) == "(a = 1)")
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to