This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new b076d30ebbfd [SPARK-60983][SQL] Fix query errors by 
ConstantPropagation when a constant is substituted into an expression that can 
fail
b076d30ebbfd is described below

commit b076d30ebbfd85ae340c5976a5594f97e5914168
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Fri Nov 6 18:37:50 2026 -0800

    [SPARK-60983][SQL] Fix query errors by ConstantPropagation when a constant 
is substituted into an expression that can fail
    
    ### What changes were proposed in this pull request?
    
    `ConstantPropagation` replaces an attribute with the literal of an equality 
conjunct, such as `y = 0`, in the other comparisons of the same `AND`. The 
literal is the value of the attribute only on the rows that pass the whole 
`AND`. However, the rewritten conjuncts also run on other rows, and a part that 
becomes constant is folded by `ConstantFolding` before any row is read. So an 
expression that can fail, such as a division, an ANSI cast or an array index, 
can raise an error that the [...]
    
    - `10 / (id - y) > 0 AND y = 0` becomes `10 / (id - 0) > 0 AND y = 0`, 
which divides by zero on a row with `id = 0` and `y = 1`.
    - `y = 0 AND id > 10 / y` becomes `y = 0 AND id > 10 / 0`, which fails 
during the optimization, even on an empty table.
    
    This PR replaces an attribute only where its value cannot change whether an 
expression fails:
    
    - under operators that cannot fail on their own, such as comparisons, 
`AND`, `OR`, `NOT`, `IS NULL` and `COALESCE`. This is the allowlist of 
`ExprUtils.canEvaluateUnconditionally`, which is now also exposed as 
`ExprUtils.isTotalOperator`.
    - in a subexpression whose attributes are all replaced, when it folds to a 
literal without an error. For example, `y = 5 AND x = y + 1` still becomes `y = 
5 AND x = 6`.
    
    Otherwise, the expression keeps its attributes. The comparisons in the 
pulled up condition of a correlated `IN` or `EXISTS` subquery are rewritten 
within the same boundary.
    
    A conjunct that keeps an attribute can still run on rows where the equality 
does not hold, while a guard over the same attribute may become a constant, 
such as `d <> 0` in `d <> 0 AND n / d > 1 AND d = 5`. So when a conjunct that 
can fail keeps a propagated attribute, this PR also moves the equalities ahead 
of the other conjuncts. An equality between an attribute and a literal cannot 
fail, so running it first raises no error.
    
    ### Why are the changes needed?
    
    The queries below cannot raise an error in any order of their conjuncts, 
but fail with `DIVIDE_BY_ZERO`.
    
    ```sql
    CREATE TABLE t1(id BIGINT, y BIGINT) USING parquet;
    INSERT INTO t1 VALUES (0, 1), (6, 0);
    -- It should return (6, 0), but fails with DIVIDE_BY_ZERO.
    SELECT * FROM t1 WHERE 10 / (id - y) > 0 AND y = 0;
    
    CREATE TABLE t2(id BIGINT, y BIGINT) USING parquet;
    -- It should return no rows, but fails with DIVIDE_BY_ZERO, with or without 
rows in `t2`.
    SELECT * FROM t2 WHERE y = 0 AND id > 10 / y;
    ```
    
    The same happens with other expressions that can fail. For example, `s = 
'abc' AND CAST(s AS INT) > 0` fails with `CAST_INVALID_INPUT` on a table 
without `'abc'`, and `arr[i] > 0 AND i = 5` fails with `INVALID_ARRAY_INDEX` on 
a row with a shorter array and `i = 1`. A UDF or `assert_true` also fails like 
this with ANSI mode off.
    
    `ConstantPropagation` has substituted into all comparisons since 
SPARK-42500 (#42038), and ANSI mode has been on by default since SPARK-44444 
(#46013). Both are in [v4.0.0](https://github.com/apache/spark/tree/v4.0.0) 
(2025-05-19). Before that, it only substituted into `=` and `<=>` (SPARK-20758, 
#17993, [v2.3.0](https://github.com/apache/spark/tree/v2.3.0) (2018-02-22)).
    
    This PR should be merged after #60162 (SPARK-60985). Without it, 
`InferFiltersFromConstraints` infers the expression that this PR no longer 
rewrites, such as `10 / (id - 0) > 0` from `10 / (id - y) > 0 AND y = 0`, and 
puts it ahead of the original conditions. Then a query that works today, such 
as `SELECT * FROM t WHERE id > 5 AND y = 0 AND 10 / (id - y) > 0` on rows `(0, 
0)` and `(6, 0)`, would fail with `DIVIDE_BY_ZERO`.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. The queries above, which fail with an error, now return their results.
    
    Some comparisons keep an attribute instead of a literal, for example `y = 5 
AND x / y > 1`, `y = 5 AND CASE WHEN y > 0 THEN x ELSE z END > 1`, and `r.d + 
l.b > 10` in a correlated subquery with `l.b = 5.0` outside. The results are 
the same, but some plans become less efficient. For example, the CASE above no 
longer pushes `x > 1` down to the scan, and `l.b = 5.0 AND l.a NOT IN (SELECT c 
FROM r WHERE r.d + l.b > 10)` is planned as a broadcast nested loop join 
instead of a null-aware br [...]
    
    ### How was this patch tested?
    
    - Added `SPARK-60983: Do not propagate a constant into an expression that 
can fail` to `ConstantPropagationSuite`. It covers an expression that can fail 
and does not become constant, one that becomes constant but fails to fold, one 
that folds, a comparison whose other side can fail, a guard that becomes a 
constant, after which the equality goes first, and the condition of an `IF`, 
which is not rewritten. Without this change, the test fails.
    - Added `SPARK-60983: Propagate a constant into a comparison with a 
correlated scalar subquery` to `SubquerySuite`. It fails with 
`NoSuchElementException` if the columns in the condition of the subquery are 
looked up as constants.
    - The existing `SPARK-32290: SingleColumn Null Aware Anti Join Optimize` in 
`SubquerySuite` covers the comparisons in the condition of a correlated 
subquery. It fails if they are not rewritten.
    - Ran all suites in `org.apache.spark.sql.catalyst.optimizer`, 
`ExprUtilsSuite`, the TPC-DS and TPC-H `PlanStabilitySuite`s with ANSI mode on 
and off, `SQLQueryTestSuite`, `SQLQuerySuite`, `DataFrameSuite`, `JoinSuite`, 
`SubquerySuite`, `ExplainSuite`, `CachedTableSuite`, 
`DynamicPartitionPruningV1SuiteAEOff`, `ParquetV1FilterSuite`, 
`ParquetV2FilterSuite`, `OrcV1FilterSuite`, `FileSourceStrategySuite`, 
`PruneFileSourcePartitionsSuite` and the DSv2 strategy and SQL suites. All pass 
an [...]
    - Ran the queries above and the variants with a cast, an array index, a 
UDF, `assert_true`, `HAVING`, a window, a view, `<=>`, a guard over the 
propagated column and a correlated scalar subquery before and after this 
change, together with the change of #60162. They no longer fail, and `y = 0 AND 
id > 10 / y` still fails on a row with `y = 0`.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60163 from dongjoon-hyun/SPARK-60983.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    (cherry picked from commit 23bf4c48a84df85943176322c9ba2c27ca3c33a5)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../spark/sql/catalyst/expressions/ExprUtils.scala | 15 ++--
 .../spark/sql/catalyst/optimizer/expressions.scala | 80 +++++++++++++++++++---
 .../optimizer/ConstantPropagationSuite.scala       | 43 +++++++++++-
 .../scala/org/apache/spark/sql/SubquerySuite.scala | 12 ++++
 4 files changed, 136 insertions(+), 14 deletions(-)

diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala
index c88825d41245..d8e7a3e1a6d1 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ExprUtils.scala
@@ -268,11 +268,18 @@ object ExprUtils extends EvalHelper with QueryErrorsBase {
 
   private def canEvaluateUnconditionallyInternal(e: Expression): Boolean = e 
match {
     case _: AttributeReference | _: Literal => true
-    case _: GetStructField | _: GetArrayStructFields | _: GetMapValue =>
-      e.children.forall(canEvaluateUnconditionallyInternal)
+    case _ if isTotalOperator(e) => 
e.children.forall(canEvaluateUnconditionallyInternal)
+    case _ => false
+  }
+
+  /**
+   * Returns true if `e` itself cannot raise an error, whatever the values of 
its children. See
+   * [[canEvaluateUnconditionally]] for the whitelist.
+   */
+  def isTotalOperator(e: Expression): Boolean = e match {
+    case _: GetStructField | _: GetArrayStructFields | _: GetMapValue => true
     case _: And | _: Or | _: Not | _: BinaryComparison | _: IsNull | _: 
IsNotNull |
-         _: IsNaN | _: NullIf | _: Coalesce | _: In | _: InSet =>
-      e.children.forall(canEvaluateUnconditionallyInternal)
+         _: IsNaN | _: NullIf | _: Coalesce | _: In | _: InSet => true
     case _ => false
   }
 }
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
index 842c8e6fed07..9bcc5cde01d4 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala
@@ -141,7 +141,7 @@ object ConstantFolding extends Rule[LogicalPlan] {
  * - Using this mapping, replace occurrence of the attributes with the 
corresponding constant values
  *   in the AND node.
  */
-object ConstantPropagation extends Rule[LogicalPlan] {
+object ConstantPropagation extends Rule[LogicalPlan] with PredicateHelper {
   def apply(plan: LogicalPlan): LogicalPlan = plan.transformUpWithPruning(
     _.containsAllPatterns(LITERAL, FILTER, BINARY_COMPARISON), ruleId) {
     case f: Filter =>
@@ -195,8 +195,10 @@ object ConstantPropagation extends Rule[LogicalPlan] {
           traverse(a.right, replaceChildren = false, nullIsFalse)
         val equalityPredicates = equalityPredicatesLeft ++ 
equalityPredicatesRight
         val newSelf = if (equalityPredicates.nonEmpty && replaceChildren) {
-          Some(And(replaceConstants(newLeft.getOrElse(a.left), 
equalityPredicates),
-            replaceConstants(newRight.getOrElse(a.right), equalityPredicates)))
+          Some(moveEqualitiesFirst(
+            And(replaceConstants(newLeft.getOrElse(a.left), 
equalityPredicates),
+              replaceConstants(newRight.getOrElse(a.right), 
equalityPredicates)),
+            equalityPredicates))
         } else {
           if (newLeft.isDefined || newRight.isDefined) {
             Some(And(newLeft.getOrElse(a.left), newRight.getOrElse(a.right)))
@@ -230,6 +232,16 @@ object ConstantPropagation extends Rule[LogicalPlan] {
   private def safeToReplace(ar: AttributeReference, nullIsFalse: Boolean) =
     !ar.nullable || nullIsFalse
 
+  /**
+   * Replaces attributes in the comparisons of `condition` with their constant 
values.
+   *
+   * The constant is the value of an attribute only on the rows that pass the 
whole condition.
+   * The rewritten condition also runs on other rows, e.g. a row with `y = 1` 
for
+   * `10 / (id - y) > 0 AND y = 0`, and a part of it that becomes constant is 
folded with no row
+   * at all, e.g. `10 / y` for `y = 0 AND id > 10 / y`. So that this raises no 
error the original
+   * condition cannot raise, an expression that can fail keeps its attributes, 
unless all of them
+   * are replaced and the result evaluates without an error.
+   */
   private def replaceConstants(
       condition: Expression,
       equalityPredicates: AttributeMap[(Literal, BinaryComparison)]): 
Expression = {
@@ -237,14 +249,64 @@ object ConstantPropagation extends Rule[LogicalPlan] {
     val predicates = equalityPredicates.values.map(_._2).toSet
     def replaceInComparison(b: BinaryComparison): Expression = {
       lazy val collationSafeReplacement = isSameCollationAttrRefComparison(b)
-      b transform {
-        case a: AttributeReference
-          if isBinaryStable(a.dataType) || collationSafeReplacement =>
-          constantsMap.getOrElse(a, a)
+      def canReplace(a: Attribute): Boolean =
+        constantsMap.contains(a) && (isBinaryStable(a.dataType) || 
collationSafeReplacement)
+      def replace(e: Expression): Expression = e match {
+        case a: AttributeReference if canReplace(a) => constantsMap(a)
+        case _ if ExprUtils.isTotalOperator(e) => e.mapChildren(replace)
+        case _ if e.references.nonEmpty && e.references.forall(canReplace) =>
+          fold(e.transform { case a: AttributeReference if canReplace(a) => 
constantsMap(a) })
+            .getOrElse(e)
+        case _ => e
       }
+      b.mapChildren(replace)
+    }
+    def replaceInCondition(e: Expression): Expression = e match {
+      case b: BinaryComparison if predicates.contains(b) => b
+      case b: BinaryComparison => replaceInComparison(b)
+      case _ if ExprUtils.isTotalOperator(e) => 
e.mapChildren(replaceInCondition)
+      // The children of these include the comparisons of a pulled up 
correlated condition.
+      case _: InSubquery | _: ListQuery | _: Exists => 
e.mapChildren(replaceInCondition)
+      case _ => e
+    }
+    replaceInCondition(condition)
+  }
+
+  /** Returns the literal that `e` folds to, or None if it is not constant or 
fails to fold. */
+  private def fold(e: Expression): Option[Literal] = {
+    if (e.foldable) {
+      try {
+        Some(ConstantFolding.constantFolding(e)).collect { case l: Literal => 
l }
+      } catch {
+        case NonFatal(_) => None
+      }
+    } else {
+      None
     }
-    condition.transformWithPruning(_.containsPattern(BINARY_COMPARISON)) {
-      case b: BinaryComparison if !predicates.contains(b) => 
replaceInComparison(b)
+  }
+
+  /**
+   * Moves the equalities ahead of the other conjuncts of `condition` if one 
that can fail still
+   * refers to a propagated attribute. Such a conjunct also runs on rows where 
an equality does
+   * not hold, with the value of the row, while a guard over the same 
attribute may be folded
+   * away, e.g. `n / d > 1` for `d <> 0 AND n / d > 1 AND d = 5`. An equality 
between an
+   * attribute and a literal cannot fail, so running it first raises no error.
+   */
+  private def moveEqualitiesFirst(
+      condition: Expression,
+      equalityPredicates: AttributeMap[(Literal, BinaryComparison)]): 
Expression = {
+    val predicates = equalityPredicates.values.map(_._2).toSet[Expression]
+    val propagated = AttributeSet(equalityPredicates.keys)
+    val conjuncts = splitConjunctivePredicates(condition)
+    val (equalities, others) = conjuncts.partition(predicates.contains)
+    val reordered = equalities ++ others
+    val keepsFailable = others.exists { c =>
+      c.references.intersect(propagated).nonEmpty && 
!ExprUtils.canEvaluateUnconditionally(c)
+    }
+    if (keepsFailable && reordered != conjuncts) {
+      reordered.reduce(And)
+    } else {
+      condition
     }
   }
 
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
index 106af71a9d65..03fbdfed1e9a 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ConstantPropagationSuite.scala
@@ -22,8 +22,10 @@ import org.apache.spark.sql.catalyst.dsl.expressions._
 import org.apache.spark.sql.catalyst.dsl.plans._
 import org.apache.spark.sql.catalyst.expressions._
 import org.apache.spark.sql.catalyst.plans.PlanTest
-import org.apache.spark.sql.catalyst.plans.logical.{LocalRelation, LogicalPlan}
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LocalRelation, 
LogicalPlan}
 import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.DoubleType
 
 /**
  * Unit tests for constant propagation in expressions.
@@ -214,4 +216,43 @@ class ConstantPropagationSuite extends PlanTest {
       Optimize.execute(testRelation.where(Not(columnA === 1 && columnA === 
columnA + 2)).analyze),
       testRelation.where(Not(columnA === 1) || Not(columnA === columnA + 
2)).analyze)
   }
+
+  test("SPARK-60983: Do not propagate a constant into an expression that can 
fail") {
+    withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+      def divide(e: Expression): Expression = Divide(Literal(10.0), 
e.cast(DoubleType))
+
+      // The conjunct can also run on rows where `b` is not 0. With `b` 
replaced, it would divide by
+      // zero on such a row with `c` = 0, where `c - b` is not 0.
+      val notConstant =
+        testRelation.where(divide(columnC - columnB) > Literal(0.0) && columnB 
=== 0).analyze
+      comparePlans(Optimize.execute(notConstant), notConstant)
+
+      // It can become constant, but keeps the attribute if it fails: no row 
may reach it.
+      val failsToFold = testRelation.where(columnB === 0 && columnA > 
divide(columnB)).analyze
+      comparePlans(Optimize.execute(failsToFold), failsToFold)
+
+      comparePlans(
+        Optimize.execute(testRelation.where(columnB === 2 && columnA > 
divide(columnB)).analyze),
+        testRelation.where(columnB === 2 && columnA > Literal(5.0)).analyze)
+
+      // An expression that fails on its own does not stop the other side of 
the comparison.
+      comparePlans(
+        Optimize.execute(testRelation.where(columnB === 2 && divide(columnC) > 
columnB).analyze),
+        testRelation.where(columnB === 2 && divide(columnC) > 
Literal(2.0)).analyze)
+
+      // The guard `b <> 0` becomes TRUE, so the equality goes ahead of the 
conjunct that keeps `b`.
+      val guarded = Optimize.execute(testRelation
+        .where(columnB =!= 0 && divide(columnC - columnB) > Literal(0.0) && 
columnB === 5).analyze)
+      comparePlans(guarded,
+        testRelation.where(columnB === 5 && divide(columnC - columnB) > 
Literal(0.0)).analyze)
+      val Filter(guardedCondition, _) = guarded
+      assert(splitConjunctivePredicates(guardedCondition).head.sql == "(b = 
5)")
+
+      // The condition of an IF decides whether the branch that can fail runs.
+      val conditional = testRelation
+        .where(If(columnC > columnB, Literal(true), divide(columnC) > 
Literal(0.0)) &&
+          columnB === 0).analyze
+      comparePlans(Optimize.execute(conditional), conditional)
+    }
+  }
 }
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala 
b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
index e2480c7b656d..36b1e4506f12 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SubquerySuite.scala
@@ -3389,4 +3389,16 @@ class SubquerySuite extends SharedSparkSession
       }
     }
   }
+
+  test("SPARK-60983: Propagate a constant into a comparison with a correlated 
scalar subquery") {
+    withTempView("t1", "t2") {
+      Seq((1, 2), (2, 3)).toDF("a", "b").createOrReplaceTempView("t1")
+      Seq((1, 2), (1, 1), (2, 3)).toDF("a", "b").createOrReplaceTempView("t2")
+      // The constant of `t1.a` is not looked up for `t2.a` in the condition 
of the subquery.
+      checkAnswer(
+        sql("SELECT * FROM t1 WHERE t1.a = 1 AND " +
+          "t1.b = (SELECT max(t2.b) FROM t2 WHERE t2.a = t1.a)"),
+        Row(1, 2))
+    }
+  }
 }


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to