This is an automated email from the ASF dual-hosted git repository.

MaxGekk pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 930c571628d7 [SPARK-60315][SQL] Fix inconsistent filter behavior with 
whole-stage codegen on and off when a conjunct is NULL
930c571628d7 is described below

commit 930c571628d7ca0bd94e455b1532e3fc2bbdca3a
Author: Max Gekk <[email protected]>
AuthorDate: Thu Oct 22 09:38:44 2026 +0200

    [SPARK-60315][SQL] Fix inconsistent filter behavior with whole-stage 
codegen on and off when a conjunct is NULL
    
    ### What changes were proposed in this pull request?
    
    This PR fixes inconsistent behavior of a filter between whole-stage codegen 
on and off: when an earlier conjunct of the condition is NULL on a row and a 
later one raises an error on that row, the query failed with codegen off but 
returned rows with codegen on. After this PR, codegen keeps evaluating the 
conjuncts after a NULL one, so it raises the error in that case too. An `IS NOT 
NULL` check that precedes the raising conjunct in the condition still drops the 
row first in both modes.
    
    Whole-stage codegen of a filter condition (`FilterExec`) and of the 
`FILTER` clause of an aggregate function (`AggregateCodegenSupport`) generates 
each top-level conjunct separately and drops the row as soon as one of them is 
FALSE **or NULL** (`if (isNull || !value) continue;`). The interpreted path 
evaluates the whole condition with `And.eval`, which stops only at FALSE. So 
when an earlier conjunct is NULL and a later one raises on that row, the query 
fails with codegen off and retu [...]
    
    This PR changes the generated code so that only a FALSE conjunct stops the 
evaluation:
    
    - `GeneratePredicateHelper` gets six helpers: `splitNotNullPreds` (splits 
the condition into the `IsNotNull` checks and the other conjuncts, keeping the 
number of checks that precede each conjunct), `mayRaise` (whether an expression 
can raise an error, is nondeterministic or may have side effects, via 
`ExprUtils.canEvaluateUnconditionally`), `nullConjunctFlag` (a fresh flag name, 
or `None`), `conjunctCheck` (the per-conjunct check), `withNullConjunctFlag` 
(declares the flag before the [...]
    - A NULL conjunct sets the flag, and the remaining conjuncts are still 
evaluated: `if (isNull) { flag = true; } else if (!value) continue;`. The row 
is dropped after all of them: `if (flag) continue;`.
    - The flag is generated only when some conjunct can be NULL after binding, 
and a conjunct after the first such one, or one of the `IsNotNull` checks, can 
raise an error, is nondeterministic or may have side effects 
(`ExprUtils.canEvaluateUnconditionally`). Otherwise the evaluation after a NULL 
conjunct is not observable and the generated code is unchanged, e.g. for `WHERE 
(a > 5 OR b > 5) AND c = 1 AND d = 2`. The `IsNotNull` checks are never NULL 
and keep their early `continue`, so t [...]
    - With the flag, once a conjunct that can be NULL has been generated, the 
`IsNotNull` checks that precede a later conjunct in the condition are generated 
before it when the conjunct or one of the checks can raise, as `And.eval` 
evaluates them, instead of lazily before the first conjunct that references 
their column. So a FALSE check drops the row before an error the later conjunct 
raises on it, as with codegen off (`splitNotNullPreds` keeps the number of 
checks that precede each conju [...]
    - With the flag, after a conjunct that can be NULL, an `IS NOT NULL` check 
on an expression (SPARK-51356) that is written after the conjunct containing 
the expression is no longer generated before that conjunct. It stays at its 
position, as `And.eval` evaluates it, so it does not raise an error on a row 
that the conjunct drops as FALSE.
    - Both code paths are changed: `generatePredicateCode`, which the aggregate 
`FILTER` clause also uses, and the CSE branch of `FilterExec.doConsume`, which 
the code comments require to stay in step with it.
    - New legacy config `spark.sql.legacy.filterShortCircuitOnNullConjunct` 
(internal, default `false`, version 4.4.0). Set to `true`, it restores the 
previous codegen behavior. The interpreted path is unchanged in both modes.
    
    Join conditions and `MergeRowsExec` generate the whole condition through 
`And.doGenCode`, which already evaluates the right side after a NULL left side, 
so they are not changed.
    
    Not fixed here, on master as well as after this PR:
    - With whole-stage codegen on, an `IS NOT NULL` check on a column is 
generated just before the first conjunct that references the column, so `WHERE 
x IS NOT NULL AND y + 1 > 0 AND x > 0` raises `ARITHMETIC_OVERFLOW` with 
codegen on and drops the row with codegen off. That is tracked in SPARK-60329. 
The check is also generated before that conjunct when it is written after it, 
and so before the conjuncts between that one and its own position, and a check 
on an expression over the column [...]
    - Before the first conjunct that can be NULL, including that conjunct 
itself, or with the legacy config, an `IS NOT NULL` check on an expression is 
still generated before the first conjunct that contains the expression, even 
when it is written after it, so `WHERE coalesce(b, cast(s AS INT)) < 0 AND 
cast(s AS INT) IS NOT NULL` raises `CAST_INVALID_INPUT` with codegen on and 
drops the row with codegen off on `(b, s) = (1, 'x')`. That is tracked in 
SPARK-60377.
    - With whole-stage codegen off, `GeneratePredicate` evaluates a 
subexpression shared by several conjuncts before the condition, so `WHERE NOT 
(a > 0 AND b > 0) AND b < 0 AND y + 1 > 0 AND y + 1 > -1` raises 
`ARITHMETIC_OVERFLOW` with codegen off and returns no rows with codegen on, 
where the FALSE `b < 0` stops the evaluation. That is tracked in SPARK-60392.
    - Whole-stage codegen evaluates the common subexpressions of the update 
expressions of an aggregate function before its `FILTER` clause, so `SELECT 
avg(y + 1) FILTER (WHERE b < 0) FROM t` raises `ARITHMETIC_OVERFLOW` with 
codegen on and returns NULL with codegen off. That is tracked in SPARK-60393.
    - The interleaving of the checks and the conjuncts is mirrored by hand in 
`generatePredicateCode` and in the CSE branch of `FilterExec.doConsume`. 
SPARK-60394 is to compute it once for both.
    
    #### SQL standard
    
    - 7.11 `<where clause>`, GR 2 (and 10.9 `<aggregate function>`, GR 4a, for 
`<filter clause>`): the search condition is effectively evaluated for each row, 
and the result keeps the rows for which it is True.
    - 6.35 `<boolean value expression>`, Table 11: Unknown AND True is Unknown, 
Unknown AND False is False. After an Unknown conjunct the result of the search 
condition still depends on the remaining ones.
    - 9075-1 "Rule evaluation order" (SQL-92 3.3.4.4): only the *inessential* 
parts, those not needed to determine the result of the expression or search 
condition, may be skipped, and whether their exceptions are raised is 
implementation-dependent. After a NULL conjunct the rest is essential. The 
order of evaluation is implementation-dependent too, but in every order the 
raising conjunct is either evaluated first or is still essential, so the 
exception is raised. Only a FALSE conjunct ma [...]
    
    #### Other DBMSes
    
    Table `t(a INT, b INT, y)` with rows `(NULL, 1, <raises>)` and `(0, 1, 
<ok>)`. `<E>` raises on the first row:
    - `y + 1 > 0` with `y` at the maximum of its type: `INT`, or `BIGINT` on 
MySQL and MariaDB, which widen `INT + INT`;
    - `10 / y > 0` with `y = 0` on Oracle, whose `NUMBER` does not overflow at 
32 bits;
    - `abs(y) > 0` with `y` at the minimum `BIGINT` on SQLite, where `+` 
switches to floating point.
    
    The images and credentials are those of 
`connector/docker-integration-tests`.
    
    | Query | PostgreSQL 18.2 | MySQL 9.6.0 | MariaDB 12.2.2 | SQL Server 2022 
| Oracle Free 23.26 | DB2 11.5.9 | H2 2.3.232 | Derby 10.16 | SQLite 3.46 |
    |---|---|---|---|---|---|---|---|---|---|
    | `WHERE NOT (a > 0 AND b > 0) AND <E>` | skip | skip | skip | **raise** | 
skip | **raise** | **raise** | skip | skip |
    | `WHERE <E> AND NOT (a > 0 AND b > 0)` | raise | raise | raise | raise | 
**skip** | raise | raise | **skip** | raise |
    | `WHERE a > 0 AND <E>` | skip | skip | skip | skip | skip | **raise** | 
**raise** | skip | skip |
    | `WHERE b < 0 AND <E>` (FALSE first) | - | - | - | - | - | raise | - | - | 
- |
    | `count(a) FILTER (WHERE NOT (a > 0 AND b > 0) AND <E>)` | **raise** | n/a 
| n/a | n/a | **raise** | n/a | **raise** | n/a | skip |
    
    "-" means no error and an empty result; "n/a" means the `FILTER` clause is 
not supported.
    
    - In WHERE, most engines drop the row at the NULL conjunct. Oracle and 
Derby do it even when the raising conjunct is written first, so they reorder.
    - DB2 and H2 raise. SQL Server raises for the JIRA's shape but not for a 
plain `a > 0` first.
    - For the aggregate `FILTER` clause, PostgreSQL, Oracle and H2 evaluate the 
whole `AND` and raise, as this PR does.
    
    We considered the opposite fix: make the interpreted path stop at the first 
NULL conjunct too. It matches most engines' WHERE behavior, but it departs from 
the standard's text and from what Spark already does with codegen off. Setting 
`spark.sql.legacy.filterShortCircuitOnNullConjunct` to `true` only restores the 
previous whole-stage codegen behavior; with whole-stage codegen off, the query 
still raises.
    
    ### Why are the changes needed?
    
    The same query must not raise with whole-stage codegen off and return rows 
with it on (SPARK-60315). Reproducer (ANSI mode):
    
    ```sql
    CREATE TABLE t USING parquet AS
      SELECT * FROM VALUES (NULL, 1, 2147483647), (0, 1, 0) AS v(a, b, y);
    -- The rows can be written to different files, and `NOT (a > 0 AND b > 0)` 
is pushed down as
    -- `a <= 0 OR b <= 0`, which skips the first row's file by its statistics.
    SET spark.sql.parquet.filterPushdown=false;
    
    SET spark.sql.codegen.wholeStage=true;
    SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0;
    -- before: 0
    -- after:  [ARITHMETIC_OVERFLOW] overflow. ... SQLSTATE: 22003
    
    SET spark.sql.codegen.wholeStage=false;
    SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0;
    -- before and after: [ARITHMETIC_OVERFLOW] overflow. ... SQLSTATE: 22003
    ```
    
    The same happens in non-ANSI mode with year-month interval arithmetic, and 
with the aggregate `FILTER` clause.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. With whole-stage codegen on, an error raised by a later conjunct on a 
row where an earlier conjunct is NULL now fails the query, as it already did 
with codegen off. A row can still be skipped before the filter, for example by 
Parquet row-group statistics, and then no error is raised.
    
    The later conjuncts also run on more rows than before:
    - a nondeterministic conjunct such as `rand(seed)` can return different 
values on the remaining rows;
    - a side-effecting UDF is called on rows that are then dropped;
    - an expensive later conjunct costs more on rows where an earlier, nullable 
conjunct is NULL.
    
    Once a condition has a conjunct that can be NULL, the `IS NOT NULL` checks 
after that conjunct are evaluated in the order of the condition on every row, 
not only on rows where a conjunct is NULL, so a check written before a conjunct 
can now raise an error on a row that the conjunct dropped before, as it does 
with codegen off. Setting 
`spark.sql.legacy.filterShortCircuitOnNullConjunct=true` restores the previous 
behavior. The migration guide has an entry for this.
    
    ### How was this patch tested?
    
    Six new tests in `WholeStageCodegenSuite`. Each runs with whole-stage 
codegen on and off, and with the legacy config on and off; it expects the row 
to be skipped only with codegen on and the legacy config set, and the error in 
every other combination:
    
    - the ANSI reproducer, and a variant with a common `y + 1` that takes the 
CSE path of `FilterExec` (checked from the generated code, with 
`subexpressionElimination.filterExec.enabled` on and off), and a NULL conjunct 
from `OR`; without an error, the CSE path drops the row whose condition is NULL;
    - the non-ANSI year-month interval reproducer 
(`INTERVAL_ARITHMETIC_OVERFLOW`), and a user-written `cast(s AS INT) IS NOT 
NULL` check, which is FALSE on the row, before the NULL conjunct: the row is 
dropped in every combination;
    - a NULL conjunct followed by a FALSE one, which must not raise; it also 
checks that no flag is generated when no conjunct can be NULL, or when nothing 
after the first conjunct that can be NULL can raise, and that it is generated 
once a conjunct can raise;
    - an `IS NOT NULL` check on a column that is FALSE on the row and precedes 
the overflowing conjunct in the condition, inferred or user-written, in the 
non-CSE and CSE paths of `FilterExec` and in the aggregate `FILTER` clause: the 
row is dropped in every combination. A check written after the overflowing 
conjunct stays after it, in both paths, so the query still raises;
    - an `IS NOT NULL` check on `cast(s AS INT)`, which raises on the row, 
written after a conjunct that contains the expression, in the non-CSE and CSE 
paths: the row is dropped in every combination when the conjunct is FALSE, and 
the query raises `CAST_INVALID_INPUT` when it is TRUE. The same check written 
before a conjunct that can be NULL raises on a row where that conjunct is TRUE 
and the next one FALSE, since the checks after the first conjunct that can be 
NULL are placed in the ord [...]
    - the aggregate `FILTER` clause.
    
    The tests use a Parquet table with filter pushdown off, so the filter is 
neither evaluated on a local relation by the optimizer nor skipped by row-group 
statistics.
    
    Also ran `WholeStageCodegenSuite`, `SQLQuerySuite`, `DataFrameSuite`, 
`DataFrameAggregateSuite`, `SubexpressionEliminationSuite`, 
`FilterPushdownSuite` and `SQLQueryTestSuite` locally. The only failure is 
`udtf/udtf.sql - Python UDTFs`, whose golden output differs from the local 
Python's `ValueError` message (`'input' is not in list` vs `list.index(x): x 
not in list`), which is unrelated to this change.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Code (Claude Opus 5.5)
    
    Closes #59573 from MaxGekk/SPARK-60315-filter-null-conjunct.
    
    Authored-by: Max Gekk <[email protected]>
    Signed-off-by: Max Gekk <[email protected]>
---
 docs/sql-migration-guide.md                        |   1 +
 .../org/apache/spark/sql/internal/SQLConf.scala    |  20 ++
 .../sql/execution/basicPhysicalOperators.scala     | 311 +++++++++++++++++----
 .../sql/execution/WholeStageCodegenSuite.scala     | 311 ++++++++++++++++++++-
 4 files changed, 582 insertions(+), 61 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 91cb7c438b56..8642fed0d085 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -56,6 +56,7 @@ license: |
 - Since Spark 4.4, the value columns of `UNPIVOT` and `Dataset.unpivot` merge 
the nested nullability (array element, map value, and struct field nullability) 
of all unpivoted values. Previously, a value column took the nested nullability 
of the first listed value, so a `NULL` array element, map value, or struct 
field of a later value could be returned as a non-`NULL` value such as `0`, and 
`IS NULL` on it returned `false`. A view created by an earlier Spark version 
over such an `UNPIVOT` [...]
 - Since Spark 4.4, when `spark.sql.parquet.fieldId.read.enabled` is true, a 
Parquet read fails with `PARQUET_FIELD_REQUESTED_MORE_THAN_ONCE` if two or more 
requested fields of a struct (or of the top-level schema) resolve to the same 
Parquet field. This happens when one requested field is matched by field ID and 
another by name, or when two requested fields have the same field ID. In Spark 
4.3 and earlier, such a read returned wrong values without an error: with the 
vectorized reader one [...]
 - Since Spark 4.4, when `spark.sql.groupByOrdinal` is true (the default), the 
pipe `AGGREGATE` operator rejects an aliased integer literal in its `GROUP BY` 
clause, such as `TABLE t |> AGGREGATE sum(x) AS s GROUP BY 2 AS grp`, with 
`UNSUPPORTED_FEATURE.PIPE_OPERATOR_AGGREGATE_UNSUPPORTED_CASE`, because a bare 
integer literal in the same position is an ordinal. Previously the aliased 
literal was grouped as a constant. Bare ordinals and aliases on other grouping 
expressions are not affecte [...]
+- Since Spark 4.4, with whole-stage codegen enabled, a filter condition, 
including the `FILTER` clause of an aggregate function, evaluates the remaining 
top-level conjuncts of the condition on a row after one of them evaluates to 
`NULL`, as the SQL standard requires, and does so in the order they appear in 
the condition, as Spark already did with whole-stage codegen disabled, which 
also evaluates a subexpression shared by several conjuncts before the 
condition. Only a `FALSE` conjunct st [...]
 
 - Since Spark 4.4, when a format is specified, `to_timestamp`, 
`try_to_timestamp`, and `to_timestamp_ltz` cast a numeric first argument to a 
string and parse it with the format, for example `to_timestamp(20201231, 
'yyyyMMdd')` returns `2020-12-31 00:00:00`. This matches the existing behavior 
when `spark.sql.timestampType` is `TIMESTAMP_NTZ`. Previously, with the default 
`TIMESTAMP_LTZ` timestamp type, such calls failed with `INTERNAL_ERROR`. Note 
that the string form of the number is par [...]
 
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 3b9d0d1f6ae0..63a1cdd4811d 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -1763,6 +1763,23 @@ object SQLConf {
       .booleanConf
       .createWithDefault(true)
 
+  val LEGACY_FILTER_SHORT_CIRCUIT_ON_NULL_CONJUNCT =
+    buildConf("spark.sql.legacy.filterShortCircuitOnNullConjunct")
+      .internal()
+      .doc("When true, whole stage codegen of a filter condition, including 
the FILTER clause " +
+        "of an aggregate function, drops a row at the first top-level conjunct 
that evaluates " +
+        "to NULL without evaluating the remaining conjuncts, so an error that 
one of them " +
+        "would raise on that row is not raised. When false, only a FALSE 
conjunct stops the " +
+        "evaluation, as the SQL standard requires and as the interpreted 
evaluation does, and " +
+        "the row is dropped after the remaining conjuncts are evaluated. Once 
a condition has " +
+        "a conjunct that can be NULL, the IS NOT NULL checks after that 
conjunct are then " +
+        "evaluated in the order of the condition on every row, not only on 
rows where a " +
+        "conjunct is NULL.")
+      .version("4.4.0")
+      .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+      .booleanConf
+      .createWithDefault(false)
+
   val CASE_SENSITIVE = buildConf(SqlApiConfHelper.CASE_SENSITIVE_KEY)
     .internal()
     .doc("Whether the query analyzer should be case sensitive or not. " +
@@ -9689,6 +9706,9 @@ class SQLConf extends Serializable with Logging with 
SqlApiConf {
   def subexpressionEliminationFilterExecEnabled: Boolean =
     getConf(SUBEXPRESSION_ELIMINATION_FILTER_EXEC_ENABLED)
 
+  def legacyFilterShortCircuitOnNullConjunct: Boolean =
+    getConf(LEGACY_FILTER_SHORT_CIRCUIT_ON_NULL_CONJUNCT)
+
   def autoBroadcastJoinThreshold: Long = getConf(AUTO_BROADCASTJOIN_THRESHOLD)
 
   def limitInitialNumPartitions: Int = getConf(LIMIT_INITIAL_NUM_PARTITIONS)
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
index d8314b8807a0..3c94cab249b5 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/basicPhysicalOperators.scala
@@ -137,47 +137,182 @@ trait GeneratePredicateHelper extends PredicateHelper {
       condition: Expression,
       inputAttrs: Seq[Attribute],
       inputExprCode: Seq[ExprCode]): String = {
-    val (notNullPreds, otherPreds) = 
splitConjunctivePredicates(condition).partition {
-      case IsNotNull(a) => isNullIntolerant(a) && 
a.references.subsetOf(AttributeSet(inputAttrs))
-      case _ => false
-    }
+    val (notNullPreds, otherPreds, precedingNotNulls) =
+      splitNotNullPreds(condition, AttributeSet(inputAttrs))
     val nonNullAttrExprIds = 
notNullPreds.flatMap(_.references).distinct.map(_.exprId)
     val outputAttrs = outputWithNullability(inputAttrs, nonNullAttrExprIds)
     generatePredicateCode(
-      ctx, inputAttrs, inputExprCode, outputAttrs, notNullPreds, otherPreds,
+      ctx, inputAttrs, inputExprCode, notNullPreds, otherPreds,
+      otherPreds.map(BindReferences.bindReference(_, outputAttrs)), 
precedingNotNulls,
       nonNullAttrExprIds)
   }
 
+  /**
+   * Splits `condition` into the `IsNotNull` checks on null-intolerant 
expressions over
+   * `inputAttrs`, the other conjuncts, and for each of the other conjuncts 
the number of the
+   * checks that precede it in `condition`. The conjuncts keep their order 
within each group, so
+   * the checks before the k-th other conjunct are the first 
`precedingNotNulls(k)` checks.
+   */
+  protected def splitNotNullPreds(
+      condition: Expression,
+      inputAttrs: AttributeSet): (Seq[Expression], Seq[Expression], Seq[Int]) 
= {
+    val notNullPreds = mutable.ArrayBuffer.empty[Expression]
+    val otherPreds = mutable.ArrayBuffer.empty[Expression]
+    val precedingNotNulls = mutable.ArrayBuffer.empty[Int]
+    splitConjunctivePredicates(condition).foreach {
+      case p @ IsNotNull(a) if isNullIntolerant(a) && 
a.references.subsetOf(inputAttrs) =>
+        notNullPreds += p
+      case p =>
+        otherPreds += p
+        precedingNotNulls += notNullPreds.length
+    }
+    (notNullPreds.toSeq, otherPreds.toSeq, precedingNotNulls.toSeq)
+  }
+
+  /**
+   * Whether evaluating `e` on a row can be observed other than through its 
value: it can raise
+   * an error, is nondeterministic or may have side effects, such as a UDF. 
Such an expression
+   * has to be evaluated on the rows, and in the order, that `And.eval` 
evaluates it on.
+   */
+  protected def mayRaise(e: Expression): Boolean = 
!ExprUtils.canEvaluateUnconditionally(e)
+
+  /**
+   * Returns a fresh name for the flag that records a conjunct evaluated to 
NULL, or None if the
+   * row is dropped at the first NULL conjunct (the legacy behavior), if none 
of `otherPreds`,
+   * bound with the nullability they are generated with in `boundOtherPreds`, 
can be NULL, or if
+   * neither a conjunct after the first one that can be NULL nor one of the 
`notNullPreds` checks
+   * `mayRaise`: the evaluation after a NULL conjunct is then not observable, 
and the generated
+   * code is the same as without the flag.
+   *
+   * A row is kept only when the whole condition is TRUE, so both FALSE and 
NULL drop it. Only
+   * a FALSE conjunct decides the result of the AND on its own, though: after 
a NULL one, the
+   * remaining conjuncts must still be evaluated, as in `And.eval`, so that an 
error they raise
+   * on the row is raised (SPARK-60315). The flag drops the row once all of 
them are evaluated.
+   */
+  protected def nullConjunctFlag(
+      ctx: CodegenContext,
+      otherPreds: Seq[Expression],
+      boundOtherPreds: Seq[Expression],
+      notNullPreds: Seq[Expression]): Option[String] = {
+    val firstNullable = boundOtherPreds.indexWhere(_.nullable)
+    if (conf.legacyFilterShortCircuitOnNullConjunct || firstNullable == -1 ||
+        !(otherPreds.drop(firstNullable + 1).exists(mayRaise) || 
notNullPreds.exists(mayRaise))) {
+      None
+    } else {
+      Some(ctx.freshName("nullConjunct"))
+    }
+  }
+
+  /**
+   * Generates the check that drops the row when the conjunct evaluated into 
`ev` is FALSE, or
+   * when it is NULL and `nullFlag` is None. With `nullFlag`, a NULL conjunct 
sets the flag.
+   */
+  protected def conjunctCheck(ev: ExprCode, nullable: Boolean, nullFlag: 
Option[String]): String = {
+    (nullable, nullFlag) match {
+      case (false, _) => s"if (!${ev.value}) continue;"
+      case (true, None) => s"if (${ev.isNull} || !${ev.value}) continue;"
+      case (true, Some(flag)) =>
+        s"if (${ev.isNull}) { $flag = true; } else if (!${ev.value}) continue;"
+    }
+  }
+
+  /** Wraps the checks of all conjuncts in `code` with the declaration and 
test of `nullFlag`. */
+  protected def withNullConjunctFlag(code: String, nullFlag: Option[String]): 
String = {
+    nullFlag match {
+      case Some(flag) =>
+        s"""
+           |boolean $flag = false;
+           |$code
+           |if ($flag) continue;
+         """.stripMargin
+      case None => code
+    }
+  }
+
+  /**
+   * Returns the checks of `notNullPreds` that precede the k-th other conjunct 
`conjunct` in the
+   * condition and are not generated yet, and marks them generated, when 
`afterNullable` is set
+   * and the conjunct or one of these checks `mayRaise`, or Nil.
+   *
+   * With the NULL-conjunct flag, a NULL conjunct no longer drops the row, so 
the conjuncts after
+   * it are evaluated in the order of the condition, as `And.eval` does: a 
FALSE IsNotNull check
+   * drops the row before an error a later conjunct raises on it 
(SPARK-60315). `afterNullable`
+   * is set once a conjunct that can be NULL has been generated. The conjuncts 
before it are
+   * never reached after a NULL conjunct, so the checks before them keep the 
lazy placement. So
+   * do the checks before a conjunct when neither it nor one of them can 
raise: their order is
+   * not observable, and the checks are still generated before the next 
conjunct that can raise.
+   *
+   * The exception to the order is a check on a column, or on an expression 
over the column, that
+   * a later conjunct references: the check on the column is generated before 
the first such
+   * conjunct even when it is written after it, since the conjunct is bound 
with the column
+   * non-nullable, and so before the conjuncts between that one and its own 
position. The check
+   * is never NULL, so it only drops a row that `And.eval` drops too, unless 
one of these
+   * conjuncts raises an error on the row first. Also, the first conjunct that 
can be NULL keeps
+   * the checks on the expressions it contains before it (SPARK-51356).
+   */
+  protected def precedingNotNullChecks(
+      notNullPreds: Seq[Expression],
+      precedingNotNulls: Seq[Int],
+      k: Int,
+      conjunct: Expression,
+      afterNullable: Boolean,
+      generated: Array[Boolean]): Seq[Expression] = {
+    if (afterNullable) {
+      val pending = 
notNullPreds.take(precedingNotNulls(k)).zipWithIndex.filterNot {
+        case (_, idx) => generated(idx)
+      }
+      if (pending.nonEmpty && (mayRaise(conjunct) || pending.exists(p => 
mayRaise(p._1)))) {
+        pending.map { case (p, idx) =>
+          generated(idx) = true
+          p
+        }
+      } else {
+        Nil
+      }
+    } else {
+      Nil
+    }
+  }
+
   protected def generatePredicateCode(
       ctx: CodegenContext,
       inputAttrs: Seq[Attribute],
       inputExprCode: Seq[ExprCode],
-      outputAttrs: Seq[Attribute],
       notNullPreds: Seq[Expression],
       otherPreds: Seq[Expression],
+      boundOtherPreds: Seq[Expression],
+      precedingNotNulls: Seq[Int],
       nonNullAttrExprIds: Seq[ExprId]): String = {
+    // The IsNotNull checks are never NULL, so only `otherPreds` can need the 
flag. The caller
+    // binds them against the output with the checked columns tightened to 
non-nullable, in
+    // `boundOtherPreds`.
+    val nullFlag = nullConjunctFlag(ctx, otherPreds, boundOtherPreds, 
notNullPreds)
+
     /**
-     * Generates code for `c`, using `in` for input attributes and `attrs` for 
nullability.
+     * Generates code for `c`, given as `bound` after binding against the 
attributes whose
+     * nullability it is generated with. A NULL value sets `flag`, or drops 
the row if None.
      */
-    def genPredicate(c: Expression, in: Seq[ExprCode], attrs: Seq[Attribute]): 
String = {
-      val bound = BindReferences.bindReference(c, attrs)
-      val evaluated = evaluateRequiredVariables(inputAttrs, in, c.references)
+    def genPredicate(c: Expression, bound: Expression, flag: Option[String]): 
String = {
+      val evaluated = evaluateRequiredVariables(inputAttrs, inputExprCode, 
c.references)
 
       // Generate the code for the predicate.
       val ev = ExpressionCanonicalizer.execute(bound).genCode(ctx)
-      val nullCheck = if (bound.nullable) {
-        s"${ev.isNull} || "
-      } else {
-        ""
-      }
 
       s"""
          |$evaluated
          |${ev.code}
-         |if (${nullCheck}!${ev.value}) continue;
+         |${conjunctCheck(ev, bound.nullable, flag)}
        """.stripMargin
     }
 
+    /**
+     * Generates an IsNotNull check with the nullability the child produced. 
The check is never
+     * NULL, and must drop the row before the conjuncts bound with its column 
non-nullable, so
+     * it does not get the flag.
+     */
+    def genNotNull(c: Expression): String =
+      genPredicate(c, BindReferences.bindReference(c, inputAttrs), None)
+
     // To generate the predicates we will follow this algorithm.
     // For each predicate that is not IsNotNull, we will generate them one by 
one loading attributes
     // as necessary. For each of both attributes, if there is an IsNotNull 
predicate we will
@@ -186,13 +321,24 @@ trait GeneratePredicateHelper extends PredicateHelper {
     // This has the property of not doing redundant IsNotNull checks and 
taking better advantage of
     // short-circuiting, not loading attributes until they are needed.
     // This is very perf sensitive.
+    // With the NULL-conjunct flag, once a predicate that can be NULL is 
generated, the IsNotNull
+    // checks that precede a later predicate in the condition are generated 
before it, see
+    // `precedingNotNullChecks`, and a check on an expression that is written 
after the predicate
+    // is not generated before it, see the nested checks below.
     // NOTE: `FilterExec.doConsume`'s CSE branch inlines the same interleaving 
so it can
     // emit CSE state precomputes between the IsNotNull checks and each 
otherPred body.
     // Any change to this algorithm must be mirrored there (and vice versa).
     // TODO: revisit this. We can consider reordering predicates as well.
     val generatedIsNotNullChecks = new Array[Boolean](notNullPreds.length)
     val extraIsNotNullAttrs = mutable.Set[Attribute]()
-    val generated = otherPreds.map { c =>
+    var nullableSeen = false
+    val generated = otherPreds.zip(boundOtherPreds).zipWithIndex.map { case 
((c, bound), k) =>
+      val afterNullable = nullFlag.isDefined && nullableSeen
+      val precedingChecks = precedingNotNullChecks(
+        notNullPreds, precedingNotNulls, k, c, afterNullable,
+        generatedIsNotNullChecks).map(genNotNull).mkString("\n").trim
+      nullableSeen ||= bound.nullable
+
       val nullChecks = c.references.map { r =>
         val idx = notNullPreds.indexWhere {
           case IsNotNull(n) => n.semanticEquals(r)
@@ -201,44 +347,58 @@ trait GeneratePredicateHelper extends PredicateHelper {
         if (idx != -1 && !generatedIsNotNullChecks(idx)) {
           generatedIsNotNullChecks(idx) = true
           // Use the child's output. The nullability is what the child 
produced.
-          genPredicate(notNullPreds(idx), inputExprCode, inputAttrs)
-        } else if (nonNullAttrExprIds.contains(r.exprId) && 
!extraIsNotNullAttrs.contains(r)) {
+          genNotNull(notNullPreds(idx))
+        } else if ((idx == -1 || !afterNullable) && 
nonNullAttrExprIds.contains(r.exprId) &&
+            !extraIsNotNullAttrs.contains(r)) {
+          // A synthetic check for a column covered only by a check on an 
expression, or, before
+          // the flag applies, for a column whose check was generated before 
an earlier predicate.
+          // After a predicate that can be NULL, the check generated by 
`precedingNotNullChecks`
+          // already covers the column.
           extraIsNotNullAttrs += r
-          genPredicate(IsNotNull(r), inputExprCode, inputAttrs)
+          genNotNull(IsNotNull(r))
         } else {
           ""
         }
       }.mkString("\n").trim
 
-      val nestedNullChecks = notNullPreds.zipWithIndex.collect {
+      // A check on an expression that the predicate contains is generated 
before the predicate
+      // (SPARK-51356). After a predicate that can be NULL, the checks written 
before this one
+      // are generated above when one of them or the predicate can raise, and 
a check written
+      // after it is left to the remaining checks, as `And.eval` evaluates it 
after the
+      // predicate, so that it does not raise an error on a row that the 
predicate drops. The
+      // columns of the expression are covered by the checks above.
+      val nestedNullChecks = if (afterNullable) "" else 
notNullPreds.zipWithIndex.collect {
         case (p @ IsNotNull(n), idx)
             if !generatedIsNotNullChecks(idx) && !n.isInstanceOf[Attribute] &&
               c.exists(_.semanticEquals(n)) =>
           generatedIsNotNullChecks(idx) = true
-          genPredicate(p, inputExprCode, inputAttrs)
+          genNotNull(p)
       }.mkString("\n").trim
 
       // Here we use *this* operator's output with this output's nullability 
since we already
       // enforced them with the IsNotNull checks above.
       s"""
+         |$precedingChecks
          |$nullChecks
          |$nestedNullChecks
-         |${genPredicate(c, inputExprCode, outputAttrs)}
+         |${genPredicate(c, bound, nullFlag)}
        """.stripMargin.trim
     }.mkString("\n")
 
     val nullChecks = notNullPreds.zipWithIndex.map { case (c, idx) =>
       if (!generatedIsNotNullChecks(idx)) {
-        genPredicate(c, inputExprCode, inputAttrs)
+        genNotNull(c)
       } else {
         ""
       }
     }.mkString("\n")
 
-    s"""
-       |$generated
-       |$nullChecks
-     """.stripMargin
+    withNullConjunctFlag(
+      s"""
+         |$generated
+         |$nullChecks
+       """.stripMargin,
+      nullFlag)
   }
 }
 
@@ -246,18 +406,18 @@ trait GeneratePredicateHelper extends PredicateHelper {
 case class FilterExec(condition: Expression, child: SparkPlan)
   extends UnaryExecNode with CodegenSupport with GeneratePredicateHelper with 
SafeForKWayMerge {
 
-  // Split out all the IsNotNulls from condition.
-  private val (notNullPreds, otherPreds) = 
splitConjunctivePredicates(condition).partition {
-    case IsNotNull(a) => isNullIntolerant(a) && 
a.references.subsetOf(child.outputSet)
-    case _ => false
-  }
+  // Split out all the IsNotNulls from condition, keeping how many of them 
precede each of the
+  // other conjuncts.
+  private val (notNullPreds, otherPreds, precedingNotNulls) =
+    splitNotNullPreds(condition, child.outputSet)
 
   // The columns that will filtered out by `IsNotNull` could be considered as 
not nullable.
   private val notNullAttributes = 
notNullPreds.flatMap(_.references).distinct.map(_.exprId)
 
-  // `otherPreds` bound against this operator's `output`, shared between the 
CSE gate in
-  // `doConsume` and the CSE codegen itself. Codegen-only derived state, so 
`@transient`: it is
-  // computed on the driver during code generation and never accessed on 
executors.
+  // `otherPreds` bound against this operator's `output`, shared by the CSE 
gate in `doConsume`
+  // and both predicate code paths, the CSE branch and 
`generatePredicateCode`. Codegen-only
+  // derived state, so `@transient`: it is computed on the driver during code 
generation and
+  // never accessed on executors.
   @transient private lazy val boundOtherPreds: Seq[Expression] =
     otherPreds.map(BindReferences.bindReference(_, output))
 
@@ -311,7 +471,13 @@ case class FilterExec(condition: Expression, child: 
SparkPlan)
     //       the tightened-output binding below is safe without evaluating the
     //       throw-capable inner expression. Then emit the otherPred's CSE 
precompute
     //       and body. Remaining notNullPreds (including complex-child forms 
whose refs
-    //       got synthetic coverage) emit after all otherPreds.
+    //       got synthetic coverage) emit after all otherPreds. With the 
NULL-conjunct
+    //       flag, once an otherPred that can be NULL is emitted, the 
notNullPreds that
+    //       precede a later otherPred in the condition are emitted before it 
first, when it
+    //       or one of them can raise (`precedingNotNullChecks`), and a complex
+    //       `IsNotNull(expr)` written after the otherPred is left to the 
remaining
+    //       notNullPreds, so it does not raise on a row the otherPred 
rejects, as in the
+    //       non-CSE branch (SPARK-60315).
     //   (c) CSE state materialization: binding otherPreds (and the CSE 
analysis)
     //       against `output` (with `notNullAttributes` tightened to 
non-nullable)
     //       requires that the matching IsNotNull has already short-circuited 
-- else
@@ -360,6 +526,8 @@ case class FilterExec(condition: Expression, child: 
SparkPlan)
         val subExprs =
           
ctx.subexpressionEliminationForWholeStageCodegen(otherPredsEquivalentExpressions)
 
+        val nullFlag = nullConjunctFlag(ctx, otherPreds, boundOtherPreds, 
notNullPreds)
+
         // Group CSE states by the index of the first otherPred that 
references them.
         // `evaluateSubExprEliminationState` recursively emits each state's 
children
         // before the state itself and marks emitted code as `EmptyBlock`, so 
emitting
@@ -373,29 +541,52 @@ case class FilterExec(condition: Expression, child: 
SparkPlan)
         // Emit an IsNotNull check, binding against child.output so the inner 
expression
         // is evaluated with its original nullability (the 
tightened-nullability `output`
         // is only appropriate inside otherPreds, where the required checks 
have fired).
-        // The `bound.nullable` gate below is defensive -- `IsNotNull` always 
produces a
-        // non-nullable Boolean, so this branch is structurally unreachable 
today; it
-        // mirrors the non-CSE helper's shape to keep future refactors safe.
+        // Passing `bound.nullable` to `conjunctCheck` is defensive -- 
`IsNotNull` always
+        // produces a non-nullable Boolean, so its NULL handling is 
unreachable here; the
+        // non-CSE helper generates its IsNotNull checks the same way. The 
check does not get
+        // the NULL-conjunct flag: it must drop the row before the otherPreds 
and CSE
+        // precomputes bound with its column non-nullable, see (c).
         def genNotNull(pred: Expression): String = {
           val bound = BindReferences.bindReference(pred, child.output)
           val evaluated = evaluateRequiredVariables(child.output, input, 
pred.references)
           val ev = ExpressionCanonicalizer.execute(bound).genCode(ctx)
-          val nullCheck = if (bound.nullable) s"${ev.isNull} || " else ""
           s"""
              |$evaluated
              |${ev.code}
-             |if (${nullCheck}!${ev.value}) continue;
+             |${conjunctCheck(ev, bound.nullable, None)}
            """.stripMargin
         }
 
+        // Emit a check on a complex expression with no CSE state: the 
expression is bound
+        // against child.output, with its original nullability, while the CSE 
states are
+        // computed over `boundOtherPreds`, bound against the tightened 
`output`, and a state
+        // precomputed for the check could materialize a NULL into a shared 
variable, see (c).
+        def genNotNullWithoutCSE(pred: Expression): String = {
+          var checkCode: String = null
+          ctx.withSubExprEliminationExprs(Map.empty) {
+            checkCode = genNotNull(pred)
+            Seq.empty
+          }
+          checkCode
+        }
+
         val generatedIsNotNullChecks = new Array[Boolean](notNullPreds.length)
         val extraIsNotNullAttrs = mutable.Set[Attribute]()
+        var nullableSeen = false
 
         val predCode: String = {
           val parts = new StringBuilder
           ctx.withSubExprEliminationExprs(subExprs.states) {
             otherPreds.zip(boundOtherPreds).zipWithIndex.foreach {
               case ((orig, bound), idx) =>
+                val afterNullable = nullFlag.isDefined && nullableSeen
+                precedingNotNullChecks(
+                    notNullPreds, precedingNotNulls, idx, orig, afterNullable,
+                    generatedIsNotNullChecks).foreach { p =>
+                  parts.append(genNotNullWithoutCSE(p))
+                  parts.append('\n')
+                }
+                nullableSeen ||= bound.nullable
                 orig.references.foreach { r =>
                   val ni = notNullPreds.indexWhere {
                     case IsNotNull(c) => c.semanticEquals(r)
@@ -405,35 +596,34 @@ case class FilterExec(condition: Expression, child: 
SparkPlan)
                     generatedIsNotNullChecks(ni) = true
                     parts.append(genNotNull(notNullPreds(ni)))
                     parts.append('\n')
-                  } else if (notNullAttributes.contains(r.exprId) &&
+                  } else if ((ni == -1 || !afterNullable) && 
notNullAttributes.contains(r.exprId) &&
                       !extraIsNotNullAttrs.contains(r)) {
+                    // As in `generatePredicateCode`: after a nullable 
otherPred, a check
+                    // generated by `precedingNotNullChecks` already covers 
the column.
                     extraIsNotNullAttrs += r
                     parts.append(genNotNull(IsNotNull(r)))
                     parts.append('\n')
                   }
                 }
-                notNullPreds.zipWithIndex.foreach {
-                  case (p @ IsNotNull(n), ni)
-                      if !generatedIsNotNullChecks(ni) && 
!n.isInstanceOf[Attribute] &&
-                        orig.exists(_.semanticEquals(n)) =>
-                    generatedIsNotNullChecks(ni) = true
-                    var checkCode: String = null
-                    ctx.withSubExprEliminationExprs(Map.empty) {
-                      checkCode = genNotNull(p)
-                      Seq.empty
-                    }
-                    parts.append(checkCode)
-                    parts.append('\n')
-                  case _ =>
+                // A complex `IsNotNull(expr)` is skipped after a nullable 
otherPred, see (b).
+                if (!afterNullable) {
+                  notNullPreds.zipWithIndex.foreach {
+                    case (p @ IsNotNull(n), ni)
+                        if !generatedIsNotNullChecks(ni) && 
!n.isInstanceOf[Attribute] &&
+                          orig.exists(_.semanticEquals(n)) =>
+                      generatedIsNotNullChecks(ni) = true
+                      parts.append(genNotNullWithoutCSE(p))
+                      parts.append('\n')
+                    case _ =>
+                  }
                 }
                 statesByFirstUse.get(idx).foreach { states =>
                   parts.append(ctx.evaluateSubExprEliminationState(states))
                   parts.append('\n')
                 }
                 val ev = ExpressionCanonicalizer.execute(bound).genCode(ctx)
-                val nullCheck = if (bound.nullable) s"${ev.isNull} || " else ""
                 parts.append(ev.code.toString)
-                parts.append(s"\nif (${nullCheck}!${ev.value}) continue;\n")
+                parts.append(s"\n${conjunctCheck(ev, bound.nullable, 
nullFlag)}\n")
             }
             Seq.empty
           }
@@ -451,10 +641,11 @@ case class FilterExec(condition: Expression, child: 
SparkPlan)
           case (p, i) if !generatedIsNotNullChecks(i) => genNotNull(p)
         }.mkString("\n")
 
-        (inputVarsEvalCode, predCode + "\n" + leftoverNotNull)
+        (inputVarsEvalCode, withNullConjunctFlag(predCode + "\n" + 
leftoverNotNull, nullFlag))
       } else {
         ("", generatePredicateCode(
-          ctx, child.output, input, output, notNullPreds, otherPreds, 
notNullAttributes))
+          ctx, child.output, input, notNullPreds, otherPreds, boundOtherPreds,
+          precedingNotNulls, notNullAttributes))
       }
 
     // Reset the isNull to false for the not-null columns, then the followed 
operators could
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
index 1a17dd700c5f..15396c3c661c 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/WholeStageCodegenSuite.scala
@@ -26,7 +26,7 @@ import scala.util.{Failure, Success, Try}
 import org.apache.logging.log4j.Level
 import org.scalatest.time.SpanSugar._
 
-import org.apache.spark.SparkException
+import org.apache.spark.{SparkArithmeticException, SparkException, 
SparkNumberFormatException}
 import org.apache.spark.metrics.source.CodegenMetrics
 import org.apache.spark.rdd.MapPartitionsWithEvaluatorRDD
 import org.apache.spark.sql.{DataFrame, Dataset, Row, SaveMode}
@@ -35,6 +35,7 @@ import 
org.apache.spark.sql.catalyst.expressions.codegen.{ByteCodeStats, CodeAnd
 import org.apache.spark.sql.catalyst.plans.logical.MergeRows
 import org.apache.spark.sql.catalyst.plans.logical.MergeRows.{Keep, Update}
 import org.apache.spark.sql.classic
+import org.apache.spark.sql.errors.DataTypeErrors.{toSQLConf, toSQLId}
 import org.apache.spark.sql.execution.adaptive.DisableAdaptiveExecutionSuite
 import org.apache.spark.sql.execution.aggregate.{HashAggregateExec, 
SortAggregateExec}
 import org.apache.spark.sql.execution.columnar.InMemoryTableScanExec
@@ -2847,6 +2848,314 @@ class WholeStageCodegenSuite extends SharedSparkSession
     }
   }
 
+  // A Parquet table, so that the filter is not evaluated by the optimizer on 
a local relation.
+  // On the first row `NOT (a > 0 AND b > 0)` is NULL, and no `isnotnull(a)` 
is inferred from it.
+  // The second row passes every condition of the tests below, except `b < 0`, 
`(a > 0 OR b < 0)`
+  // and `coalesce(b, cast(s AS INT)) < 0`. The rows can be
+  // written to different files, so the filter is not pushed down, which would 
skip the first
+  // row's file by its statistics.
+  private def withNullConjunctTable(y: Int)(f: => Unit): Unit = {
+    withTable("t") {
+      sql("CREATE TABLE t USING parquet AS " +
+        s"SELECT * FROM VALUES (NULL, 1, $y, NULL, 'x'), (0, 1, 0, 1, '1') AS 
v(a, b, y, x, s)")
+      f
+    }
+  }
+
+  // A common subexpression after the NULL conjunct takes the CSE path of 
FilterExec.
+  private val nullConjunctCSEQuery =
+    "SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0 AND y + 1 > -1"
+
+  private def checkArithmeticOverflow(query: String): Unit = {
+    checkError(
+      exception = intercept[SparkArithmeticException] {
+        sql(query).collect()
+      },
+      condition = "ARITHMETIC_OVERFLOW",
+      parameters = Map(
+        "message" -> "overflow",
+        "alternative" -> " Use 'try_add' to tolerate overflow and return NULL 
instead.",
+        "config" -> toSQLConf(SQLConf.ANSI_ENABLED.key)),
+      context = ExpectedContext(
+        fragment = "y + 1",
+        start = query.indexOf("y + 1"),
+        stop = query.indexOf("y + 1") + 4))
+  }
+
+  private def checkCastInvalidInput(query: String): Unit = {
+    val start = query.lastIndexOf("cast(s AS INT)")
+    checkError(
+      exception = intercept[SparkNumberFormatException] {
+        sql(query).collect()
+      },
+      condition = "CAST_INVALID_INPUT",
+      parameters = Map(
+        "expression" -> "'x'",
+        "sourceType" -> "\"STRING\"",
+        "targetType" -> "\"INT\"",
+        "ansiConfig" -> toSQLConf(SQLConf.ANSI_ENABLED.key)),
+      context = ExpectedContext(fragment = "cast(s AS INT)", start = start, 
stop = start + 13))
+  }
+
+  // Runs `f` with whole-stage codegen on and off and with the legacy behavior 
on and off, and
+  // passes it whether the filter drops a row at its first NULL conjunct.
+  private def withNullConjunctConfs(otherConfs: (String, String)*)(f: Boolean 
=> Unit): Unit = {
+    for (codegen <- Seq(true, false); legacy <- Seq(true, false)) {
+      withSQLConf(otherConfs ++ Seq(
+          SQLConf.PARQUET_FILTER_PUSHDOWN_ENABLED.key -> "false",
+          SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegen.toString,
+          SQLConf.LEGACY_FILTER_SHORT_CIRCUIT_ON_NULL_CONJUNCT.key -> 
legacy.toString): _*) {
+        withClue(s"wholeStage=$codegen, legacy=$legacy: ") {
+          f(codegen && legacy)
+        }
+      }
+    }
+  }
+
+  test("SPARK-60315: a filter evaluates the conjuncts after a NULL one") {
+    withNullConjunctTable(Int.MaxValue) {
+      // The CSE path evaluates the common `y + 1` once, the non-CSE path at 
each use. Both
+      // generate the flag that keeps the row's evaluation going after a NULL 
conjunct.
+      def cseQueryCode(filterExecCSE: Boolean): String = {
+        withSQLConf(
+            SQLConf.ANSI_ENABLED.key -> "true",
+            SQLConf.SUBEXPRESSION_ELIMINATION_FILTER_EXEC_ENABLED.key -> 
filterExecCSE.toString) {
+          val code = 
codegenString(sql(nullConjunctCSEQuery).queryExecution.executedPlan)
+          assert(code.contains("nullConjunct"))
+          code
+        }
+      }
+      assert("addExact".r.findAllIn(cseQueryCode(filterExecCSE = true)).length 
<
+        "addExact".r.findAllIn(cseQueryCode(filterExecCSE = false)).length)
+
+      Seq("true", "false").foreach { filterExecCSE =>
+        Seq(
+          "SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0",
+          nullConjunctCSEQuery
+        ).foreach { query =>
+          withNullConjunctConfs(
+              SQLConf.ANSI_ENABLED.key -> "true",
+              SQLConf.SUBEXPRESSION_ELIMINATION_FILTER_EXEC_ENABLED.key -> 
filterExecCSE) {
+            skipsAtNull =>
+              if (skipsAtNull) {
+                checkAnswer(sql(query), Row(0))
+              } else {
+                checkArithmeticOverflow(query)
+              }
+          }
+        }
+      }
+    }
+    // A NULL conjunct from OR. The second row does not pass it.
+    withNullConjunctTable(Int.MaxValue) {
+      val query = "SELECT a FROM t WHERE (a > 0 OR b < 0) AND y + 1 > 0"
+      withNullConjunctConfs(SQLConf.ANSI_ENABLED.key -> "true") { skipsAtNull 
=>
+        if (skipsAtNull) {
+          checkAnswer(sql(query), Seq.empty)
+        } else {
+          checkArithmeticOverflow(query)
+        }
+      }
+    }
+    // Without an error, the CSE path drops the first row after its last 
conjunct is evaluated.
+    withNullConjunctTable(0) {
+      withNullConjunctConfs(
+          SQLConf.ANSI_ENABLED.key -> "true",
+          SQLConf.SUBEXPRESSION_ELIMINATION_FILTER_EXEC_ENABLED.key -> "true") 
{ _ =>
+        checkAnswer(sql(nullConjunctCSEQuery), Row(0))
+      }
+    }
+  }
+
+  test("SPARK-60315: a filter evaluates the conjuncts after a NULL one in 
non-ANSI mode") {
+    // Year-month interval arithmetic raises on overflow in every mode.
+    withNullConjunctTable(178956970) {
+      val condition = "NOT (a > 0 AND b > 0) AND " +
+        "make_ym_interval(y, 0) + INTERVAL '1' YEAR > INTERVAL '0' YEAR"
+      val query = s"SELECT a FROM t WHERE $condition"
+      withNullConjunctConfs(SQLConf.ANSI_ENABLED.key -> "false") { skipsAtNull 
=>
+        if (skipsAtNull) {
+          checkAnswer(sql(query), Row(0))
+        } else {
+          checkError(
+            exception = intercept[SparkArithmeticException] {
+              sql(query).collect()
+            },
+            condition = "INTERVAL_ARITHMETIC_OVERFLOW.WITH_SUGGESTION",
+            parameters = Map("functionName" -> toSQLId("try_add")))
+        }
+        // A check on an expression, not on a column, that precedes the NULL 
conjunct:
+        // `cast(s AS INT)` is NULL on the first row, so the check is FALSE 
and drops the row
+        // before the overflowing conjunct, as with codegen off.
+        checkAnswer(
+          sql(s"SELECT a FROM t WHERE cast(s AS INT) IS NOT NULL AND 
$condition"), Row(0))
+      }
+    }
+  }
+
+  test("SPARK-60315: a filter drops a row with a NULL conjunct and stops at a 
FALSE one") {
+    withNullConjunctTable(Int.MaxValue) {
+      withNullConjunctConfs(SQLConf.ANSI_ENABLED.key -> "true") { _ =>
+        // The first row is dropped after its last conjunct is evaluated.
+        checkAnswer(sql("SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y > 
-1"), Row(0))
+        // FALSE decides the AND, so the overflowing conjunct after it is not 
evaluated.
+        checkAnswer(
+          sql("SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND b < 0 AND y + 1 
> 0"),
+          Seq.empty)
+      }
+      def hasNullConjunctFlag(query: String): Boolean =
+        genCode(sql(query)).exists(_.body.contains("nullConjunct"))
+      // `b > 0` cannot be NULL after the inferred `isnotnull(b)`, so no flag 
is generated.
+      val noNullConjunct = "SELECT a FROM t WHERE b > 0 AND y > -1"
+      checkAnswer(sql(noNullConjunct), Seq(Row(null), Row(0)))
+      assert(!hasNullConjunctFlag(noNullConjunct))
+      // `a > 0 OR b > 0` can be NULL, but neither the conjuncts after it nor 
the inferred checks
+      // can raise an error, so the evaluation after a NULL conjunct is not 
observable and no
+      // flag is generated either: the generated code is the one of the legacy 
behavior.
+      val noRaise = "SELECT a FROM t WHERE (a > 0 OR b > 0) AND x = 1 AND y = 
0"
+      checkAnswer(sql(noRaise), Row(0))
+      assert(!hasNullConjunctFlag(noRaise))
+      // `y + 1 > 0` can raise, so the flag is generated.
+      val raise = "SELECT a FROM t WHERE (a > 0 OR b > 0) AND x = 1 AND y + 1 
> 0"
+      checkAnswer(sql(raise), Row(0))
+      assert(hasNullConjunctFlag(raise))
+    }
+  }
+
+  test("SPARK-60315: a filter checks IS NOT NULL before the conjuncts it 
precedes") {
+    withNullConjunctTable(Int.MaxValue) {
+      // `x` is NULL on the first row. With codegen off, a FALSE 
`isnotnull(x)` drops the row
+      // before a conjunct after it in the condition is evaluated. With 
codegen on, the check is
+      // generated just before the first conjunct that references `x`, or 
after all of them, so
+      // after a NULL conjunct it has to be generated before the conjuncts it 
precedes.
+      //
+      // The common `b * 2` takes the CSE path of FilterExec. The overflowing 
`y + 1` is not
+      // common because `GeneratePredicate` (codegen off) evaluates common 
subexpressions first.
+      val cseQuery = "SELECT a FROM t WHERE x IS NOT NULL AND NOT (a > 0 AND b 
> 0) AND " +
+        "y + 1 > 0 AND b * 2 > 0 AND b * 2 > -1"
+      def multiplyExactCount(filterExecCSE: Boolean): Int = {
+        withSQLConf(
+            SQLConf.ANSI_ENABLED.key -> "true",
+            SQLConf.SUBEXPRESSION_ELIMINATION_FILTER_EXEC_ENABLED.key -> 
filterExecCSE.toString) {
+          val code = codegenString(sql(cseQuery).queryExecution.executedPlan)
+          "multiplyExact".r.findAllIn(code).length
+        }
+      }
+      assert(multiplyExactCount(filterExecCSE = true) < 
multiplyExactCount(filterExecCSE = false))
+
+      Seq("true", "false").foreach { filterExecCSE =>
+        withNullConjunctConfs(
+            SQLConf.ANSI_ENABLED.key -> "true",
+            SQLConf.SUBEXPRESSION_ELIMINATION_FILTER_EXEC_ENABLED.key -> 
filterExecCSE) {
+          skipsAtNull =>
+            // `isnotnull(x)` is inferred from `x > 0` and precedes the 
overflowing conjunct.
+            checkAnswer(
+              sql("SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0 
AND x > 0"),
+              Row(0))
+            // A user-written check that no other conjunct references.
+            checkAnswer(
+              sql("SELECT a FROM t WHERE x IS NOT NULL AND NOT (a > 0 AND b > 
0) AND y + 1 > 0"),
+              Row(0))
+            // The same with a common subexpression.
+            checkAnswer(sql(cseQuery), Row(0))
+            // A check written after the overflowing conjunct stays after it.
+            val checkAfter =
+              "SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND y + 1 > 0 AND x 
IS NOT NULL"
+            if (skipsAtNull) {
+              checkAnswer(sql(checkAfter), Row(0))
+            } else {
+              checkArithmeticOverflow(checkAfter)
+            }
+            // The same through the CSE branch of FilterExec, which generates 
the pending checks
+            // before `y + 1 > 0`, the first conjunct after the NULL one that 
can raise, and not
+            // before the conjuncts after it.
+            val checkAfterCSE = checkAfter + " AND b * 2 > 0 AND b * 2 > -1"
+            if (skipsAtNull) {
+              checkAnswer(sql(checkAfterCSE), Row(0))
+            } else {
+              checkArithmeticOverflow(checkAfterCSE)
+            }
+        }
+      }
+      // The aggregate FILTER clause goes through the same code generation.
+      withNullConjunctConfs(SQLConf.ANSI_ENABLED.key -> "true") { _ =>
+        checkAnswer(
+          sql("SELECT count(1) FILTER (WHERE x IS NOT NULL AND NOT (a > 0 AND 
b > 0) AND " +
+            "y + 1 > 0) FROM t"),
+          Row(1))
+      }
+    }
+  }
+
+  test("SPARK-60315: a filter checks IS NOT NULL on an expression after the 
conjunct it follows") {
+    withNullConjunctTable(Int.MaxValue) {
+      // `cast(s AS INT)` raises on the first row. With codegen on, a check on 
an expression is
+      // generated just before the first conjunct that contains the expression 
(SPARK-51356),
+      // even when it is written after that conjunct. After a NULL conjunct, 
it has to stay after
+      // the conjunct, as with codegen off: `coalesce(b, cast(s AS INT)) < 0` 
is FALSE on the row
+      // and drops it before the check raises, while `coalesce(b, cast(s AS 
INT)) < 2` is TRUE and
+      // the check raises after it. The second row passes the latter condition.
+      //
+      // The common `b * 2` takes the CSE path of FilterExec.
+      def query(conjunct: String, cse: Boolean): String = {
+        val common = if (cse) "b * 2 > 0 AND b * 2 > -1 AND " else ""
+        s"SELECT a FROM t WHERE NOT (a > 0 AND b > 0) AND $common$conjunct AND 
" +
+          "cast(s AS INT) IS NOT NULL"
+      }
+      def multiplyExactCount(filterExecCSE: Boolean): Int = {
+        withSQLConf(
+            SQLConf.ANSI_ENABLED.key -> "true",
+            SQLConf.SUBEXPRESSION_ELIMINATION_FILTER_EXEC_ENABLED.key -> 
filterExecCSE.toString) {
+          val plan = sql(query("coalesce(b, cast(s AS INT)) < 0", cse = 
true)).queryExecution
+          "multiplyExact".r.findAllIn(codegenString(plan.executedPlan)).length
+        }
+      }
+      assert(multiplyExactCount(filterExecCSE = true) < 
multiplyExactCount(filterExecCSE = false))
+
+      Seq(true, false).foreach { cse =>
+        withNullConjunctConfs(
+            SQLConf.ANSI_ENABLED.key -> "true",
+            SQLConf.SUBEXPRESSION_ELIMINATION_FILTER_EXEC_ENABLED.key -> 
cse.toString) {
+          skipsAtNull =>
+            checkAnswer(sql(query("coalesce(b, cast(s AS INT)) < 0", cse)), 
Seq.empty)
+            val raising = query("coalesce(b, cast(s AS INT)) < 2", cse)
+            if (skipsAtNull) {
+              checkAnswer(sql(raising), Row(0))
+            } else {
+              checkCastInvalidInput(raising)
+            }
+        }
+      }
+      // The placement of the checks is decided at code generation, so once a 
conjunct can be
+      // NULL, the checks after it are evaluated in the order of the condition 
on every row. Here
+      // the check written first is generated before `y < 0` and raises on the 
first row, as with
+      // codegen off, although `a > 0 OR b > 0` is TRUE on the row. The legacy 
behavior evaluates
+      // `y < 0` first, which is FALSE on both rows, and never reaches the 
check.
+      val checkFirst =
+        "SELECT a FROM t WHERE cast(s AS INT) IS NOT NULL AND (a > 0 OR b > 0) 
AND y < 0"
+      withNullConjunctConfs(SQLConf.ANSI_ENABLED.key -> "true") { skipsAtNull 
=>
+        if (skipsAtNull) {
+          checkAnswer(sql(checkFirst), Seq.empty)
+        } else {
+          checkCastInvalidInput(checkFirst)
+        }
+      }
+    }
+  }
+
+  test("SPARK-60315: an aggregate FILTER evaluates the conjuncts after a NULL 
one") {
+    withNullConjunctTable(Int.MaxValue) {
+      val query = "SELECT count(a) FILTER (WHERE NOT (a > 0 AND b > 0) AND y + 
1 > 0) FROM t"
+      withNullConjunctConfs(SQLConf.ANSI_ENABLED.key -> "true") { skipsAtNull 
=>
+        if (skipsAtNull) {
+          checkAnswer(sql(query), Row(1))
+        } else {
+          checkArithmeticOverflow(query)
+        }
+      }
+    }
+  }
+
   test("SPARK-60403: full outer joins compile with their functions spilled to 
nested classes") {
     // A full outer join outputs its rows through a function of its own, and 
generates the code of
     // the operators above it, for the body of that function, before adding 
the function: here a


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to