This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 736339b29890 [SPARK-60885][SS] Fix wrong results by changing 
`avoidDoubleFilterEval` across streaming query restarts
736339b29890 is described below

commit 736339b298907c86e517963ce201c6fa7a77d8b5
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 13:43:09 2026 -0800

    [SPARK-60885][SS] Fix wrong results by changing `avoidDoubleFilterEval` 
across streaming query restarts
    
    ### What changes were proposed in this pull request?
    
    This PR fixes a correctness issue. A streaming query can return rows that 
do not pass its filter when `spark.sql.optimizer.avoidDoubleFilterEval` changes 
across a restart from the same checkpoint.
    
    `spark.sql.optimizer.avoidDoubleFilterEval` was added by SPARK-47672 in 
Spark 4.2.0, with default `true`. It decides whether 
`PushPredicateThroughNonJoin` keeps a filter on an expensive alias, such as a 
UDF, above its `Project`. With `false`, the filter is pushed below the 
`Project` and can move further below a stateful operator such as a 
stream-stream join or an aggregation. The rows already in the state then skip 
the filter. The conf was not recorded in the offset log, so each resta [...]
    
    This PR:
    - adds `AVOID_DOUBLE_FILTER_EVAL` to `OffsetSeqMetadata.relevantSQLConfs`, 
so a query keeps the value recorded in its checkpoint.
    - adds `defaultValueForMissingConf` for checkpoints without the key. A 
missing key has two meanings, and `ENABLE_STREAMING_SOURCE_EVOLUTION`, which is 
recorded since Spark 4.2, tells them apart.
    
    | Checkpoint written by | Value used |
    |---|---|
    | Spark 4.1 or before | `false`, because the optimizer always behaved like 
`false` before Spark 4.2 |
    | Spark 4.2.x or 4.3.x | the session value at the first restart, which is 
then recorded |
    | Spark with this change | the recorded value |
    
    No static entry in `relevantSQLConfDefaultValues` works:
    - `false` would move the filters of 4.2.x and 4.3.x checkpoints under their 
stateful operators, which causes Issue 1 below.
    - `true` would change the plan of checkpoints from Spark 4.1 or before, 
which causes Issue 2 below, and the session could no longer override it.
    - With no default, a failed first restart writes the session value into the 
uncommitted offset entry, so setting `false` afterwards no longer recovers the 
query.
    
    SPARK-60632 records `spark.sql.optimizer.regexPatternCostEstimateVersion` 
for a similar reason. That key is new, so a missing key has only one meaning 
and a static default is enough there.
    
    ### Why are the changes needed?
    
    Issue 1 is a correctness issue. A query restarted with a different value 
returns rows that do not pass its filter.
    
    ```scala
    val left = MemoryStream[(Int, Int)]
    val right = MemoryStream[(Int, Int)]
    val expensive = udf((value: Int) => value)
    val df = left.toDF().toDF("key", "leftValue")
      .join(right.toDF().toDF("key", "rightValue"), "key")
      .select($"key", expensive($"leftValue").as("result"), $"rightValue")
      .where($"result" > 0)
    
    // Run 1 with spark.sql.optimizer.avoidDoubleFilterEval=true (the default).
    left.addData((1, -1), (2, 10))
    
    // Run 2 from the same checkpoint with 
spark.sql.optimizer.avoidDoubleFilterEval=false.
    right.addData((1, 100), (2, 200))
    // Issue 1: It should return [2,10,200], but it also returns [1,-1,100], 
whose result is -1.
    ```
    
    The same happens for a left outer join, a left semi join, and a complete or 
append mode aggregation under a projection that `CollapseProject` cannot merge.
    
    Issue 2: When only the filter uses a column, the state schema differs 
between the two plans, so the restart fails with 
`STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE`. This happens in both directions. It 
also happens without any conf change when a checkpoint of Spark 4.1 or before 
is restarted on Spark 4.2.0 or 4.3.0, because the older versions always planned 
the query like `false`. Before this change, the only workaround was to set 
`false` by hand.
    
    ```scala
    // Same as above, but only the filter uses leftValue.
    val df2 = df.select($"rightValue")
    ```
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. This fixes wrong results and restart failures of streaming queries 
since Spark [4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0) 
(2026-07-11), including 
[4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27).
    
    - A streaming query keeps the value of 
`spark.sql.optimizer.avoidDoubleFilterEval` recorded in its checkpoint. Setting 
another value in the session only logs a warning. Changing the value requires a 
new checkpoint.
    - A checkpoint written by Spark 4.1 or before uses `false`, so its plan 
stays as it was before the upgrade.
    - A checkpoint written by Spark 4.2.x or 4.3.x does not record which value 
it ran with, so its first restart records the session value. That restart 
should use the same value as before.
    
    ### How was this patch tested?
    
    New tests:
    - `OffsetSeqLogSuite`:
      - A checkpoint written before Spark 4.2 reads `false`, which overrides 
the session value.
      - A checkpoint written by Spark 4.2.0 keeps the session value.
    - `StreamingQueryManagerSuite`: a stream-stream join started with `true` 
and restarted with `false` keeps `true` and returns only the rows that pass the 
filter.
    
    Without this change, the `StreamingQueryManagerSuite` test and the first 
`OffsetSeqLogSuite` test fail. Each of the following was then changed alone, 
and each change made at least one test fail:
    - the new entry in `relevantSQLConfs`
    - each branch of `defaultValueForMissingConf`
    - its two call sites
    - a static default `true`
    
    Checkpoints written by the released Spark 4.1.3, 4.2.0 and 4.3.0 were also 
restarted with this change:
    - A Spark 4.1.3 checkpoint of a query with a filter-only column restarts 
with either session value. Spark 4.2.0 and 4.3.0 fail to restart it with 
`STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE`.
    - For Spark 4.2.0 and 4.3.0 checkpoints, the first restart records the 
session value and later restarts keep it.
    
    Existing tests:
    - `sql/testOnly *OffsetSeqLogSuite *StreamingQueryManagerSuite`
    - `sql/testOnly *StreamSuite *StreamingQuerySuite 
*StreamingAggregationSuite *StreamingDeduplicationSuite 
*StreamingInnerJoinWithVCFSuite *StreamingOuterJoinWithVCFSuite 
*StreamingLeftSemiJoinWithVCFSuite *EventTimeWatermarkSuite 
*StateStoreCompatibilitySuite *StateDataSourceReadSuite 
*ProtobufExtensionsSupportOffsetLogSuite`
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60102 from dongjoon-hyun/SPARK-60885.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../streaming/checkpointing/OffsetSeq.scala        | 31 +++++++++++++++++++---
 .../execution/streaming/OffsetSeqLogSuite.scala    | 27 +++++++++++++++++++
 .../sql/streaming/StreamingQueryManagerSuite.scala | 29 ++++++++++++++++++++
 3 files changed, 83 insertions(+), 4 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
index b4947e04422f..2a79da4332cf 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
@@ -208,7 +208,8 @@ object OffsetSeqMetadata extends Logging {
     STATEFUL_OPERATOR_ALWAYS_NULLABLE_OUTPUT,
     DROP_DUPLICATES_DETERMINISTIC_KEY_ORDER,
     ALLOW_EXCEPT_ON_STREAMING_DATAFRAME,
-    REGEX_PATTERN_COST_ESTIMATE_VERSION
+    REGEX_PATTERN_COST_ESTIMATE_VERSION,
+    AVOID_DOUBLE_FILTER_EVAL
   )
 
   /**
@@ -285,6 +286,27 @@ object OffsetSeqMetadata extends Logging {
     REGEX_PATTERN_COST_ESTIMATE_VERSION.key -> "0"
   )
 
+  /**
+   * Returns the value to use for a relevant configuration missing in the 
given metadata, if any.
+   *
+   * `AVOID_DOUBLE_FILTER_EVAL` cannot be in `relevantSQLConfDefaultValues` 
because its absence has
+   * two meanings. Before Spark 4.2 the behavior was always that of `false`. 
Spark 4.2 and 4.3 used
+   * the session value, `true` by default, without recording it. Forcing 
`false` on the latter could
+   * move a filter under a stateful operator, so that the rows already in the 
state skip it. Forcing
+   * `true` on the former could change the state schema. They are told apart by
+   * `ENABLE_STREAMING_SOURCE_EVOLUTION`, which is recorded since Spark 4.2.
+   */
+  private def defaultValueForMissingConf(
+      metadata: OffsetSeqMetadataBase,
+      confKey: String): Option[String] = {
+    if (confKey == AVOID_DOUBLE_FILTER_EVAL.key &&
+        !metadata.conf.contains(ENABLE_STREAMING_SOURCE_EVOLUTION.key)) {
+      Some("false")
+    } else {
+      relevantSQLConfDefaultValues.get(confKey)
+    }
+  }
+
   def readValue[T](metadataLog: OffsetSeqMetadataBase, confKey: 
ConfigEntry[T]): String = {
     readValueOpt(metadataLog, confKey).getOrElse(confKey.defaultValueString)
   }
@@ -296,7 +318,8 @@ object OffsetSeqMetadata extends Logging {
       rebindSQLConfsSessionToOffsetLog(confKey)
     } else confKey
 
-    
metadataLog.conf.get(actualKey.key).orElse(relevantSQLConfDefaultValues.get(actualKey.key))
+    metadataLog.conf.get(actualKey.key)
+      .orElse(defaultValueForMissingConf(metadataLog, actualKey.key))
   }
 
   def apply(json: String): OffsetSeqMetadata = 
Serialization.read[OffsetSeqMetadata](json)
@@ -337,9 +360,9 @@ object OffsetSeqMetadata extends Logging {
 
         case None =>
           // For backward compatibility, if a config was not recorded in the 
offset log,
-          // then either inject a default value (if specified in 
`relevantSQLConfDefaultValues`) or
+          // then either inject a default value (if 
`defaultValueForMissingConf` returns one) or
           // let the existing conf value in SparkSession prevail.
-          relevantSQLConfDefaultValues.get(confKeyInOffsetLog) match {
+          defaultValueForMissingConf(metadata, confKeyInOffsetLog) match {
 
             case Some(defaultValue) =>
               sessionConf.setConfString(confKeyInSession, defaultValue)
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
index 0a31aba4aa8f..43eba51292a3 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
@@ -66,6 +66,33 @@ class OffsetSeqLogSuite extends SharedSparkSession {
     
assert(metadata.conf.get(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION.key).contains("1"))
   }
 
+  test("SPARK-60885: avoidDoubleFilterEval defaults to false for checkpoints 
written " +
+      "before Spark 4.2") {
+    val key = SQLConf.AVOID_DOUBLE_FILTER_EVAL
+    val (_, offsetSeq) = readFromResource("offset-log-version-2.1.0")
+    val metadata = offsetSeq.metadataOpt.get
+    assert(OffsetSeqMetadata.readValue(metadata, key) === "false")
+
+    val sessionConf = new SQLConf
+    sessionConf.setConfString(key.key, "true")
+    OffsetSeqMetadata.setSessionConf(metadata, sessionConf)
+    assert(sessionConf.getConfString(key.key) === "false")
+  }
+
+  test("SPARK-60885: avoidDoubleFilterEval keeps the session value for 
checkpoints of " +
+      "Spark 4.2 and 4.3") {
+    val key = SQLConf.AVOID_DOUBLE_FILTER_EVAL
+    val (_, offsetSeq) = readFromResource(
+      "checkpoint-version-4.2.0-deduplication-metadata-boundary/offsets")
+    val metadata = offsetSeq.metadataOpt.get
+    assert(OffsetSeqMetadata.readValueOpt(metadata, key).isEmpty)
+
+    val sessionConf = new SQLConf
+    sessionConf.setConfString(key.key, "true")
+    OffsetSeqMetadata.setSessionConf(metadata, sessionConf)
+    assert(sessionConf.getConfString(key.key) === "true")
+  }
+
   test("SPARK-59919: v1 and v2 metadata persist rebound stateful shuffle 
partitions") {
     withSQLConf(
       SQLConf.SHUFFLE_PARTITIONS.key -> "10",
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
index da010e3c4ec5..0c067bddb35c 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
@@ -37,6 +37,7 @@ import 
org.apache.spark.sql.execution.datasources.v2.StreamingDataSourceV2ScanRe
 import org.apache.spark.sql.execution.streaming.checkpointing.{OffsetMap, 
OffsetSeq, OffsetSeqLog,
   OffsetSeqMetadata, OffsetSeqMetadataV2}
 import org.apache.spark.sql.execution.streaming.runtime._
+import org.apache.spark.sql.functions.udf
 import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.streaming.util.BlockingSource
 import org.apache.spark.tags.SlowSQLTest
@@ -274,6 +275,34 @@ class StreamingQueryManagerSuite extends StreamTest {
     }
   }
 
+  test("SPARK-60885: avoidDoubleFilterEval is read from the offset log") {
+    val key = SQLConf.AVOID_DOUBLE_FILTER_EVAL.key
+    val left = MemoryStream[(Int, Int)]
+    val right = MemoryStream[(Int, Int)]
+    val expensive = udf((value: Int) => value)
+    // With the conf on, the filter stays above the join and the left state 
also keeps the rows
+    // failing it. Turning it off would push the filter under the join, so 
those rows would join.
+    val df = left.toDF().toDF("key", "leftValue")
+      .join(right.toDF().toDF("key", "rightValue"), "key")
+      .select($"key", expensive($"leftValue").as("result"), $"rightValue")
+      .where($"result" > 0)
+
+    withTempDir { dir =>
+      withSQLConf(key -> "true", SQLConf.SHUFFLE_PARTITIONS.key -> "1") {
+        testStream(df)(
+          StartStream(checkpointLocation = dir.getCanonicalPath),
+          AddData(left, (1, -1), (2, 10)),
+          CheckNewAnswer(),
+          StopStream,
+          StartStream(
+            checkpointLocation = dir.getCanonicalPath,
+            additionalConfs = Map(key -> "false")),
+          AddData(right, (1, 100), (2, 200)),
+          CheckNewAnswer((2, 10, 200)))
+      }
+    }
+  }
+
   testQuietly("listing") {
     val (m1, ds1) = makeDataset
     val (m2, ds2) = makeDataset


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to