This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 736339b29890 [SPARK-60885][SS] Fix wrong results by changing
`avoidDoubleFilterEval` across streaming query restarts
736339b29890 is described below
commit 736339b298907c86e517963ce201c6fa7a77d8b5
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Nov 4 13:43:09 2026 -0800
[SPARK-60885][SS] Fix wrong results by changing `avoidDoubleFilterEval`
across streaming query restarts
### What changes were proposed in this pull request?
This PR fixes a correctness issue. A streaming query can return rows that
do not pass its filter when `spark.sql.optimizer.avoidDoubleFilterEval` changes
across a restart from the same checkpoint.
`spark.sql.optimizer.avoidDoubleFilterEval` was added by SPARK-47672 in
Spark 4.2.0, with default `true`. It decides whether
`PushPredicateThroughNonJoin` keeps a filter on an expensive alias, such as a
UDF, above its `Project`. With `false`, the filter is pushed below the
`Project` and can move further below a stateful operator such as a
stream-stream join or an aggregation. The rows already in the state then skip
the filter. The conf was not recorded in the offset log, so each resta [...]
This PR:
- adds `AVOID_DOUBLE_FILTER_EVAL` to `OffsetSeqMetadata.relevantSQLConfs`,
so a query keeps the value recorded in its checkpoint.
- adds `defaultValueForMissingConf` for checkpoints without the key. A
missing key has two meanings, and `ENABLE_STREAMING_SOURCE_EVOLUTION`, which is
recorded since Spark 4.2, tells them apart.
| Checkpoint written by | Value used |
|---|---|
| Spark 4.1 or before | `false`, because the optimizer always behaved like
`false` before Spark 4.2 |
| Spark 4.2.x or 4.3.x | the session value at the first restart, which is
then recorded |
| Spark with this change | the recorded value |
No static entry in `relevantSQLConfDefaultValues` works:
- `false` would move the filters of 4.2.x and 4.3.x checkpoints under their
stateful operators, which causes Issue 1 below.
- `true` would change the plan of checkpoints from Spark 4.1 or before,
which causes Issue 2 below, and the session could no longer override it.
- With no default, a failed first restart writes the session value into the
uncommitted offset entry, so setting `false` afterwards no longer recovers the
query.
SPARK-60632 records `spark.sql.optimizer.regexPatternCostEstimateVersion`
for a similar reason. That key is new, so a missing key has only one meaning
and a static default is enough there.
### Why are the changes needed?
Issue 1 is a correctness issue. A query restarted with a different value
returns rows that do not pass its filter.
```scala
val left = MemoryStream[(Int, Int)]
val right = MemoryStream[(Int, Int)]
val expensive = udf((value: Int) => value)
val df = left.toDF().toDF("key", "leftValue")
.join(right.toDF().toDF("key", "rightValue"), "key")
.select($"key", expensive($"leftValue").as("result"), $"rightValue")
.where($"result" > 0)
// Run 1 with spark.sql.optimizer.avoidDoubleFilterEval=true (the default).
left.addData((1, -1), (2, 10))
// Run 2 from the same checkpoint with
spark.sql.optimizer.avoidDoubleFilterEval=false.
right.addData((1, 100), (2, 200))
// Issue 1: It should return [2,10,200], but it also returns [1,-1,100],
whose result is -1.
```
The same happens for a left outer join, a left semi join, and a complete or
append mode aggregation under a projection that `CollapseProject` cannot merge.
Issue 2: When only the filter uses a column, the state schema differs
between the two plans, so the restart fails with
`STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE`. This happens in both directions. It
also happens without any conf change when a checkpoint of Spark 4.1 or before
is restarted on Spark 4.2.0 or 4.3.0, because the older versions always planned
the query like `false`. Before this change, the only workaround was to set
`false` by hand.
```scala
// Same as above, but only the filter uses leftValue.
val df2 = df.select($"rightValue")
```
### Does this PR introduce _any_ user-facing change?
Yes. This fixes wrong results and restart failures of streaming queries
since Spark [4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0)
(2026-07-11), including
[4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27).
- A streaming query keeps the value of
`spark.sql.optimizer.avoidDoubleFilterEval` recorded in its checkpoint. Setting
another value in the session only logs a warning. Changing the value requires a
new checkpoint.
- A checkpoint written by Spark 4.1 or before uses `false`, so its plan
stays as it was before the upgrade.
- A checkpoint written by Spark 4.2.x or 4.3.x does not record which value
it ran with, so its first restart records the session value. That restart
should use the same value as before.
### How was this patch tested?
New tests:
- `OffsetSeqLogSuite`:
- A checkpoint written before Spark 4.2 reads `false`, which overrides
the session value.
- A checkpoint written by Spark 4.2.0 keeps the session value.
- `StreamingQueryManagerSuite`: a stream-stream join started with `true`
and restarted with `false` keeps `true` and returns only the rows that pass the
filter.
Without this change, the `StreamingQueryManagerSuite` test and the first
`OffsetSeqLogSuite` test fail. Each of the following was then changed alone,
and each change made at least one test fail:
- the new entry in `relevantSQLConfs`
- each branch of `defaultValueForMissingConf`
- its two call sites
- a static default `true`
Checkpoints written by the released Spark 4.1.3, 4.2.0 and 4.3.0 were also
restarted with this change:
- A Spark 4.1.3 checkpoint of a query with a filter-only column restarts
with either session value. Spark 4.2.0 and 4.3.0 fail to restart it with
`STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE`.
- For Spark 4.2.0 and 4.3.0 checkpoints, the first restart records the
session value and later restarts keep it.
Existing tests:
- `sql/testOnly *OffsetSeqLogSuite *StreamingQueryManagerSuite`
- `sql/testOnly *StreamSuite *StreamingQuerySuite
*StreamingAggregationSuite *StreamingDeduplicationSuite
*StreamingInnerJoinWithVCFSuite *StreamingOuterJoinWithVCFSuite
*StreamingLeftSemiJoinWithVCFSuite *EventTimeWatermarkSuite
*StateStoreCompatibilitySuite *StateDataSourceReadSuite
*ProtobufExtensionsSupportOffsetLogSuite`
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60102 from dongjoon-hyun/SPARK-60885.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../streaming/checkpointing/OffsetSeq.scala | 31 +++++++++++++++++++---
.../execution/streaming/OffsetSeqLogSuite.scala | 27 +++++++++++++++++++
.../sql/streaming/StreamingQueryManagerSuite.scala | 29 ++++++++++++++++++++
3 files changed, 83 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
index b4947e04422f..2a79da4332cf 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
@@ -208,7 +208,8 @@ object OffsetSeqMetadata extends Logging {
STATEFUL_OPERATOR_ALWAYS_NULLABLE_OUTPUT,
DROP_DUPLICATES_DETERMINISTIC_KEY_ORDER,
ALLOW_EXCEPT_ON_STREAMING_DATAFRAME,
- REGEX_PATTERN_COST_ESTIMATE_VERSION
+ REGEX_PATTERN_COST_ESTIMATE_VERSION,
+ AVOID_DOUBLE_FILTER_EVAL
)
/**
@@ -285,6 +286,27 @@ object OffsetSeqMetadata extends Logging {
REGEX_PATTERN_COST_ESTIMATE_VERSION.key -> "0"
)
+ /**
+ * Returns the value to use for a relevant configuration missing in the
given metadata, if any.
+ *
+ * `AVOID_DOUBLE_FILTER_EVAL` cannot be in `relevantSQLConfDefaultValues`
because its absence has
+ * two meanings. Before Spark 4.2 the behavior was always that of `false`.
Spark 4.2 and 4.3 used
+ * the session value, `true` by default, without recording it. Forcing
`false` on the latter could
+ * move a filter under a stateful operator, so that the rows already in the
state skip it. Forcing
+ * `true` on the former could change the state schema. They are told apart by
+ * `ENABLE_STREAMING_SOURCE_EVOLUTION`, which is recorded since Spark 4.2.
+ */
+ private def defaultValueForMissingConf(
+ metadata: OffsetSeqMetadataBase,
+ confKey: String): Option[String] = {
+ if (confKey == AVOID_DOUBLE_FILTER_EVAL.key &&
+ !metadata.conf.contains(ENABLE_STREAMING_SOURCE_EVOLUTION.key)) {
+ Some("false")
+ } else {
+ relevantSQLConfDefaultValues.get(confKey)
+ }
+ }
+
def readValue[T](metadataLog: OffsetSeqMetadataBase, confKey:
ConfigEntry[T]): String = {
readValueOpt(metadataLog, confKey).getOrElse(confKey.defaultValueString)
}
@@ -296,7 +318,8 @@ object OffsetSeqMetadata extends Logging {
rebindSQLConfsSessionToOffsetLog(confKey)
} else confKey
-
metadataLog.conf.get(actualKey.key).orElse(relevantSQLConfDefaultValues.get(actualKey.key))
+ metadataLog.conf.get(actualKey.key)
+ .orElse(defaultValueForMissingConf(metadataLog, actualKey.key))
}
def apply(json: String): OffsetSeqMetadata =
Serialization.read[OffsetSeqMetadata](json)
@@ -337,9 +360,9 @@ object OffsetSeqMetadata extends Logging {
case None =>
// For backward compatibility, if a config was not recorded in the
offset log,
- // then either inject a default value (if specified in
`relevantSQLConfDefaultValues`) or
+ // then either inject a default value (if
`defaultValueForMissingConf` returns one) or
// let the existing conf value in SparkSession prevail.
- relevantSQLConfDefaultValues.get(confKeyInOffsetLog) match {
+ defaultValueForMissingConf(metadata, confKeyInOffsetLog) match {
case Some(defaultValue) =>
sessionConf.setConfString(confKeyInSession, defaultValue)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
index 0a31aba4aa8f..43eba51292a3 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
@@ -66,6 +66,33 @@ class OffsetSeqLogSuite extends SharedSparkSession {
assert(metadata.conf.get(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION.key).contains("1"))
}
+ test("SPARK-60885: avoidDoubleFilterEval defaults to false for checkpoints
written " +
+ "before Spark 4.2") {
+ val key = SQLConf.AVOID_DOUBLE_FILTER_EVAL
+ val (_, offsetSeq) = readFromResource("offset-log-version-2.1.0")
+ val metadata = offsetSeq.metadataOpt.get
+ assert(OffsetSeqMetadata.readValue(metadata, key) === "false")
+
+ val sessionConf = new SQLConf
+ sessionConf.setConfString(key.key, "true")
+ OffsetSeqMetadata.setSessionConf(metadata, sessionConf)
+ assert(sessionConf.getConfString(key.key) === "false")
+ }
+
+ test("SPARK-60885: avoidDoubleFilterEval keeps the session value for
checkpoints of " +
+ "Spark 4.2 and 4.3") {
+ val key = SQLConf.AVOID_DOUBLE_FILTER_EVAL
+ val (_, offsetSeq) = readFromResource(
+ "checkpoint-version-4.2.0-deduplication-metadata-boundary/offsets")
+ val metadata = offsetSeq.metadataOpt.get
+ assert(OffsetSeqMetadata.readValueOpt(metadata, key).isEmpty)
+
+ val sessionConf = new SQLConf
+ sessionConf.setConfString(key.key, "true")
+ OffsetSeqMetadata.setSessionConf(metadata, sessionConf)
+ assert(sessionConf.getConfString(key.key) === "true")
+ }
+
test("SPARK-59919: v1 and v2 metadata persist rebound stateful shuffle
partitions") {
withSQLConf(
SQLConf.SHUFFLE_PARTITIONS.key -> "10",
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
index da010e3c4ec5..0c067bddb35c 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryManagerSuite.scala
@@ -37,6 +37,7 @@ import
org.apache.spark.sql.execution.datasources.v2.StreamingDataSourceV2ScanRe
import org.apache.spark.sql.execution.streaming.checkpointing.{OffsetMap,
OffsetSeq, OffsetSeqLog,
OffsetSeqMetadata, OffsetSeqMetadataV2}
import org.apache.spark.sql.execution.streaming.runtime._
+import org.apache.spark.sql.functions.udf
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.streaming.util.BlockingSource
import org.apache.spark.tags.SlowSQLTest
@@ -274,6 +275,34 @@ class StreamingQueryManagerSuite extends StreamTest {
}
}
+ test("SPARK-60885: avoidDoubleFilterEval is read from the offset log") {
+ val key = SQLConf.AVOID_DOUBLE_FILTER_EVAL.key
+ val left = MemoryStream[(Int, Int)]
+ val right = MemoryStream[(Int, Int)]
+ val expensive = udf((value: Int) => value)
+ // With the conf on, the filter stays above the join and the left state
also keeps the rows
+ // failing it. Turning it off would push the filter under the join, so
those rows would join.
+ val df = left.toDF().toDF("key", "leftValue")
+ .join(right.toDF().toDF("key", "rightValue"), "key")
+ .select($"key", expensive($"leftValue").as("result"), $"rightValue")
+ .where($"result" > 0)
+
+ withTempDir { dir =>
+ withSQLConf(key -> "true", SQLConf.SHUFFLE_PARTITIONS.key -> "1") {
+ testStream(df)(
+ StartStream(checkpointLocation = dir.getCanonicalPath),
+ AddData(left, (1, -1), (2, 10)),
+ CheckNewAnswer(),
+ StopStream,
+ StartStream(
+ checkpointLocation = dir.getCanonicalPath,
+ additionalConfs = Map(key -> "false")),
+ AddData(right, (1, 100), (2, 200)),
+ CheckNewAnswer((2, 10, 200)))
+ }
+ }
+ }
+
testQuietly("listing") {
val (m1, ds1) = makeDataset
val (m2, ds2) = makeDataset
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]