This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new cba30a77a059 [SPARK-60905][SS] Fix streaming query restarts from
checkpoints before Spark 4.4 when a filter uses a Python UDF
cba30a77a059 is described below
commit cba30a77a059e2292ec0d0f6dfc39005f36bf296
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Nov 5 10:37:32 2026 -0800
[SPARK-60905][SS] Fix streaming query restarts from checkpoints before
Spark 4.4 when a filter uses a Python UDF
### What changes were proposed in this pull request?
This PR fixes restarts of streaming queries from checkpoints written by
Spark 4.3.x or before, when a filter uses the result of a Python UDF.
SPARK-59693 made every Python UDF (`PythonFuncExpression`) expensive. With
`spark.sql.optimizer.avoidDoubleFilterEval=true` (the default),
`PushPredicateThroughNonJoin` now keeps a filter on a Python UDF alias above
its `Project`. Before, the filter was pushed below the `Project`. From there it
could move further below a stateful operator, such as into the input of a
stream-stream join, or below `EventTimeWatermark`. The checkpoint does not
record which classification a query ran with [...]
This PR:
- adds an internal config,
`spark.sql.optimizer.avoidDoubleFilterEvalForPythonUDF` (default `true`). With
`false`, a Python UDF is only as expensive as its arguments, as in Spark 4.3
and before.
- adds it to `OffsetSeqMetadata.relevantSQLConfs`, with `false` in
`relevantSQLConfDefaultValues` for checkpoints without the key. Every released
version treats Python UDFs as cheap, so a missing key has one meaning and a
static default is enough.
- checks it in `PushPredicateThroughNonJoin.isExpensive` rather than in
`PythonFuncExpression.expensive`, because a parent expression caches
`hasExpensiveChild`. SPARK-60632 does the same for
`spark.sql.optimizer.regexPatternCostEstimateVersion`. With `false`, it
classifies a copy of the expression in which each Python UDF is replaced by
`PrettyPythonUDF`. That expression is not foldable either, and it is expensive
only through its arguments. The replacement can only make an expressio [...]
- does not apply it to the sharing analysis of the projection split
(`splitProjectForExpensiveConditions`). There a Python UDF still counts as
expensive, so two aliases that share one are computed in one layer and do not
run it twice. The split only adds layers above the child of the `Project`, so a
restart does not depend on it.
| Checkpoint written by | Value used |
|---|---|
| Spark 4.3.x or before | `false`, so the filter stays where those versions
put it |
| Spark with this change | the recorded value, `true` by default |
This is a separate config rather than a new version of
`spark.sql.optimizer.regexPatternCostEstimateVersion`. That config is about
LIKE and RLIKE patterns, and its versions are shared with `branch-4.2` and
`branch-4.3`, which do not have SPARK-59693.
### Why are the changes needed?
Without this change, a query that ran on Spark 4.3.0 fails or returns
different rows after an upgrade to the unreleased `master` or `branch-4.x`. No
config is changed in either case.
Issue 1: When only the filter uses the input column of the UDF, the state
schema of the stream-stream join differs between the two plans, so the restart
fails.
```python
from pyspark.sql.functions import col, udf
udf("int")
def identity(v):
return v
left = spark.readStream.schema("k INT, lv INT").json(left_path)
right = spark.readStream.schema("k INT, rv INT").json(right_path)
df = (left.join(right, "k")
.select(identity(col("lv")).alias("u"), "rv")
.where(col("u") > 0)
.select("rv"))
# Run 1 on Spark 4.3.0 with the left rows (1, -1) and (2, 10).
# Run 2 on master from the same checkpoint with the right rows (1, 100) and
(2, 200).
# Issue 1: It should return [200], but it fails with
STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE.
```
Issue 2: A filter over `withWatermark` stays above the watermark after the
restart. Rows that fail the filter then advance the watermark, so later rows
can be dropped as late.
```python
from pyspark.sql.functions import window
events = spark.readStream.schema("t TIMESTAMP, v INT").json(events_path)
df = (events.withWatermark("t", "10 seconds")
.select("t", identity(col("v")).alias("u"))
.where(col("u") > 0)
.groupBy(window("t", "10 seconds"))
.count())
# Run 1 on Spark 4.3.0 with (00:01:40, 1).
# Run 2 on master with (00:16:40, -1), then run 3 with (00:01:45, 1) and
(00:33:20, 1).
# Issue 2: It should return the window [00:01:40, 00:01:50) with count 2,
but it returns count 1.
```
Setting `spark.sql.optimizer.avoidDoubleFilterEval=false` works around both
issues. However, it also moves the filters on other expensive expressions, such
as Scala UDFs. In addition, since SPARK-60885 a failed first restart records
`true`, so setting `false` afterwards no longer recovers the query.
### Does this PR introduce _any_ user-facing change?
No, compared to the released versions. A query restarted from a checkpoint
of Spark [4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0)
(2026-10-27) or before keeps the plan of that version.
Within the unreleased `master` and `branch-4.x`, yes:
- A query restarted from a checkpoint of Spark 4.3.x or before no longer
treats Python UDFs as expensive. The value is recorded again in each batch, so
such a query needs a new checkpoint to benefit from SPARK-59693.
- A new query still treats them as expensive.
- A checkpoint written by a snapshot build after SPARK-59693 has no key, so
it now reads `false`.
### How was this patch tested?
New tests:
- `FilterPushdownSuite`: with `false`, a filter on a `pythonUDF(a) + 1`
alias is pushed below the `Project`, and a filter on a Python UDF over an
expensive RLIKE stays above it. The plan is optimized with the default first,
so the test also covers the cached `hasExpensiveChild`.
- `OffsetSeqLogSuite`: the offset log of a Spark 4.2.0 checkpoint, which
has no key and keeps `spark.sql.optimizer.avoidDoubleFilterEval=true`, reads
`false`. A new checkpoint records `true`.
Existing tests: `catalyst/testOnly *FilterPushdownSuite` and `sql/testOnly
*OffsetSeqLogSuite`.
The test suite has no checkpoints written by released Spark versions, so
the issues were also checked manually with PySpark (file source,
`Trigger.AvailableNow`). Run 1 used the released Spark 4.3.0. Later runs used a
build of `master` without this PR and a build with it, from the same
checkpoint. `S1` is Issue 1 with the UDF result in the output, which should not
change.
| Query | 4.3.0 -> 4.3.0 | 4.3.0 -> master | 4.3.0 -> this PR | this PR ->
this PR |
|---|---|---|---|---|
| S1 | `[2,10,200]` | `[2,10,200]` | `[2,10,200]` | not run |
| Issue 1 | `[200]` | `STATE_STORE_VALUE_SCHEMA_NOT_COMPATIBLE` | `[200]` |
`[200]` |
| Issue 2 | count 2 | count 1 | count 2 | count 1 |
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60117 from dongjoon-hyun/SPARK-60905.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit a11d7fc8f93d09bac7087604d6de1457182e0ddb)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../spark/sql/catalyst/optimizer/Optimizer.scala | 28 +++++++++++++++++-----
.../org/apache/spark/sql/internal/SQLConf.scala | 12 ++++++++++
.../catalyst/optimizer/FilterPushdownSuite.scala | 26 ++++++++++++++++++++
.../streaming/checkpointing/OffsetSeq.scala | 6 +++--
.../execution/streaming/OffsetSeqLogSuite.scala | 9 +++++++
5 files changed, 73 insertions(+), 8 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
index 665b3f94e030..6c751bc09af0 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala
@@ -2928,7 +2928,10 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
// Expensive sub-expressions behind each expensive alias, to spot when two
aliases are the
// same piece of work. Cheap aliases have none, so we skip the (possibly
wide) cheap rest.
// Two aliases share work exactly when they share a regex, so version 0 of
the estimate
- // collects the regex expressions and not their ancestors.
+ // collects the regex expressions and not their ancestors. A Python UDF
counts here even with
+ // `avoidDoubleFilterEvalForPythonUDF` off, so two aliases sharing one
stay in one layer and
+ // do not run it twice. The split only stacks layers above `baseChild`, so
a restart from a
+ // checkpoint does not depend on it.
val regexIsExpensive =
conf.getConf(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION) == 0
val expensivePartsOf = AttributeMap(expensiveAliases.toSeq.map { attr =>
attr -> aliasMap(attr).child.collect {
@@ -3000,12 +3003,25 @@ object PushPredicateThroughNonJoin extends
Rule[LogicalPlan] with PredicateHelpe
/**
* Whether `e` is expensive. Version 0 of the regex pattern cost estimate
treats every LIKE and
- * RLIKE as expensive. It is checked here rather than in
`StringRegexExpression.expensive`,
- * because `hasExpensiveChild` caches the answer of a parent expression.
+ * RLIKE as expensive, and with `avoidDoubleFilterEvalForPythonUDF` off a
Python UDF is only as
+ * expensive as its arguments. Both are checked here rather than in
`expensive`, because
+ * `hasExpensiveChild` caches the answer of a parent expression.
*/
- private def isExpensive(e: Expression): Boolean = e.expensive ||
- (conf.getConf(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION) == 0 &&
- e.exists(_.isInstanceOf[StringRegexExpression]))
+ private def isExpensive(e: Expression): Boolean = {
+ // The stand-in below can only make `e` cheaper, so a cheap `e` needs no
rewrite.
+ val classified =
+ if (!e.expensive ||
conf.getConf(SQLConf.AVOID_DOUBLE_FILTER_EVAL_FOR_PYTHON_UDF)) {
+ e
+ } else {
+ // The stand-in is not foldable either, so a LIKE/RLIKE pattern from a
UDF stays expensive.
+ e.transformUp {
+ case p: PythonFuncExpression => PrettyPythonUDF(p.name, p.dataType,
p.children)
+ }
+ }
+ classified.expensive ||
+ (conf.getConf(SQLConf.REGEX_PATTERN_COST_ESTIMATE_VERSION) == 0 &&
+ e.exists(_.isInstanceOf[StringRegexExpression]))
+ }
/**
* Check if we can safely push a filter through a projection, by making sure
that predicate
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index eeb0b3099627..9d86f86ee52a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -684,6 +684,18 @@ object SQLConf {
.checkValue(_ >= 0, "The version of the regex pattern cost estimate must
be nonnegative.")
.createWithDefault(1)
+ val AVOID_DOUBLE_FILTER_EVAL_FOR_PYTHON_UDF =
+ buildConf("spark.sql.optimizer.avoidDoubleFilterEvalForPythonUDF")
+ .internal()
+ .withBindingPolicy(ConfigBindingPolicy.NOT_APPLICABLE)
+ .doc(s"When true, ${AVOID_DOUBLE_FILTER_EVAL.key} treats a Python UDF as
expensive. When " +
+ "false, a Python UDF is only as expensive as its arguments, as before
4.4.0. A streaming " +
+ "query keeps the value recorded in its checkpoint, and one restarted
from a checkpoint " +
+ "without it uses false.")
+ .version("4.4.0")
+ .booleanConf
+ .createWithDefault(true)
+
val ATTEMPT_TRANSPILATION_OF_PYTHON_UDFS =
buildConf("spark.sql.experimental.optimizer.transpilePyUDFs")
.withBindingPolicy(ConfigBindingPolicy.SESSION)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
index 7b2dbd30d5b5..bf9bb54eefa5 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/FilterPushdownSuite.scala
@@ -270,6 +270,32 @@ class FilterPushdownSuite extends PlanTest {
}
}
+ test("SPARK-60905: Push a filter on a Python UDF alias when Python UDFs are
not expensive") {
+ def pythonUDF(child: Expression): PythonUDF = PythonUDF(
+ "pythonUDF", null, IntegerType, Seq(child),
PythonEvalType.SQL_BATCHED_UDF,
+ udfDeterministic = true)
+ // `g` passes the UDF an expensive argument.
+ val f = pythonUDF($"a") + 1
+ val g = pythonUDF(expensiveRLike("magic"))
+ val originalQuery = testStringRelation
+ .select($"a", f as "f", g as "g")
+ .where($"f" > 0 && $"g" > 0)
+ .analyze
+
+ // A parent expression caches whether its child is expensive, so optimize
the same plan
+ // with the default first.
+ Optimize.execute(originalQuery)
+ withSQLConf(SQLConf.AVOID_DOUBLE_FILTER_EVAL_FOR_PYTHON_UDF.key ->
"false") {
+ val correctAnswer = testStringRelation
+ .where(f > 0)
+ .select($"a", f as "f", g as "g")
+ .where($"g" > 0)
+ .analyze
+
+ comparePlans(Optimize.execute(originalQuery), correctAnswer)
+ }
+ }
+
// Case 1: Multiple filters that don't reference any projection aliases -
all should be pushed
test("SPARK-59693: Case 1 - multiple filters not referencing projection
aliases") {
val originalQuery = testStringRelation
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
index 2a79da4332cf..827d3a004637 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/OffsetSeq.scala
@@ -209,7 +209,8 @@ object OffsetSeqMetadata extends Logging {
DROP_DUPLICATES_DETERMINISTIC_KEY_ORDER,
ALLOW_EXCEPT_ON_STREAMING_DATAFRAME,
REGEX_PATTERN_COST_ESTIMATE_VERSION,
- AVOID_DOUBLE_FILTER_EVAL
+ AVOID_DOUBLE_FILTER_EVAL,
+ AVOID_DOUBLE_FILTER_EVAL_FOR_PYTHON_UDF
)
/**
@@ -283,7 +284,8 @@ object OffsetSeqMetadata extends Logging {
STATEFUL_OPERATOR_ALWAYS_NULLABLE_OUTPUT.key -> "false",
DROP_DUPLICATES_DETERMINISTIC_KEY_ORDER.key -> "false",
ALLOW_EXCEPT_ON_STREAMING_DATAFRAME.key -> "true",
- REGEX_PATTERN_COST_ESTIMATE_VERSION.key -> "0"
+ REGEX_PATTERN_COST_ESTIMATE_VERSION.key -> "0",
+ AVOID_DOUBLE_FILTER_EVAL_FOR_PYTHON_UDF.key -> "false"
)
/**
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
index 43eba51292a3..4f905da8a720 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala
@@ -93,6 +93,15 @@ class OffsetSeqLogSuite extends SharedSparkSession {
assert(sessionConf.getConfString(key.key) === "true")
}
+ test("SPARK-60905: Python UDFs are expensive only for checkpoints that
record it") {
+ val entry = SQLConf.AVOID_DOUBLE_FILTER_EVAL_FOR_PYTHON_UDF
+ val (_, offsetSeq) = readFromResource(
+ "checkpoint-version-4.2.0-deduplication-metadata-boundary/offsets")
+ assert(OffsetSeqMetadata.readValue(offsetSeq.metadataOpt.get, entry) ==
"false")
+ val metadata = OffsetSeqMetadata(batchWatermarkMs = 0, batchTimestampMs =
0, spark.conf)
+ assert(metadata.conf.get(entry.key).contains("true"))
+ }
+
test("SPARK-59919: v1 and v2 metadata persist rebound stateful shuffle
partitions") {
withSQLConf(
SQLConf.SHUFFLE_PARTITIONS.key -> "10",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]