This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 0bf4d061815d [SPARK-60375][SS] Base left semi join output watermarks
on emitted event time
0bf4d061815d is described below
commit 0bf4d061815dd59ddd532304a3af48791fd5904b
Author: Jungtaek Lim <[email protected]>
AuthorDate: Thu Oct 22 05:32:56 2026 +0900
[SPARK-60375][SS] Base left semi join output watermarks on emitted event
time
### What changes were proposed in this pull request?
Derive a left semi stream-stream join's output watermark from the
event-time column that survives in its output. A left semi join returns only
left columns:
- If the output contains a watermark-bearing event-time column, use the
minimum of the left state watermark and input watermark.
- If no output column carries watermark metadata, return no output
watermark (`None`). A watermark defined only on the right is lost with the
right columns.
### Why are the changes needed?
An output watermark describes the event times a downstream operator may
receive. A left semi join emits left rows, including buffered left rows that
match a later right input. Its output watermark should therefore follow the
bound on that left event-time column.
The existing calculation also includes the right-state watermark. Retaining
older right rows is necessary for future matches, but their timestamps are
never emitted by a left semi join. Including that bound unnecessarily delays
downstream window completion and state cleanup.
For `L > R - INTERVAL 5 SECONDS AND L < R + INTERVAL 30 SECONDS`, an input
watermark of 120 seconds gives left and right state watermarks of 115 and 90
seconds. Using the bound on emitted left event time advances the output
watermark to 115 seconds and closes the downstream window `[100, 110)`.
Including the right-state bound holds the output watermark at 90 seconds and
leaves the window buffered.
When the left output has no watermark-bearing column, reporting no output
watermark explicitly represents that loss.
### Does this PR introduce _any_ user-facing change?
Yes. Append-mode aggregations after bounded left semi joins can emit
completed windows and clean up downstream state earlier. A left semi join
without watermark metadata on its output no longer propagates a numeric output
watermark.
This is an improvement to watermark propagation and downstream progress.
Join matching, state eviction, and checkpoint/state layout are unchanged.
### How was this patch tested?
Added equality and range streaming-query cases with and without a
watermark-bearing left output column in `StreamingLeftSemiJoinBase`. The four
cases run across six state-format/provider configurations, for 24 cases. They
check both the output-watermark value and its absence when watermark metadata
is lost.
Added an end-to-end left semi join followed by windowed aggregation in
`MultiStatefulOperatorsSuite`, including a checkpoint restart. The three-batch
scenario verifies that window `[100, 110)` is emitted when the output watermark
reaches 115 seconds. It runs in six configurations: V2 with HDFS/RocksDB, and
V3/V4 with RocksDB changelog checkpointing off/on. This regression fails with
the original both-side calculation and passes with the change.
The 24 metadata cases and six end-to-end configurations passed. SQL Test
scalastyle passed with zero errors and warnings.
```
build/sbt 'sql/testOnly *StreamingLeftSemiJoin*Suite -- -z "left semi
output watermark"'
build/sbt 'sql/testOnly *MultiStatefulOperatorsSuite -- -z "left semi
output watermark closes"'
build/sbt 'sql/Test/scalastyle'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59626 from
HeartSaVioR/jungtaek-lim_data/fix-left-semi-output-watermark.
Authored-by: Jungtaek Lim <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
(cherry picked from commit 080731eb2a69ee30eb24023b6868b7db36a28abf)
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../join/StreamingSymmetricHashJoinExec.scala | 19 ++++---
.../streaming/MultiStatefulOperatorsSuite.scala | 61 +++++++++++++++++++++-
.../spark/sql/streaming/StreamingJoinSuite.scala | 41 +++++++++++++++
3 files changed, 114 insertions(+), 7 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
index f63c82371bad..23c58c5556e4 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
@@ -1117,17 +1117,24 @@ case class StreamingSymmetricHashJoinExec(
Nil
}
- // This operator will evict based on the state watermark on both side of
inputs; we would like
- // to let users leverage both sides of event time column for output of join,
so the watermark
- // must be lower bound of both sides of event time column. The lower bound
of event time column
- // for each side is determined by state watermark, hence we take a minimum
of (left state
- // watermark, right state watermark, input watermark) to decide the output
watermark.
+ // The output watermark bounds the event-time columns emitted by the join.
State watermarks
+ // bound buffered rows that can still be emitted, and the input watermark
bounds new rows.
+ // Joins emitting both sides take the minimum of both state watermarks and
the input watermark.
override def produceOutputWatermark(inputWatermarkMs: Long): Option[Long] = {
val (leftStateWatermark, rightStateWatermark) =
StreamingSymmetricHashJoinHelper.getStateWatermark(
left.output, right.output, leftKeys, rightKeys, condition.full,
Some(inputWatermarkMs),
!allowMultipleStatefulOperators)
- Some((leftStateWatermark ++ rightStateWatermark ++
Some(inputWatermarkMs)).min)
+ if (joinType == LeftSemi) {
+ // Only left columns are output. Without a watermarked left column, the
watermark is lost;
+ // the right state watermark cannot replace it or constrain the emitted
left event time.
+ WatermarkSupport.findEventTimeColumn(output,
+ allowMultipleEventTimeColumns = !allowMultipleStatefulOperators).map {
_ =>
+ (leftStateWatermark ++ Some(inputWatermarkMs)).min
+ }
+ } else {
+ Some((leftStateWatermark ++ rightStateWatermark ++
Some(inputWatermarkMs)).min)
+ }
}
}
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
index cd901deae8e1..92d326234b96 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/MultiStatefulOperatorsSuite.scala
@@ -25,7 +25,7 @@ import org.apache.spark.sql.{DataFrame, SparkSession}
import
org.apache.spark.sql.execution.streaming.operators.stateful.StateStoreSaveExec
import
org.apache.spark.sql.execution.streaming.operators.stateful.join.StreamingSymmetricHashJoinExec
import org.apache.spark.sql.execution.streaming.runtime.MemoryStream
-import org.apache.spark.sql.execution.streaming.state.StateStore
+import
org.apache.spark.sql.execution.streaming.state.{HDFSBackedStateStoreProvider,
RocksDBStateStoreProvider, StateStore}
import org.apache.spark.sql.functions._
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.tags.SlowSQLTest
@@ -887,6 +887,65 @@ class MultiStatefulOperatorsSuite
testOutputWatermarkInJoin(join3, input1, -40L * 1000 - 1)
}
+ Seq(
+ (2, classOf[HDFSBackedStateStoreProvider], false),
+ (2, classOf[RocksDBStateStoreProvider], false),
+ (3, classOf[RocksDBStateStoreProvider], false),
+ (3, classOf[RocksDBStateStoreProvider], true),
+ (4, classOf[RocksDBStateStoreProvider], false),
+ (4, classOf[RocksDBStateStoreProvider], true)
+ ).foreach { case (stateFormatVersion, provider, changelogEnabled) =>
+ test("left semi output watermark closes downstream windows across restart:
" +
+ s"state format=$stateFormatVersion, ${provider.getSimpleName}, " +
+ s"changelog=$changelogEnabled") {
+ val leftInput = MemoryStream[(Int, Int)]
+ val rightInput = MemoryStream[(Int, Int)]
+ val left = leftInput.toDF()
+ .select($"_1".as("leftKey"), timestamp_seconds($"_2").as("leftTime"))
+ .withWatermark("leftTime", "0 seconds")
+ val right = rightInput.toDF()
+ .select($"_1".as("rightKey"), timestamp_seconds($"_2").as("rightTime"))
+ .withWatermark("rightTime", "0 seconds")
+ val result = left.join(right, expr(
+ "leftKey = rightKey AND leftTime > rightTime - INTERVAL 5 SECONDS AND
" +
+ "leftTime < rightTime + INTERVAL 30 SECONDS"), "left_semi")
+ .groupBy(window($"leftTime", "10 seconds"))
+ .count()
+ .select($"window.start".cast("long"), $"count")
+
+ withSQLConf(
+ SQLConf.STREAMING_JOIN_STATE_FORMAT_VERSION.key ->
stateFormatVersion.toString,
+ SQLConf.STATE_STORE_PROVIDER_CLASS.key -> provider.getName,
+
"spark.sql.streaming.stateStore.rocksdb.changelogCheckpointing.enabled" ->
+ changelogEnabled.toString,
+ SQLConf.STATEFUL_OPERATOR_ALLOW_MULTIPLE.key -> "true",
+ SQLConf.STREAMING_NO_DATA_MICRO_BATCHES_ENABLED.key -> "false") {
+ withTempDir { checkpointDir =>
+ // Watermarks below are in seconds and describe the bounds used in
each batch.
+ // inputW is the join's eviction watermark; leftW/rightW are its
state bounds;
+ // outputW is the downstream aggregation's eviction watermark.
+ // Join rows are (key, event time); aggregation state is [window
start, end): count.
+ testStream(result)(
+ StartStream(checkpointLocation = checkpointDir.getCanonicalPath),
+ // Batch 0: inputW=0, leftW=-5, rightW=-30; propagation starts at
outputW=0.
+ // Join emits (1, 105); aggregation buffers [100, 110): 1.
+ MultiAddData(leftInput, (1, 105))(rightInput, (1, 105)),
+ CheckNewAnswer(),
+ // Batch 1: inputW=105, leftW=100, rightW=75, outputW=100.
+ // Join emits (2, 120); aggregation buffers [100, 110): 1 and
[120, 130): 1.
+ MultiAddData(leftInput, (2, 120))(rightInput, (2, 120)),
+ CheckNewAnswer(),
+ StopStream,
+ StartStream(checkpointLocation = checkpointDir.getCanonicalPath),
+ // Batch 2: inputW=120, leftW=115, rightW=90, outputW=115 (after
restart).
+ // Join emits nothing; aggregation emits [100, 110): 1 and keeps
[120, 130): 1.
+ AddData(rightInput, (9, 120)),
+ CheckNewAnswer((100L, 1L)))
+ }
+ }
+ }
+ }
+
test("SPARK-49829 time window agg per each source followed by stream-stream
join") {
val inputStream1 = MemoryStream[Long]
val inputStream2 = MemoryStream[Long]
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
index 2adb5b25bc00..37ed60e9370b 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
@@ -2328,6 +2328,47 @@ abstract class StreamingLeftSemiJoinBase extends
StreamingJoinSuite {
import testImplicits._
+ for (leftWatermark <- Seq(false, true); joinOnTime <- Seq(false, true)) {
+ test(s"left semi output watermark follows output event time: " +
+ s"left watermark=$leftWatermark, equality=$joinOnTime") {
+ val leftInput = MemoryStream[(Int, Int)]
+ val rightInput = MemoryStream[(Int, Int)]
+ val leftWithoutWatermark = leftInput.toDF()
+ .select($"_1".as("leftKey"), timestamp_seconds($"_2").as("leftTime"))
+ val left = if (leftWatermark) {
+ leftWithoutWatermark.withWatermark("leftTime", "0 seconds")
+ } else {
+ leftWithoutWatermark
+ }
+ val right = rightInput.toDF()
+ .select($"_1".as("rightKey"), timestamp_seconds($"_2").as("rightTime"))
+ .withWatermark("rightTime", "0 seconds")
+ val timeCondition = if (joinOnTime) {
+ "leftTime = rightTime"
+ } else {
+ "leftTime > rightTime - INTERVAL 5 SECONDS AND " +
+ "leftTime < rightTime + INTERVAL 30 SECONDS"
+ }
+ val joined = left.join(right, expr(s"leftKey = rightKey AND
$timeCondition"), "left_semi")
+ .select($"leftKey", $"leftTime".cast("int"))
+ val expectedWatermark = if (leftWatermark) {
+ Some(if (joinOnTime) 120000L else 115000L)
+ } else {
+ None
+ }
+
+ testStream(joined)(
+ MultiAddData(leftInput, (1, 5))(rightInput, (1, 5)),
+ CheckNewAnswer((1, 5)),
+ Execute { query =>
+ val joinOperator = query.lastExecution.executedPlan.collect {
+ case j: StreamingSymmetricHashJoinExec => j
+ }.head
+ assert(joinOperator.produceOutputWatermark(120000L) ===
expectedWatermark)
+ })
+ }
+ }
+
testWithAppendAndUpdate("windowed left semi join") { outputMode =>
withTempDir { checkpointDir =>
val (leftInput, rightInput, joined) = setupWindowedJoin("left_semi")
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]