This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 868870d641a9 [SPARK-60373][SS] Count state rows removed while
producing outer join output
868870d641a9 is described below
commit 868870d641a9517f551a161d9ccc3450803398a8
Author: Jungtaek Lim <[email protected]>
AuthorDate: Thu Oct 22 14:40:52 2026 +0900
[SPARK-60373][SS] Count state rows removed while producing outer join output
### What changes were proposed in this pull request?
Count state rows removed by `removeAndReturnOldState` in the streaming
join's `numRowsRemoved` metric. Increment the counter as each removed row is
consumed, before filtering matched rows from outer-join output.
### Why are the changes needed?
Left and right outer joins currently count removals only from the side
cleaned up after output generation. Full outer joins remove state from both
sides while generating output, so their removal metric stays zero even when
state is deleted.
The windowed full-outer regression stores five rows on each side, including
three matching pairs. After restarting from the same checkpoint and advancing
the watermark, all ten old state rows are removed, while four unmatched rows
produce null-extended output. Query progress should report ten removed rows;
the original implementation reports zero. The left/right range-join regressions
similarly remove six state rows, but the original metric omits removals from
the preserved side.
The omission dates to Spark 3.2.0, when `numRowsRemoved` was introduced.
### Does this PR introduce _any_ user-facing change?
Yes. Streaming query progress correctly counts all evicted outer-join state
rows in `numRowsRemoved`, including matched rows that do not produce
null-extended output. This changes metric accounting only; query results,
eviction decisions, state layout, and checkpoint compatibility remain unchanged.
### How was this patch tested?
Added query-progress assertions to the existing end-to-end left/right
range-join and full-outer checkpoint-restart tests, retaining their output and
state-row assertions.
```text
build/sbt 'sql/testOnly *StreamingOuterJoin*Suite -- -z "with watermark
range condition"'
build/sbt 'sql/testOnly *StreamingFullOuterJoin*Suite -- -z "windowed full
outer join"'
```
Before the fix, the full-outer regression failed with both HDFS and RocksDB
state store providers (`0 did not equal 10`). With the fix, all 18 cases
passed: 12 left/right range-join cases and 6 full-outer checkpoint-restart
cases. Coverage includes state format V2 with HDFS/RocksDB and V3/V4 with
RocksDB changelog checkpointing disabled/enabled. SQL Compile/Test scalastyle
reported zero errors and warnings.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59621 from
HeartSaVioR/jungtaek-lim_data/fix-streaming-join-outer-removal-metrics.
Authored-by: Jungtaek Lim <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../stateful/join/StreamingSymmetricHashJoinExec.scala | 7 ++++++-
.../org/apache/spark/sql/streaming/StreamingJoinSuite.scala | 13 ++++++++++---
2 files changed, 16 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
index 23c58c5556e4..3835b67bdab8 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
@@ -974,7 +974,8 @@ case class StreamingSymmetricHashJoinExec(
* processing the rows for outer join.
*/
def removeAndReturnOldState(): Iterator[KeyToValuePair] = {
- stateWatermarkPredicate match {
+ val numRemovedStateRows = longMetric("numRemovedStateRows")
+ val removedRowIter = stateWatermarkPredicate match {
case Some(JoinStateKeyWatermarkPredicate(_, stateWatermark,
prevStateWatermark)) =>
joinStateManager match {
case s: SupportsEvictByCondition =>
@@ -997,6 +998,10 @@ case class StreamingSymmetricHashJoinExec(
}
case _ => Iterator.empty
}
+ removedRowIter.map { row =>
+ numRemovedStateRows += 1
+ row
+ }
}
/**
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
index 7027625a8d34..0a62a68b3df7 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
@@ -1556,7 +1556,9 @@ abstract class StreamingOuterJoinBase extends
StreamingJoinSuite {
assertNumStateRows(total = 7, updated = 2),
AddData(rightInput, (0, 30)), // watermark = 30 - 10 = 20,
no-data-batch computes nulls
CheckNewAnswer(outerResult),
- assertNumStateRows(total = 2, updated = 1)
+ assertNumStateRows(
+ total = Seq(2), updated = Seq(1), droppedByWatermark = Seq(0),
+ removed = Some(Seq(6)))
)
Seq(
@@ -1584,7 +1586,9 @@ abstract class StreamingOuterJoinBase extends
StreamingJoinSuite {
assertNumStateRows(total = 7, updated = 2),
AddData(rightInput2, (0, 1800)),
CheckNewAnswer(outerResult),
- assertNumStateRows(total = 2, updated = 1)
+ assertNumStateRows(
+ total = Seq(2), updated = Seq(1), droppedByWatermark = Seq(0),
+ removed = Some(Seq(6)))
)
}
}
@@ -2185,7 +2189,10 @@ abstract class StreamingFullOuterJoinBase extends
StreamingJoinSuite {
// states evicted
// left: 1, 2, 3, 4 ,5 (below watermark)
// right: 3, 4, 5, 6, 7 (below watermark)
- assertNumStateRows(total = 2, updated = 2),
+ // Count matched rows too: only four of the ten removed rows produce
null output.
+ assertNumStateRows(
+ total = Seq(2), updated = Seq(2), droppedByWatermark = Seq(0),
+ removed = Some(Seq(10))),
StopStream,
// Restart the query from the same checkpoint
StartStream(checkpointLocation = checkpointDir.getCanonicalPath),
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]