This is an automated email from the ASF dual-hosted git repository.

HeartSaVioR pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new 18887c7ce1fa [SPARK-60373][SS] Count state rows removed while 
producing outer join output
18887c7ce1fa is described below

commit 18887c7ce1fafa7f8031d298f6ebb990203e7adf
Author: Jungtaek Lim <[email protected]>
AuthorDate: Thu Oct 22 14:40:52 2026 +0900

    [SPARK-60373][SS] Count state rows removed while producing outer join output
    
    ### What changes were proposed in this pull request?
    
    Count state rows removed by `removeAndReturnOldState` in the streaming 
join's `numRowsRemoved` metric. Increment the counter as each removed row is 
consumed, before filtering matched rows from outer-join output.
    
    ### Why are the changes needed?
    
    Left and right outer joins currently count removals only from the side 
cleaned up after output generation. Full outer joins remove state from both 
sides while generating output, so their removal metric stays zero even when 
state is deleted.
    
    The windowed full-outer regression stores five rows on each side, including 
three matching pairs. After restarting from the same checkpoint and advancing 
the watermark, all ten old state rows are removed, while four unmatched rows 
produce null-extended output. Query progress should report ten removed rows; 
the original implementation reports zero. The left/right range-join regressions 
similarly remove six state rows, but the original metric omits removals from 
the preserved side.
    
    The omission dates to Spark 3.2.0, when `numRowsRemoved` was introduced.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Streaming query progress correctly counts all evicted outer-join state 
rows in `numRowsRemoved`, including matched rows that do not produce 
null-extended output. This changes metric accounting only; query results, 
eviction decisions, state layout, and checkpoint compatibility remain unchanged.
    
    ### How was this patch tested?
    
    Added query-progress assertions to the existing end-to-end left/right 
range-join and full-outer checkpoint-restart tests, retaining their output and 
state-row assertions.
    
    ```text
    build/sbt 'sql/testOnly *StreamingOuterJoin*Suite -- -z "with watermark 
range condition"'
    build/sbt 'sql/testOnly *StreamingFullOuterJoin*Suite -- -z "windowed full 
outer join"'
    ```
    
    Before the fix, the full-outer regression failed with both HDFS and RocksDB 
state store providers (`0 did not equal 10`). With the fix, all 18 cases 
passed: 12 left/right range-join cases and 6 full-outer checkpoint-restart 
cases. Coverage includes state format V2 with HDFS/RocksDB and V3/V4 with 
RocksDB changelog checkpointing disabled/enabled. SQL Compile/Test scalastyle 
reported zero errors and warnings.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6)
    
    Closes #59621 from 
HeartSaVioR/jungtaek-lim_data/fix-streaming-join-outer-removal-metrics.
    
    Authored-by: Jungtaek Lim <[email protected]>
    Signed-off-by: Jungtaek Lim <[email protected]>
    (cherry picked from commit 868870d641a9517f551a161d9ccc3450803398a8)
    Signed-off-by: Jungtaek Lim <[email protected]>
---
 .../stateful/join/StreamingSymmetricHashJoinExec.scala      |  7 ++++++-
 .../org/apache/spark/sql/streaming/StreamingJoinSuite.scala | 13 ++++++++++---
 2 files changed, 16 insertions(+), 4 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
index 23c58c5556e4..3835b67bdab8 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
@@ -974,7 +974,8 @@ case class StreamingSymmetricHashJoinExec(
      * processing the rows for outer join.
      */
     def removeAndReturnOldState(): Iterator[KeyToValuePair] = {
-      stateWatermarkPredicate match {
+      val numRemovedStateRows = longMetric("numRemovedStateRows")
+      val removedRowIter = stateWatermarkPredicate match {
         case Some(JoinStateKeyWatermarkPredicate(_, stateWatermark, 
prevStateWatermark)) =>
           joinStateManager match {
             case s: SupportsEvictByCondition =>
@@ -997,6 +998,10 @@ case class StreamingSymmetricHashJoinExec(
           }
         case _ => Iterator.empty
       }
+      removedRowIter.map { row =>
+        numRemovedStateRows += 1
+        row
+      }
     }
 
     /**
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
index 7027625a8d34..0a62a68b3df7 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
@@ -1556,7 +1556,9 @@ abstract class StreamingOuterJoinBase extends 
StreamingJoinSuite {
         assertNumStateRows(total = 7, updated = 2),
         AddData(rightInput, (0, 30)), // watermark = 30 - 10 = 20, 
no-data-batch computes nulls
         CheckNewAnswer(outerResult),
-        assertNumStateRows(total = 2, updated = 1)
+        assertNumStateRows(
+          total = Seq(2), updated = Seq(1), droppedByWatermark = Seq(0),
+          removed = Some(Seq(6)))
       )
 
       Seq(
@@ -1584,7 +1586,9 @@ abstract class StreamingOuterJoinBase extends 
StreamingJoinSuite {
           assertNumStateRows(total = 7, updated = 2),
           AddData(rightInput2, (0, 1800)),
           CheckNewAnswer(outerResult),
-          assertNumStateRows(total = 2, updated = 1)
+          assertNumStateRows(
+            total = Seq(2), updated = Seq(1), droppedByWatermark = Seq(0),
+            removed = Some(Seq(6)))
         )
       }
     }
@@ -2185,7 +2189,10 @@ abstract class StreamingFullOuterJoinBase extends 
StreamingJoinSuite {
         // states evicted
         // left: 1, 2, 3, 4 ,5 (below watermark)
         // right: 3, 4, 5, 6, 7 (below watermark)
-        assertNumStateRows(total = 2, updated = 2),
+        // Count matched rows too: only four of the ten removed rows produce 
null output.
+        assertNumStateRows(
+          total = Seq(2), updated = Seq(2), droppedByWatermark = Seq(0),
+          removed = Some(Seq(10))),
         StopStream,
         // Restart the query from the same checkpoint
         StartStream(checkpointLocation = checkpointDir.getCanonicalPath),


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to