This is an automated email from the ASF dual-hosted git repository.

HeartSaVioR pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.x by this push:
     new 188b99d8e76f [SPARK-60376][SS] Restrict V4 join matching scans to 
event-time bounds
188b99d8e76f is described below

commit 188b99d8e76fde0f9201d4e887ce6cd9df17877e
Author: Jungtaek Lim <[email protected]>
AuthorDate: Thu Oct 22 05:30:43 2026 +0900

    [SPARK-60376][SS] Restrict V4 join matching scans to event-time bounds
    
    ### What changes were proposed in this pull request?
    
    Restrict V4 stream-stream join matching-scan offset extraction to each 
input's selected event-time column, matching the timestamp stored in the index. 
If either event-time offset cannot be derived, scan the rows for the join key 
and apply the full join condition.
    
    The production change narrows the attribute sets passed to the existing 
arithmetic extractor. Fixed-interval extraction, state layout, eviction, and 
output-watermark calculation are unchanged.
    
    ### Why are the changes needed?
    
    V4 joins index state by event time. A condition on another timestamp column 
can incorrectly become a bound on that index, excluding valid matches before 
the full join condition is evaluated.
    
    For example, consider two streams watermarked on `L.ts` and `R.ts` with 
this join condition:
    
    ```sql
    L.key = R.key AND R.ts BETWEEN L.ts AND L.expiry
    ```
    
    `L.ts = 100s`, `L.expiry = 110s`, and `R.ts = 105s` satisfy the condition. 
However, treating the constraint on `L.expiry` as a bound on the indexed `L.ts` 
can cause the matching scan to skip this pair. Restricting extraction to the 
selected event-time columns prevents that invalid scan bound.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. V4 stream-stream joins with bounds on a separate timestamp column 
return valid matches that were previously skipped. Queries whose bounds cannot 
be derived for the indexed event time use a full scan of the join key, which 
may read more rows than the incorrect narrowed scan.
    
    Checkpoint/state layout, eviction, and output-watermark calculation are 
unchanged. Missing outputs from already completed batches are not reconstructed.
    
    ### How was this patch tested?
    
    Added a regression in `StreamingInnerJoinBase`, inherited by the existing 
V2, V3, and V4 inner-join suites. It covers BETWEEN and equivalent conjunction 
conditions using a separate timestamp column, fixed-interval controls, 
simultaneous and one-sided arrivals, checkpoint restart in both directions, and 
boundary exclusions.
    
    ```text
    build/sbt 'sql/testOnly *StreamingInnerJoin*Suite -- -z "matching scan on 
event time"'
    ```
    
    All 90 cases passed across the three suites, with zero failures or aborted 
suites. SQL Compile/Test scalastyle also passed with zero errors and warnings.
    
    With the pre-fix executor, all 20 V4 regression cases using the separate 
timestamp column fail with missing matches, while all 10 fixed-interval 
controls pass.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6)
    
    Closes #59628 from 
HeartSaVioR/jungtaek-lim_data/fix-join-event-time-matching-scan.
    
    Authored-by: Jungtaek Lim <[email protected]>
    Signed-off-by: Jungtaek Lim <[email protected]>
    (cherry picked from commit 1d8d40ed1e22c92c77a60bad7bf36b270c9e37dc)
    Signed-off-by: Jungtaek Lim <[email protected]>
---
 .../join/StreamingSymmetricHashJoinExec.scala      | 11 ++++-
 .../spark/sql/streaming/StreamingJoinSuite.scala   | 53 ++++++++++++++++++++++
 2 files changed, 62 insertions(+), 2 deletions(-)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
index 853117c49c9f..f63c82371bad 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
@@ -739,10 +739,17 @@ case class StreamingSymmetricHashJoinExec(
           case RightSide => (right.output, left.output)
         }
 
+        // Scan bounds must constrain the event-time columns used by the index 
and probe.
+        // Bounds on other columns cannot limit this scan. If either 
event-time bound is
+        // missing, scan all rows for the join key.
+        val thisEventTime = WatermarkSupport.findEventTimeColumn(
+          thisSideAttrs, !allowMultipleStatefulOperators).toSeq
+        val otherEventTime = WatermarkSupport.findEventTimeColumn(
+          otherSideAttrs, !allowMultipleStatefulOperators).toSeq
         val lowerBoundMs = StreamingJoinHelper.getStateValueWatermark(
-          AttributeSet(otherSideAttrs), AttributeSet(thisSideAttrs), 
condition.full, Some(0L))
+          AttributeSet(otherEventTime), AttributeSet(thisEventTime), 
condition.full, Some(0L))
         val upperBoundMs = StreamingJoinHelper.getStateValueWatermark(
-          AttributeSet(thisSideAttrs), AttributeSet(otherSideAttrs), 
condition.full, Some(0L))
+          AttributeSet(thisEventTime), AttributeSet(otherEventTime), 
condition.full, Some(0L))
 
         (lowerBoundMs, upperBoundMs) match {
           case (Some(lower), Some(upper)) =>
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
index fed678c77585..2adb5b25bc00 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
@@ -577,6 +577,59 @@ abstract class StreamingInnerJoinBase extends 
StreamingJoinSuite {
     )
   }
 
+  private object MatchingScanArrival extends Enumeration {
+    val Together, LeftFirst, RightFirst, LeftRestart, RightRestart = Value
+  }
+
+  private object MatchingScanShape extends Enumeration {
+    val Between, Conjunction, FixedInterval = Value
+  }
+
+  for {
+    arrival <- MatchingScanArrival.values
+    shape <- MatchingScanShape.values
+  } {
+    test(s"stream stream inner join matching scan on event time, $arrival 
$shape") {
+      val leftInput = MemoryStream[(Int, Long)]
+      val rightInput = MemoryStream[(Int, Long)]
+      val left = leftInput.toDF().toDF("leftKey", "time")
+        .select($"leftKey", timestamp_millis($"time").as("leftTime"))
+        .withColumn("expiry", $"leftTime" + expr("INTERVAL 10 SECONDS"))
+        .withWatermark("leftTime", "60 seconds")
+      val right = rightInput.toDF().toDF("rightKey", "time")
+        .select($"rightKey", timestamp_millis($"time").as("rightTime"))
+        .withWatermark("rightTime", "60 seconds")
+      val range = shape match {
+        case MatchingScanShape.Between => "rightTime BETWEEN leftTime AND 
expiry"
+        case MatchingScanShape.Conjunction => "rightTime >= leftTime AND 
rightTime <= expiry"
+        case MatchingScanShape.FixedInterval =>
+          "rightTime BETWEEN leftTime AND leftTime + INTERVAL 10 SECONDS"
+      }
+      val joined = left.join(right, expr(s"leftKey = rightKey AND $range"))
+        .select(unix_millis($"rightTime") - 100000L)
+      val offsets = Seq(0L, 1L, 2L, 1000L, 5000L, 10000L)
+      // Include nonmatching rows immediately outside both inclusive 
boundaries.
+      val rows = (Seq(-1L) ++ offsets ++ Seq(10001L)).map(t => (1, 100000L + 
t))
+      val input = arrival match {
+        case MatchingScanArrival.Together =>
+          Seq(MultiAddData(leftInput, (1, 100000L))(rightInput, rows: _*))
+        case MatchingScanArrival.LeftFirst =>
+          Seq(AddData(leftInput, (1, 100000L)), CheckNewAnswer(),
+            AddData(rightInput, rows: _*))
+        case MatchingScanArrival.RightFirst =>
+          Seq(AddData(rightInput, rows: _*), CheckNewAnswer(),
+            AddData(leftInput, (1, 100000L)))
+        case MatchingScanArrival.LeftRestart =>
+          Seq(AddData(leftInput, (1, 100000L)), CheckNewAnswer(),
+            StopStream, StartStream(), AddData(rightInput, rows: _*))
+        case MatchingScanArrival.RightRestart =>
+          Seq(AddData(rightInput, rows: _*), CheckNewAnswer(),
+            StopStream, StartStream(), AddData(leftInput, (1, 100000L)))
+      }
+      testStream(joined)((input :+ CheckNewAnswer(offsets.map(Row(_)): _*)): 
_*)
+    }
+  }
+
   testQuietly("stream stream inner join without equality predicate") {
     val input1 = MemoryStream[Int]
     val input2 = MemoryStream[Int]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to