This is an automated email from the ASF dual-hosted git repository.
HeartSaVioR pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 188b99d8e76f [SPARK-60376][SS] Restrict V4 join matching scans to
event-time bounds
188b99d8e76f is described below
commit 188b99d8e76fde0f9201d4e887ce6cd9df17877e
Author: Jungtaek Lim <[email protected]>
AuthorDate: Thu Oct 22 05:30:43 2026 +0900
[SPARK-60376][SS] Restrict V4 join matching scans to event-time bounds
### What changes were proposed in this pull request?
Restrict V4 stream-stream join matching-scan offset extraction to each
input's selected event-time column, matching the timestamp stored in the index.
If either event-time offset cannot be derived, scan the rows for the join key
and apply the full join condition.
The production change narrows the attribute sets passed to the existing
arithmetic extractor. Fixed-interval extraction, state layout, eviction, and
output-watermark calculation are unchanged.
### Why are the changes needed?
V4 joins index state by event time. A condition on another timestamp column
can incorrectly become a bound on that index, excluding valid matches before
the full join condition is evaluated.
For example, consider two streams watermarked on `L.ts` and `R.ts` with
this join condition:
```sql
L.key = R.key AND R.ts BETWEEN L.ts AND L.expiry
```
`L.ts = 100s`, `L.expiry = 110s`, and `R.ts = 105s` satisfy the condition.
However, treating the constraint on `L.expiry` as a bound on the indexed `L.ts`
can cause the matching scan to skip this pair. Restricting extraction to the
selected event-time columns prevents that invalid scan bound.
### Does this PR introduce _any_ user-facing change?
Yes. V4 stream-stream joins with bounds on a separate timestamp column
return valid matches that were previously skipped. Queries whose bounds cannot
be derived for the indexed event time use a full scan of the join key, which
may read more rows than the incorrect narrowed scan.
Checkpoint/state layout, eviction, and output-watermark calculation are
unchanged. Missing outputs from already completed batches are not reconstructed.
### How was this patch tested?
Added a regression in `StreamingInnerJoinBase`, inherited by the existing
V2, V3, and V4 inner-join suites. It covers BETWEEN and equivalent conjunction
conditions using a separate timestamp column, fixed-interval controls,
simultaneous and one-sided arrivals, checkpoint restart in both directions, and
boundary exclusions.
```text
build/sbt 'sql/testOnly *StreamingInnerJoin*Suite -- -z "matching scan on
event time"'
```
All 90 cases passed across the three suites, with zero failures or aborted
suites. SQL Compile/Test scalastyle also passed with zero errors and warnings.
With the pre-fix executor, all 20 V4 regression cases using the separate
timestamp column fail with missing matches, while all 10 fixed-interval
controls pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59628 from
HeartSaVioR/jungtaek-lim_data/fix-join-event-time-matching-scan.
Authored-by: Jungtaek Lim <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
(cherry picked from commit 1d8d40ed1e22c92c77a60bad7bf36b270c9e37dc)
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../join/StreamingSymmetricHashJoinExec.scala | 11 ++++-
.../spark/sql/streaming/StreamingJoinSuite.scala | 53 ++++++++++++++++++++++
2 files changed, 62 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
index 853117c49c9f..f63c82371bad 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/operators/stateful/join/StreamingSymmetricHashJoinExec.scala
@@ -739,10 +739,17 @@ case class StreamingSymmetricHashJoinExec(
case RightSide => (right.output, left.output)
}
+ // Scan bounds must constrain the event-time columns used by the index
and probe.
+ // Bounds on other columns cannot limit this scan. If either
event-time bound is
+ // missing, scan all rows for the join key.
+ val thisEventTime = WatermarkSupport.findEventTimeColumn(
+ thisSideAttrs, !allowMultipleStatefulOperators).toSeq
+ val otherEventTime = WatermarkSupport.findEventTimeColumn(
+ otherSideAttrs, !allowMultipleStatefulOperators).toSeq
val lowerBoundMs = StreamingJoinHelper.getStateValueWatermark(
- AttributeSet(otherSideAttrs), AttributeSet(thisSideAttrs),
condition.full, Some(0L))
+ AttributeSet(otherEventTime), AttributeSet(thisEventTime),
condition.full, Some(0L))
val upperBoundMs = StreamingJoinHelper.getStateValueWatermark(
- AttributeSet(thisSideAttrs), AttributeSet(otherSideAttrs),
condition.full, Some(0L))
+ AttributeSet(thisEventTime), AttributeSet(otherEventTime),
condition.full, Some(0L))
(lowerBoundMs, upperBoundMs) match {
case (Some(lower), Some(upper)) =>
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
index fed678c77585..2adb5b25bc00 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingJoinSuite.scala
@@ -577,6 +577,59 @@ abstract class StreamingInnerJoinBase extends
StreamingJoinSuite {
)
}
+ private object MatchingScanArrival extends Enumeration {
+ val Together, LeftFirst, RightFirst, LeftRestart, RightRestart = Value
+ }
+
+ private object MatchingScanShape extends Enumeration {
+ val Between, Conjunction, FixedInterval = Value
+ }
+
+ for {
+ arrival <- MatchingScanArrival.values
+ shape <- MatchingScanShape.values
+ } {
+ test(s"stream stream inner join matching scan on event time, $arrival
$shape") {
+ val leftInput = MemoryStream[(Int, Long)]
+ val rightInput = MemoryStream[(Int, Long)]
+ val left = leftInput.toDF().toDF("leftKey", "time")
+ .select($"leftKey", timestamp_millis($"time").as("leftTime"))
+ .withColumn("expiry", $"leftTime" + expr("INTERVAL 10 SECONDS"))
+ .withWatermark("leftTime", "60 seconds")
+ val right = rightInput.toDF().toDF("rightKey", "time")
+ .select($"rightKey", timestamp_millis($"time").as("rightTime"))
+ .withWatermark("rightTime", "60 seconds")
+ val range = shape match {
+ case MatchingScanShape.Between => "rightTime BETWEEN leftTime AND
expiry"
+ case MatchingScanShape.Conjunction => "rightTime >= leftTime AND
rightTime <= expiry"
+ case MatchingScanShape.FixedInterval =>
+ "rightTime BETWEEN leftTime AND leftTime + INTERVAL 10 SECONDS"
+ }
+ val joined = left.join(right, expr(s"leftKey = rightKey AND $range"))
+ .select(unix_millis($"rightTime") - 100000L)
+ val offsets = Seq(0L, 1L, 2L, 1000L, 5000L, 10000L)
+ // Include nonmatching rows immediately outside both inclusive
boundaries.
+ val rows = (Seq(-1L) ++ offsets ++ Seq(10001L)).map(t => (1, 100000L +
t))
+ val input = arrival match {
+ case MatchingScanArrival.Together =>
+ Seq(MultiAddData(leftInput, (1, 100000L))(rightInput, rows: _*))
+ case MatchingScanArrival.LeftFirst =>
+ Seq(AddData(leftInput, (1, 100000L)), CheckNewAnswer(),
+ AddData(rightInput, rows: _*))
+ case MatchingScanArrival.RightFirst =>
+ Seq(AddData(rightInput, rows: _*), CheckNewAnswer(),
+ AddData(leftInput, (1, 100000L)))
+ case MatchingScanArrival.LeftRestart =>
+ Seq(AddData(leftInput, (1, 100000L)), CheckNewAnswer(),
+ StopStream, StartStream(), AddData(rightInput, rows: _*))
+ case MatchingScanArrival.RightRestart =>
+ Seq(AddData(rightInput, rows: _*), CheckNewAnswer(),
+ StopStream, StartStream(), AddData(leftInput, (1, 100000L)))
+ }
+ testStream(joined)((input :+ CheckNewAnswer(offsets.map(Row(_)): _*)):
_*)
+ }
+ }
+
testQuietly("stream stream inner join without equality predicate") {
val input1 = MemoryStream[Int]
val input2 = MemoryStream[Int]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]