Jungtaek Lim created SPARK-61513:
------------------------------------

             Summary: Stream-stream join in state format version 4 misses 
matches because its range scan window is too narrow
                 Key: SPARK-61513
                 URL: https://issues.apache.org/jira/browse/SPARK-61513
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


In state format version 4, a stream-stream join row reads only the other side's 
rows in a time window. Spark derives the window's offsets from the join 
condition with the watermark solver, at a watermark of 0. Three things make the 
window narrower than the condition, so the join does not emit a match, in 
either arrival order. Versions 1 to 3 emit it.

# *Casts that drop part of a time.* The solver reads {{CAST(ts AS BIGINT)}} as 
{{ts}}, although the cast drops the fraction of a second.
#* Example: {{CAST(rightTime AS BIGINT) >= CAST(leftTime AS BIGINT) + 60 AND 
CAST(rightTime AS BIGINT) <= CAST(leftTime AS BIGINT) + 120}}.
#* A left row at 100.6 seconds matches a right row at 160.2 seconds (160 >= 100 
+ 60), but the left row's window starts near 160.6 seconds.
#* The same happens for a cast to any integral type, to FLOAT, or to DECIMAL 
with fewer than 6 fractional digits.
# *Whole days and months.* The solver counts a day as 24 hours, but Spark adds 
days and months in the session time zone, where a day can have 23 or 25 hours.
#* Example: {{rightTime BETWEEN leftTime AND leftTime + INTERVAL 1 DAY}} with 
the session time zone America/Los_Angeles.
#* A right row 24.5 hours after a left row at noon on 2026-10-31 matches, 
because that day has 25 hours, but the left row's window ends after 24 hours.
# *Sub-millisecond offsets.* The solver returns whole milliseconds, truncated 
toward zero.
#* Example: {{rightTime > leftTime - INTERVAL 500 MICROSECONDS AND rightTime < 
leftTime + INTERVAL 1 SECOND}}.
#* The lower offset of -0.5 ms becomes 0, so the join misses a right row 0.3 ms 
before the left row.

State eviction is not affected by these issues in the cases above.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to