[ 
https://issues.apache.org/jira/browse/SPARK-61513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61513:
-----------------------------------
    Labels: pull-request-available  (was: )

> Stream-stream join in state format version 4 misses matches because its range 
> scan window is too narrow
> -------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61513
>                 URL: https://issues.apache.org/jira/browse/SPARK-61513
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.2.0, 4.3.0, 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>
> In state format version 4, a stream-stream join row reads only the other 
> side's rows in a time window. Spark derives the window's offsets from the 
> join condition with the watermark solver, at a watermark of 0. Three things 
> make the window narrower than the condition, so the join does not emit a 
> match, in either arrival order. Versions 1 to 3 emit it.
> # *Casts that drop part of a time.* The solver reads {{CAST(ts AS BIGINT)}} 
> as {{ts}}, although the cast drops the fraction of a second.
> #* Example: {{CAST(rightTime AS BIGINT) >= CAST(leftTime AS BIGINT) + 60 AND 
> CAST(rightTime AS BIGINT) <= CAST(leftTime AS BIGINT) + 120}}.
> #* A left row at 100.6 seconds matches a right row at 160.2 seconds (160 >= 
> 100 + 60), but the left row's window starts near 160.6 seconds.
> #* The same happens for a cast to any integral type, to FLOAT, or to DECIMAL 
> with fewer than 6 fractional digits.
> # *Whole days and months.* The solver counts a day as 24 hours, but Spark 
> adds days and months in the session time zone, where a day can have 23 or 25 
> hours.
> #* Example: {{rightTime BETWEEN leftTime AND leftTime + INTERVAL 1 DAY}} with 
> the session time zone America/Los_Angeles.
> #* A right row 24.5 hours after a left row at noon on 2026-10-31 matches, 
> because that day has 25 hours, but the left row's window ends after 24 hours.
> # *Sub-millisecond offsets.* The solver returns whole milliseconds, truncated 
> toward zero.
> #* Example: {{rightTime > leftTime - INTERVAL 500 MICROSECONDS AND rightTime 
> < leftTime + INTERVAL 1 SECOND}}.
> #* The lower offset of -0.5 ms becomes 0, so the join misses a right row 0.3 
> ms before the left row.
> State eviction is not affected by these issues in the cases above.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to