[ 
https://issues.apache.org/jira/browse/SPARK-62003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-62003:
---------------------------------
    Affects Version/s: 4.3.0
                       4.4.0

> Stream-stream join evicts a time window that a longer window of the other 
> input can still match, when the condition compares whole windows
> ------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62003
>                 URL: https://issues.apache.org/jira/browse/SPARK-62003
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.3.0, 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> For a join condition that compares window structs, such as {{lw > rw}}, Spark 
> derives the left state cutoff from the right watermark and evicts a left 
> window when its end reaches that cutoff. Window structs compare by start 
> first, so when the right windows are longer than the left windows, an on-time 
> right window can start before a left window that has already ended. Inner 
> joins lose the match, and outer joins emit an early null row.
> Reproducer (0-second watermarks; 10-second tumbling left windows, 20-second 
> right windows sliding by 1 second):
> {code:scala}
> left.join(right, $"leftKey" === $"rightKey" && $"lw" > $"rw")
> {code}
> Left (1, 5 s) is in window [0, 10). Rows at 15 s move the watermark to 15 s, 
> and Spark evicts [0, 10). Right (1, 18 s) falls in window [-1, 19), which is 
> not late and orders below [0, 10), so the expected output is (1, [0,10), 
> [-1,19)). Spark emits nothing. A left outer join emits (1, [0,10), null) 
> instead.
> Affects state format versions 2 to 4.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to