[
https://issues.apache.org/jira/browse/SPARK-62003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-62003:
---------------------------------
Affects Version/s: 4.3.0
4.4.0
> Stream-stream join evicts a time window that a longer window of the other
> input can still match, when the condition compares whole windows
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62003
> URL: https://issues.apache.org/jira/browse/SPARK-62003
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0, 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> For a join condition that compares window structs, such as {{lw > rw}}, Spark
> derives the left state cutoff from the right watermark and evicts a left
> window when its end reaches that cutoff. Window structs compare by start
> first, so when the right windows are longer than the left windows, an on-time
> right window can start before a left window that has already ended. Inner
> joins lose the match, and outer joins emit an early null row.
> Reproducer (0-second watermarks; 10-second tumbling left windows, 20-second
> right windows sliding by 1 second):
> {code:scala}
> left.join(right, $"leftKey" === $"rightKey" && $"lw" > $"rw")
> {code}
> Left (1, 5 s) is in window [0, 10). Rows at 15 s move the watermark to 15 s,
> and Spark evicts [0, 10). Right (1, 18 s) falls in window [-1, 19), which is
> not late and orders below [0, 10), so the expected output is (1, [0,10),
> [-1,19)). Spark emits nothing. A left outer join emits (1, [0,10), null)
> instead.
> Affects state format versions 2 to 4.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]