Jungtaek Lim created SPARK-62003:
------------------------------------
Summary: Stream-stream join evicts a time window that a longer
window of the other input can still match, when the condition compares whole
windows
Key: SPARK-62003
URL: https://issues.apache.org/jira/browse/SPARK-62003
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
For a join condition that compares window structs, such as {{lw > rw}}, Spark
derives the left state cutoff from the right watermark and evicts a left window
when its end reaches that cutoff. Window structs compare by start first, so
when the right windows are longer than the left windows, an on-time right
window can start before a left window that has already ended. Inner joins lose
the match, and outer joins emit an early null row.
Reproducer (0-second watermarks; 10-second tumbling left windows, 20-second
right windows sliding by 1 second):
{code:scala}
left.join(right, $"leftKey" === $"rightKey" && $"lw" > $"rw")
{code}
Left (1, 5 s) is in window [0, 10). Rows at 15 s move the watermark to 15 s,
and Spark evicts [0, 10). Right (1, 18 s) falls in window [-1, 19), which is
not late and orders below [0, 10), so the expected output is (1, [0,10),
[-1,19)). Spark emits nothing. A left outer join emits (1, [0,10), null)
instead.
Affects state format versions 2 to 4.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]