Jungtaek Lim created SPARK-61988:
------------------------------------
Summary: Stream-stream join evicts range state by the event time
instead of the bounded column
Key: SPARK-61988
URL: https://issues.apache.org/jira/browse/SPARK-61988
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
For an input with its own watermark, the join derives the range cutoff from
whichever column the condition bounds (for example {{expiry}} in
{{rightTime BETWEEN leftTime AND expiry}}) but evicts rows by the event time.
Outer joins emit
unmatched rows too early and lose later matches; inner and semi joins lose
matches.
Reproducer: left outer join above, left {{(1, 100 s, 700 s)}}; at watermark 340
s Spark emits
{{(1, null)}} and drops {{(1, 460 s)}}. Fix: evict by the bounded TIMESTAMP
column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]