Jungtaek Lim created SPARK-61990:
------------------------------------
Summary: Stream-stream join never evicts range state of an input
without its own watermark
Key: SPARK-61990
URL: https://issues.apache.org/jira/browse/SPARK-61990
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
The join derives a range cutoff for an input only if it has a watermark of its
own, although the cutoff depends only on the other input's watermark. Left
outer join with a
watermark only on {{rightTime}} and {{leftTime BETWEEN rightTime - 5 s AND
rightTime + 5 s}}:
left {{(1, 5 s)}} is never emitted as {{(1, null)}} and the left state grows
forever. Fix: derive
the cutoff for such an input by its bounded TIMESTAMP column.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]