Jungtaek Lim created SPARK-61990:
------------------------------------

             Summary: Stream-stream join never evicts range state of an input 
without its own watermark
                 Key: SPARK-61990
                 URL: https://issues.apache.org/jira/browse/SPARK-61990
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


The join derives a range cutoff for an input only if it has a watermark of its
own, although the cutoff depends only on the other input's watermark. Left 
outer join with a
watermark only on {{rightTime}} and {{leftTime BETWEEN rightTime - 5 s AND 
rightTime + 5 s}}:
left {{(1, 5 s)}} is never emitted as {{(1, null)}} and the left state grows 
forever. Fix: derive
the cutoff for such an input by its bounded TIMESTAMP column.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to