Jungtaek Lim created SPARK-61988:
------------------------------------

             Summary: Stream-stream join evicts range state by the event time 
instead of the bounded column
                 Key: SPARK-61988
                 URL: https://issues.apache.org/jira/browse/SPARK-61988
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


For an input with its own watermark, the join derives the range cutoff from
whichever column the condition bounds (for example {{expiry}} in
{{rightTime BETWEEN leftTime AND expiry}}) but evicts rows by the event time. 
Outer joins emit
unmatched rows too early and lose later matches; inner and semi joins lose 
matches.
Reproducer: left outer join above, left {{(1, 100 s, 700 s)}}; at watermark 340 
s Spark emits
{{(1, null)}} and drops {{(1, 460 s)}}. Fix: evict by the bounded TIMESTAMP 
column.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to