[ 
https://issues.apache.org/jira/browse/SPARK-62001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-62001:
---------------------------------
    Affects Version/s: 4.3.0
                       4.4.0

> Stream-stream join forwards a watermark that is too high when its state can 
> still emit older rows, so a downstream aggregation drops join output as late
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62001
>                 URL: https://issues.apache.org/jira/browse/SPARK-62001
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.3.0, 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> A stream-stream join computes its output watermark from the state cutoff of 
> each input and its input watermark. If one input's stored rows have no 
> cutoff, for example a join on the key only, or a range condition that bounds 
> only the other input, that input is left out of the calculation. The join 
> then forwards its full input watermark, even though a later row can still 
> join a stored row and emit an old event time.
> A downstream stateful operator in append mode, such as a window aggregation, 
> then treats these rows as late. It drops them after it has already emitted 
> the window as final, so the final result is wrong.
> Reproducer (both inputs have a 10-second watermark):
> {code:scala}
> left.join(right, "key")
>   .groupBy(window($"leftTime", "10 seconds")).count()   // append mode
> {code}
> Left (1, 5 s), (2, 5 s); right (2, 5 s). Then rows at 30 s move the watermark 
> to 20 s, and the aggregation emits window [0, 10) with count 1. Then right 
> (1, 40 s) joins the stored left (1, 5 s), and the aggregation drops that row 
> as late. The batch answer is count 2.
> The same happens with {{clickTime <= impressionTime + INTERVAL 10 SECONDS}} 
> and a window on {{clickTime}}.
> Affects state format versions 2 to 4.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to