Jungtaek Lim created SPARK-62001:
------------------------------------

             Summary: Stream-stream join forwards a watermark that is too high 
when its state can still emit older rows, so a downstream aggregation drops 
join output as late
                 Key: SPARK-62001
                 URL: https://issues.apache.org/jira/browse/SPARK-62001
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


A stream-stream join computes its output watermark from the state cutoff of 
each input and its input watermark. If one input's stored rows have no cutoff, 
for example a join on the key only, or a range condition that bounds only the 
other input, that input is left out of the calculation. The join then forwards 
its full input watermark, even though a later row can still join a stored row 
and emit an old event time.

A downstream stateful operator in append mode, such as a window aggregation, 
then treats these rows as late. It drops them after it has already emitted the 
window as final, so the final result is wrong.

Reproducer (both inputs have a 10-second watermark):
{code:scala}
left.join(right, "key")
  .groupBy(window($"leftTime", "10 seconds")).count()   // append mode
{code}
Left (1, 5 s), (2, 5 s); right (2, 5 s). Then rows at 30 s move the watermark 
to 20 s, and the aggregation emits window [0, 10) with count 1. Then right (1, 
40 s) joins the stored left (1, 5 s), and the aggregation drops that row as 
late. The batch answer is count 2.

The same happens with {{clickTime <= impressionTime + INTERVAL 10 SECONDS}} and 
a window on {{clickTime}}.

Affects state format versions 2 to 4.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to