[
https://issues.apache.org/jira/browse/SPARK-62001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62001:
-----------------------------------
Labels: pull-request-available (was: )
> Stream-stream join forwards a watermark that is too high when its state can
> still emit older rows, so a downstream aggregation drops join output as late
> --------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62001
> URL: https://issues.apache.org/jira/browse/SPARK-62001
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0, 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> A stream-stream join computes its output watermark from the state cutoff of
> each input and its input watermark. If one input's stored rows have no
> cutoff, for example a join on the key only, or a range condition that bounds
> only the other input, that input is left out of the calculation. The join
> then forwards its full input watermark, even though a later row can still
> join a stored row and emit an old event time.
> A downstream stateful operator in append mode, such as a window aggregation,
> then treats these rows as late. It drops them after it has already emitted
> the window as final, so the final result is wrong.
> Reproducer (both inputs have a 10-second watermark):
> {code:scala}
> left.join(right, "key")
> .groupBy(window($"leftTime", "10 seconds")).count() // append mode
> {code}
> Left (1, 5 s), (2, 5 s); right (2, 5 s). Then rows at 30 s move the watermark
> to 20 s, and the aggregation emits window [0, 10) with count 1. Then right
> (1, 40 s) joins the stored left (1, 5 s), and the aggregation drops that row
> as late. The batch answer is count 2.
> The same happens with {{clickTime <= impressionTime + INTERVAL 10 SECONDS}}
> and a window on {{clickTime}}.
> Affects state format versions 2 to 4.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]