Jungtaek Lim created SPARK-62001:
------------------------------------
Summary: Stream-stream join forwards a watermark that is too high
when its state can still emit older rows, so a downstream aggregation drops
join output as late
Key: SPARK-62001
URL: https://issues.apache.org/jira/browse/SPARK-62001
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
A stream-stream join computes its output watermark from the state cutoff of
each input and its input watermark. If one input's stored rows have no cutoff,
for example a join on the key only, or a range condition that bounds only the
other input, that input is left out of the calculation. The join then forwards
its full input watermark, even though a later row can still join a stored row
and emit an old event time.
A downstream stateful operator in append mode, such as a window aggregation,
then treats these rows as late. It drops them after it has already emitted the
window as final, so the final result is wrong.
Reproducer (both inputs have a 10-second watermark):
{code:scala}
left.join(right, "key")
.groupBy(window($"leftTime", "10 seconds")).count() // append mode
{code}
Left (1, 5 s), (2, 5 s); right (2, 5 s). Then rows at 30 s move the watermark
to 20 s, and the aggregation emits window [0, 10) with count 1. Then right (1,
40 s) joins the stored left (1, 5 s), and the aggregation drops that row as
late. The batch answer is count 2.
The same happens with {{clickTime <= impressionTime + INTERVAL 10 SECONDS}} and
a window on {{clickTime}}.
Affects state format versions 2 to 4.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]