Jungtaek Lim created SPARK-61344:
------------------------------------
Summary: Fix stream-stream join range eviction column and output
watermark
Key: SPARK-61344
URL: https://issues.apache.org/jira/browse/SPARK-61344
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
Stream-stream joins can remove state too early and forward an output watermark
that is too high.
1. Range eviction uses the wrong column. For {{leftKey = rightKey AND rightTime
BETWEEN leftTime AND expiry}}, a left row can still match until {{expiry}}
passes the right watermark. Spark applies that cutoff to the left event-time
column {{leftTime}} instead. The row is removed while it can still match. A
later valid match is lost, and a left outer join can emit a premature null row.
This affects state formats V2, V3 and V4.
2. The output watermark ignores what the state can still emit. An inner join on
a non-time key can keep a left row at event time 5 while the input watermark
advances to 20, and then match a right row at time 100. The join forwards
watermark 20, so a downstream deduplication or window aggregate drops the valid
output at time 5. The same happens when the event-time equality keys are
crossed, for example {{la = ra AND lb = rb}} with watermarks on {{la}} and
{{rb}}.
Fixing this changes which rows the state keeps. A checkpoint written by an
older version may not be safe to reuse for affected join shapes. Spark must
detect that and reject the checkpoint with a clear error, instead of resuming
with incorrect state.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]