[ 
https://issues.apache.org/jira/browse/SPARK-61344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-61344:
-----------------------------------
    Labels: pull-request-available  (was: )

> Fix stream-stream join range eviction column and output watermark
> -----------------------------------------------------------------
>
>                 Key: SPARK-61344
>                 URL: https://issues.apache.org/jira/browse/SPARK-61344
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 5.0.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>
> Stream-stream joins can remove state too early and forward an output 
> watermark that is too high.
> 1. Range eviction uses the wrong column. For {{leftKey = rightKey AND 
> rightTime BETWEEN leftTime AND expiry}}, a left row can still match until 
> {{expiry}} passes the right watermark. Spark applies that cutoff to the left 
> event-time column {{leftTime}} instead. The row is removed while it can still 
> match. A later valid match is lost, and a left outer join can emit a 
> premature null row. This affects state formats V2, V3 and V4.
> 2. The output watermark ignores what the state can still emit. An inner join 
> on a non-time key can keep a left row at event time 5 while the input 
> watermark advances to 20, and then match a right row at time 100. The join 
> forwards watermark 20, so a downstream deduplication or window aggregate 
> drops the valid output at time 5. The same happens when the event-time 
> equality keys are crossed, for example {{la = ra AND lb = rb}} with 
> watermarks on {{la}} and {{rb}}.
> Fixing this changes which rows the state keeps. A checkpoint written by an 
> older version may not be safe to reuse for affected join shapes. Spark must 
> detect that and reject the checkpoint with a clear error, instead of resuming 
> with incorrect state.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to