[ 
https://issues.apache.org/jira/browse/SPARK-62038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated SPARK-62038:
-----------------------------------
    Labels: pull-request-available  (was: )

> Stream-stream join state format V4 never evicts rows that an older version's 
> cutoff left in state after a restart
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62038
>                 URL: https://issues.apache.org/jira/browse/SPARK-62038
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>              Labels: pull-request-available
>
> With {{spark.sql.streaming.join.stateFormatVersion=4}}, a stream-stream join 
> restarted from an older checkpoint keeps state rows that the older version's 
> cutoff never evicted, after a fix makes them evictable. On each batch V4 
> starts its eviction scan after the previous batch's cutoff, so rows at or 
> below an old cutoff are never scanned again. State formats 2 and 3 evaluate 
> every stored row each batch and evict them.
> This shows after SPARK-61985, which removes a constant-only cap from the 
> range cutoff. Reproducer (left outer join, 1 second watermarks):
> {code}
> leftKey = rightKey AND lt > rt - INTERVAL 5 SECONDS AND CAST(lt AS LONG) > 10
> {code}
> 1. On a build without SPARK-61985, run L(1, 20 s) R(9, 20 s); L(8, 40 s) R(9, 
> 40 s); R(9, 41 s). Nothing is emitted, because the cutoff is capped at 10 s.
> 2. Restart from that checkpoint with SPARK-61985 and add L(8, 60 s) R(9, 60 
> s).
> Expected: {{(1, 20, null)}} in the first batch after the restart. V2 and V3 
> emit it. V4 emits nothing, and the row stays in state.
> A possible fix: on the first batch after a restart, scan the V4 timestamp 
> index from the start instead of after the previous batch's cutoff.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to