[
https://issues.apache.org/jira/browse/SPARK-62038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62038:
-----------------------------------
Labels: pull-request-available (was: )
> Stream-stream join state format V4 never evicts rows that an older version's
> cutoff left in state after a restart
> -----------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62038
> URL: https://issues.apache.org/jira/browse/SPARK-62038
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> With {{spark.sql.streaming.join.stateFormatVersion=4}}, a stream-stream join
> restarted from an older checkpoint keeps state rows that the older version's
> cutoff never evicted, after a fix makes them evictable. On each batch V4
> starts its eviction scan after the previous batch's cutoff, so rows at or
> below an old cutoff are never scanned again. State formats 2 and 3 evaluate
> every stored row each batch and evict them.
> This shows after SPARK-61985, which removes a constant-only cap from the
> range cutoff. Reproducer (left outer join, 1 second watermarks):
> {code}
> leftKey = rightKey AND lt > rt - INTERVAL 5 SECONDS AND CAST(lt AS LONG) > 10
> {code}
> 1. On a build without SPARK-61985, run L(1, 20 s) R(9, 20 s); L(8, 40 s) R(9,
> 40 s); R(9, 41 s). Nothing is emitted, because the cutoff is capped at 10 s.
> 2. Restart from that checkpoint with SPARK-61985 and add L(8, 60 s) R(9, 60
> s).
> Expected: {{(1, 20, null)}} in the first batch after the restart. V2 and V3
> emit it. V4 emits nothing, and the row stays in state.
> A possible fix: on the first batch after a restart, scan the V4 timestamp
> index from the start instead of after the previous batch's cutoff.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]