Jungtaek Lim created SPARK-62038:
------------------------------------
Summary: Stream-stream join state format V4 never evicts rows that
an older version's cutoff left in state after a restart
Key: SPARK-62038
URL: https://issues.apache.org/jira/browse/SPARK-62038
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
With {{spark.sql.streaming.join.stateFormatVersion=4}}, a stream-stream join
restarted from an older checkpoint keeps state rows that the older version's
cutoff never evicted, after a fix makes them evictable. On each batch V4 starts
its eviction scan after the previous batch's cutoff, so rows at or below an old
cutoff are never scanned again. State formats 2 and 3 evaluate every stored row
each batch and evict them.
This shows after SPARK-61985, which removes a constant-only cap from the range
cutoff. Reproducer (left outer join, 1 second watermarks):
{code}
leftKey = rightKey AND lt > rt - INTERVAL 5 SECONDS AND CAST(lt AS LONG) > 10
{code}
1. On a build without SPARK-61985, run L(1, 20 s) R(9, 20 s); L(8, 40 s) R(9,
40 s); R(9, 41 s). Nothing is emitted, because the cutoff is capped at 10 s.
2. Restart from that checkpoint with SPARK-61985 and add L(8, 60 s) R(9, 60 s).
Expected: {{(1, 20, null)}} in the first batch after the restart. V2 and V3
emit it. V4 emits nothing, and the row stays in state.
A possible fix: on the first batch after a restart, scan the V4 timestamp index
from the start instead of after the previous batch's cutoff.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]