Efrat Levitan created FLINK-40093:
-------------------------------------
Summary: Race condition between watermark alignment and idleness
detection locks split in idle
Key: FLINK-40093
URL: https://issues.apache.org/jira/browse/FLINK-40093
Project: Flink
Issue Type: Bug
Components: Runtime / Task
Reporter: Efrat Levitan
Usually pauseOrResumeSplits pauses idleness timer for the split so it isn't
marked idle while paused. However with low idleness timeout (observed with 1s)
+ low allowed WM drift, a race condition could cause paused splits to never
resume though they have records:
# A split becomes paused due to too advanced records.
# pauseOrResumeSplits pauses the split.
# pauseOrResumeSplits reaches to pause the split idleness clock but is
\{idlenessTimeout} too late, and the split becomes idle.
# The watermark advances but the split is excluded from the watermark
alignment check due to its idleness.
# More records arrive but the split is paused at the connector level so they
aren't processed, nor seen by watermarkGenerator so it still considers the
split idle
A possible fix could be preserving the part where idle splits are excluded from
alignment pause (to not override their idle status) while allowing alignment
check to resume splits even if they are currently idle. They are considered
idle until they emit the next record.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
