[
https://issues.apache.org/jira/browse/FLINK-40093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-40093.
----------------------------------
Fix Version/s: 2.2.2
2.3.1
2.4.0
Resolution: Fixed
merged commit
[{{a8835bd}}|https://github.com/apache/flink/commit/a8835bd37312af00d6669d3ebe4caca70cf3f220]
into apache:master
merged commit
[{{2a2d359}}|https://github.com/apache/flink/commit/2a2d35906ceb27e1aae2ca715da2aa24c9189bb7]
into apache:release-2.3
merged commit
[{{342fb95}}|https://github.com/apache/flink/commit/342fb95492144f810239bd3f0b8d46389f466df9]
into apache:release-2.2
> Race condition between watermark alignment and idleness detection locks split
> in idle
> -------------------------------------------------------------------------------------
>
> Key: FLINK-40093
> URL: https://issues.apache.org/jira/browse/FLINK-40093
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
> Reporter: Efrat Levitan
> Assignee: Efrat Levitan
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.2.2, 2.3.1, 2.4.0
>
>
> Usually pauseOrResumeSplits pauses idleness timer for the split so it isn't
> marked idle while paused. However with low idleness timeout (observed with
> 1s) + low allowed WM drift, a race condition could cause paused splits to
> never resume though they have records:
> # A split becomes paused due to too advanced records.
> # pauseOrResumeSplits pauses the split.
> # pauseOrResumeSplits reaches to pause the split idleness clock but is
> \{idlenessTimeout} too late, and the split becomes idle.
> # The watermark advances but the split is excluded from the watermark
> alignment check due to its idleness.
> # More records arrive but the split is paused at the connector level so they
> aren't processed, nor seen by watermarkGenerator so it still considers the
> split idle
> A possible fix could be preserving the part where idle splits are excluded
> from alignment pause (to not override their idle status) while allowing
> alignment check to resume splits even if they are currently idle. They are
> considered idle until they emit the next record.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
