[
https://issues.apache.org/jira/browse/FLINK-40093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18094544#comment-18094544
]
Stepan Stepanishchev commented on FLINK-40093:
----------------------------------------------
[~efrat19] hi, thank you for the ticket. I had some time to look into it today.
The problem is markPaused() on split metric group will resume idleSplit, but
won't remove split from currentlyIdleSplits. So this should work:
{code:java}
private void reportPausedOrResumed(
Collection<String> splitsToPause, Collection<String> splitsToResume) {
InternalSourceSplitMetricGroup metricGroup;
for (String splitId : splitsToResume) {
metricGroup = getOrCreateSplitMetricGroup(splitId);
metricGroup.markNotPaused();
}
for (String splitId : splitsToPause) {
metricGroup = getOrCreateSplitMetricGroup(splitId);
if (metricGroup.isIdle()) {
currentlyIdleSplits.remove(splitId);
}
metricGroup.markPaused();
}
}{code}
I'm writing a test in SourceOperatorSplitWatermarkAlignmentTest and almost done.
Would it be okay if I open a PR for this issue?
> Race condition between watermark alignment and idleness detection locks split
> in idle
> -------------------------------------------------------------------------------------
>
> Key: FLINK-40093
> URL: https://issues.apache.org/jira/browse/FLINK-40093
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
> Reporter: Efrat Levitan
> Assignee: Efrat Levitan
> Priority: Major
>
> Usually pauseOrResumeSplits pauses idleness timer for the split so it isn't
> marked idle while paused. However with low idleness timeout (observed with
> 1s) + low allowed WM drift, a race condition could cause paused splits to
> never resume though they have records:
> # A split becomes paused due to too advanced records.
> # pauseOrResumeSplits pauses the split.
> # pauseOrResumeSplits reaches to pause the split idleness clock but is
> \{idlenessTimeout} too late, and the split becomes idle.
> # The watermark advances but the split is excluded from the watermark
> alignment check due to its idleness.
> # More records arrive but the split is paused at the connector level so they
> aren't processed, nor seen by watermarkGenerator so it still considers the
> split idle
> A possible fix could be preserving the part where idle splits are excluded
> from alignment pause (to not override their idle status) while allowing
> alignment check to resume splits even if they are currently idle. They are
> considered idle until they emit the next record.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
