[
https://issues.apache.org/jira/browse/FLINK-29396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17608814#comment-17608814
]
Matthias Pohl commented on FLINK-29396:
---------------------------------------
I guess, you have a point here. Initially, I thought that there's a message
back from the {{ResourceManager}} to the {{JobMaster}} missing. That would then
trigger the shutdown of the {{JobMaster}} and, as a consequence, trigger the
stopping of the corresponding {{JobMasterServiceLeadershipRunner}} (in
[JobMasterServiceLeadershipRunner:126|https://github.com/apache/flink/blob/e8a91fd8428e417c63b299392a84f7df9d10ddb8/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMasterServiceLeadershipRunner.java#L126]).
But this message is actually send in
[ResourceManager#closeJobManagerConnection:1072|https://github.com/apache/flink/blob/0263b55288be7b569f56dd42a94c5e48bcc1607b/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java#L1072]
Essentially, we would have to instantiate a {{CompletableFuture}} in
[JobMaster#stopExecution:1022|https://github.com/apache/flink/blob/b7dd42617a46fcecfffbea3409391e204a40b9b1/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java#L1022],
compose it with the {{terminationFuture}} there and let this future be
completed in
[JobMaster#disconnectResourceManager|https://github.com/apache/flink/blob/b7dd42617a46fcecfffbea3409391e204a40b9b1/flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/JobMaster.java#L817].
That will make the JobMaster shutdown process proceed after we got the
confirmation from the {{ResourceManager}} that the disconnect succeeded. WDYT?
> Race condition in JobMaster shutdown can leak resource requirements
> -------------------------------------------------------------------
>
> Key: FLINK-29396
> URL: https://issues.apache.org/jira/browse/FLINK-29396
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.15.0
> Reporter: Chesnay Schepler
> Priority: Blocker
>
> When a JobMaster is stopped it
> a) sends a message to the RM informing it of the final job status
> b) removes itself as the leader.
> Once the JM loses leadership the RM is also informed about that.
> With that we have 2 messages being sent to the RM at about the same time.
> If the shutdown notifications arrives first (and job is in a terminal state)
> we wipe the resource requirements, and the leader loss notification is
> effectively ignored.
> If the leader loss notification arrives first we keep the resource
> requirements, assuming that another JM will pick the job up later on, and the
> shutdown notification will be ignored.
> This can cause a session cluster to essentially do nothing until the job
> timeout is triggered due to no leader being present (default 5 minutes).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
