[
https://issues.apache.org/jira/browse/FLINK-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17267804#comment-17267804
]
Yang Wang commented on FLINK-21008:
-----------------------------------
You are right. Deregistering the application from K8s(aka delete the JobManager
deployment) will let the kubelet send a SIGTERM to JobManager process. But the
Yarn has the same behavior. The reason why we do not come into this issue when
deploying Flink application on Yarn is that the SIGTERM is sent a little late.
Because Yarn ResourceManager tell NodeManager to kill(SIGTERM and followed by a
SIGKILL) the JobManager via heartbeat, which is 3 seconds by default. However,
on Kubernetes, kubelet is informed via watcher, which is no delay.
Assume that the cluster entrypoint costs more that 3 second for the internal
clean up( {{stopClusterServices}} and {{cleanupDirectories}}), we will run into
the same situation on Yarn deployment.
> ClusterEntrypoint#shutDownAsync may not be fully executed
> ---------------------------------------------------------
>
> Key: FLINK-21008
> URL: https://issues.apache.org/jira/browse/FLINK-21008
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.11.3, 1.12.1
> Reporter: Yang Wang
> Priority: Critical
> Fix For: 1.13.0
>
>
> Recently, in our internal use case for native K8s integration with K8s HA
> enabled, we found that the leader related ConfigMaps could be residual in
> some corner situations.
> After some investigations, I think it is possibly caused by the inappropriate
> shutdown process.
> In {{ClusterEntrypoint#shutDownAsync}}, we first call the
> {{closeClusterComponent}}, which also includes deregistering the Flink
> application from cluster management(e.g. Yarn, K8s). Then we call the
> {{stopClusterServices}} and {{cleanupDirectories}}. Imagine that the cluster
> management do the deregister very fast, the JobManager process receives
> SIGNAL 15 before or is being executing the {{stopClusterServices}} and
> {{cleanupDirectories}}. The jvm process will directly exit then. So the two
> methods may not be executed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
