xingbe created FLINK-31652:
------------------------------
Summary: Flink should handle the delete event if the pod was
deleted while pending
Key: FLINK-31652
URL: https://issues.apache.org/jira/browse/FLINK-31652
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Affects Versions: 1.16.1, 1.17.0
Reporter: xingbe
I found that in kubernetes deployment, if the taskmanager pod is deleted in
'Pending' phase, the flink job will get stuck and keep waiting for the pod
scheduled. We can reproduce this issue with the 'kubectl delete pod' command to
delete the pod when it is in the pending phase.
The cause reason is that the pod status will not be updated in time in this
case, so the KubernetesResourceManagerDriver won't detect the pod is
terminated, and I also verified this by logging the pod status in
KubernetesPod#isTerminated(), and it shows as follows.
{code:java}
public boolean isTerminated() {
log.info("pod status: " + getInternalResource().getStatus());
if (getInternalResource().getStatus() != null) {
final boolean podFailed =
PodPhase.Failed.name().equals(getInternalResource().getStatus().getPhase());
final boolean containersFailed =
getInternalResource().getStatus().getContainerStatuses().stream()
.anyMatch(
e ->
e.getState() != null
&& e.getState().getTerminated()
!= null);
return containersFailed || podFailed;
}
return false;
} {code}
In the case, this function will return false because `containersFailed` and
`podFailed` are both false.
{code:java}
PodStatus(conditions=[PodCondition(lastProbeTime=null,
lastTransitionTime=2023-03-28T12:35:10Z, reason=Unschedulable, status=False,
type=PodScheduled, additionalProperties={})], containerStatuses=[],
ephemeralContainerStatuses=[], hostIP=null, initContainerStatuses=[],
message=null, nominatedNodeName=null, phase=Pending, podIP=null, podIPs=[],
qosClass=Guaranteed, reason=null, startTime=null, additionalProperties={}){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
