[
https://issues.apache.org/jira/browse/YARN-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14321200#comment-14321200
]
Rohith commented on YARN-3194:
------------------------------
The current flow of notifying applications for completed containers are via RM.
# AM sends stopContainerRequest to NM
# NM kills the container and send containerStatus to RM in heartbeat ONLY ONCE.
NM keeps this container until RM piggybacks to NM saying container can be
removed.
# RM releases the resources which are allocated for containers. And sends back
to NM for removing container from NM's tracking.
# In AM heartbeat to AM, RM sends completed containers details to AM
ICO NM Restart, NM sends outstanding NMContainerStatus lists to RM during NM
registration. This status has boh RUNNING and COMPLETED containers.These
containers status never send again by RM. But on NM registration ,
RM(RMNodeImpl.AddNodeTransition#transition) is processing only RUNNING
containers. COMPLETED containers are ignored. The impact from this is Resources
which are being allocated to these containers never be released from RM. At
this time, RM state is completely inconsistent with actual cluster state.
Application is hanging because in the above inconsistent state, RM never
allocate new containers which are asked by AM.So AM will be keep waiting for
containers from RM.
Earlier to YARN-2997 fix, it should work fine because those completed
containers which are sent in NM registration are sent again in 1st heartbeat of
NM. So RMNodeImpl handles completed containers in nodeHeartBeat event. The bug
was hidden before yarn-2997 because NM was sending duplicated container status.
But actually RM should had handle it.
Does it make sense?
> After NM restart,completed containers are not released which are sent during
> NM registration
> --------------------------------------------------------------------------------------------
>
> Key: YARN-3194
> URL: https://issues.apache.org/jira/browse/YARN-3194
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.6.0
> Environment: NM restart is enabled
> Reporter: Rohith
> Assignee: Rohith
> Priority: Blocker
>
> On NM restart ,NM sends all the outstanding NMContainerStatus to RM. But RM
> process only ContainerState.RUNNING. If container is completed when NM was
> down then those containers resources wont be release which result in
> applications to hang.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
