[
https://issues.apache.org/jira/browse/YARN-1372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14132206#comment-14132206
]
Jian He commented on YARN-1372:
-------------------------------
Thanks for updating the patch! I think we are very close,
- In
getContainerStatuses/getNMContainerStatuses/removeCompletedContainersFromContext,
calling {{context.getNMStateStore().removeContainer(containerId);}} will
remove containers from state-store before containers expired in
recentlyStoppedContainers. I think we should, in the while loop of
removeVeryOldStoppedContainersFromCache, check if container exists in context
or not. if it doesn’t exist, remove from state-store.
- NodeHeartBeatResponse#addFinishedContainersPulledByAM, could you add comments
about why we adding this new API? as this is not obvious to other new people.
- Thought more about finishedContainersSentToAM in BaseFinalTransition. I think
we should always transfer the finishedContainersSentToAM to the next attempt
regardless work-preserving AM restart is enabled or not, so that the restarted
AM can continuously ack previously finishedContainers, in case the previous AM
crashes. Otherwise, the not-yet-acked containers will remain on NM for the
lifeTime of application instead of attempt. And then we can clear
justFinishedContainers and finishedContainersSentToAM in the
RMAppImpl#FinalTransition as they are not needed anymore. your opinion ?
Tests:
- Could you also explicitly check the previous containerPulledByAM doesn’t exist
{code}
Assert.assertEquals(statuses.size(), 3);
Assert.assertEquals(context.getContainers().size(), 3);
{code}
-
testPreviousCompletedContainers->testRemovePreviousCompletedContainersFromContext
- Tests covered that containers are removed from context, only if it gets the
notification from RM. I think we need one more unit test on RM side that, the
containersPulledByAM won’t be sent in the nodeHeartBeatResponse only if AM
explicitly calls allocate to ack these finishedContainers.
- missing brackets for the for loop and also the indentation.
{code}
for (Container container: containers.values())
Assert.assertEquals(ContainerState.COMPLETE,
container.cloneAndGetContainerStatus().getState());
{code}
> Ensure all completed containers are reported to the AMs across RM restart
> -------------------------------------------------------------------------
>
> Key: YARN-1372
> URL: https://issues.apache.org/jira/browse/YARN-1372
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Reporter: Bikas Saha
> Assignee: Anubhav Dhoot
> Attachments: YARN-1372.001.patch, YARN-1372.001.patch,
> YARN-1372.002_NMHandlesCompletedApp.patch,
> YARN-1372.002_RMHandlesCompletedApp.patch,
> YARN-1372.002_RMHandlesCompletedApp.patch, YARN-1372.003.patch,
> YARN-1372.004.patch, YARN-1372.005.patch, YARN-1372.005.patch,
> YARN-1372.prelim.patch, YARN-1372.prelim2.patch
>
>
> Currently the NM informs the RM about completed containers and then removes
> those containers from the RM notification list. The RM passes on that
> completed container information to the AM and the AM pulls this data. If the
> RM dies before the AM pulls this data then the AM may not be able to get this
> information again. To fix this, NM should maintain a separate list of such
> completed container notifications sent to the RM. After the AM has pulled the
> containers from the RM then the RM will inform the NM about it and the NM can
> remove the completed container from the new list. Upon re-register with the
> RM (after RM restart) the NM should send the entire list of completed
> containers to the RM along with any other containers that completed while the
> RM was dead. This ensures that the RM can inform the AM's about all completed
> containers. Some container completions may be reported more than once since
> the AM may have pulled the container but the RM may die before notifying the
> NM about the pull.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
