Jian He commented on YARN-1372:

bq. maybe we can just use NodeHeartbeatResponse#getContainersToCleanup to 
notify NM to remove the containers from the context. 
I was thinking to reuse getContainersToCleanup to notify NM to remove 
containers. i.e. do not clean up the containers until gets pulled by AM. On a 
second thought, this is not a good solution, please disregard this comment. 
sorry for confusion. 

Comments on the patch:
bq. Second patch uploaded that adds expiration to the entries in NM
How about tie the NM container lifecycle to the application itself ?  i.e. 
Clean all containers in context for each application in 

- DECOMMISSIONED/LOST state possible to receive the new event?
- In RMAppAttemptImpl, why add a new previousJustFinishedContainers?
- RMContainerImpl already has the nodeId, many changes related to adding the 
NodeId are not needed. e.g. scheduler changes, RMContainerRecoverEvent changes.
- please add tests too.

> Ensure all completed containers are reported to the AMs across RM restart
> -------------------------------------------------------------------------
>                 Key: YARN-1372
>                 URL: https://issues.apache.org/jira/browse/YARN-1372
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Bikas Saha
>            Assignee: Anubhav Dhoot
>         Attachments: YARN-1372.prelim.patch, YARN-1372.prelim2.patch
> Currently the NM informs the RM about completed containers and then removes 
> those containers from the RM notification list. The RM passes on that 
> completed container information to the AM and the AM pulls this data. If the 
> RM dies before the AM pulls this data then the AM may not be able to get this 
> information again. To fix this, NM should maintain a separate list of such 
> completed container notifications sent to the RM. After the AM has pulled the 
> containers from the RM then the RM will inform the NM about it and the NM can 
> remove the completed container from the new list. Upon re-register with the 
> RM (after RM restart) the NM should send the entire list of completed 
> containers to the RM along with any other containers that completed while the 
> RM was dead. This ensures that the RM can inform the AM's about all completed 
> containers. Some container completions may be reported more than once since 
> the AM may have pulled the container but the RM may die before notifying the 
> NM about the pull.

This message was sent by Atlassian JIRA

Reply via email to