[
https://issues.apache.org/jira/browse/YARN-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13990220#comment-13990220
]
Bikas Saha commented on YARN-2001:
----------------------------------
What if users want to have multiple standbys for fault tolerance? In a large
10000 nodes cluster there could be 3-4 distinct fault domains where more than 1
standby may be good to guarantee availability. Until now, in the design we have
not restricted the number of standby's. Having all NM's ping all RM's will
cause a lot of communication overhead in a healthy cluster.
The design already encompasses NM's discovering and syncing with the new active
RM. So that is not the problem. The problem is restart during an upgrade where
it may be common that a bunch of NM's dont come back up. The RM needs to be
resilient to that while maintaining availability. Having a threshold of NM's
sounds like a reasonable solution. The threshold can be calculated based on the
scheduling margin of error wrt queue capacity.
At this point my suggestion would be to clarify the problem being addressed in
this jira. Is the problem that after RM failover, the new RM needs to have a
certain minimum number of machines join it before it can safely make scheduling
decisions? If thats the case then please update the title to reflect that
problem and not the solution.
> Persist NMs info for RM restart
> -------------------------------
>
> Key: YARN-2001
> URL: https://issues.apache.org/jira/browse/YARN-2001
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Reporter: Jian He
> Assignee: Jian He
>
> RM should not accept allocate requests from AMs until all the NMs have
> registered with RM. For that, RM needs to remember the previous NMs and wait
> for all the NMs to register.
> This is also useful for remembering decommissioned nodes across restarts.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
