[
https://issues.apache.org/jira/browse/YARN-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000229#comment-15000229
]
Varun Vasudev commented on YARN-4344:
-------------------------------------
An example of a situation is shown below -
{code}
2015-11-09 10:43:51,784 INFO resourcemanager.ResourceTrackerService
(ResourceTrackerService.java:registerNodeManager(345)) - NodeManager from node
10.0.0.64(cmPort: 30050 httpPort: 30060) registered with capability:
<memory:5632, vCores:8>, assigned nodeId 10.0.0.64:30050
2015-11-09 10:43:51,786 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(434)) -
10.0.0.64:30050 Node Transitioned from NEW to RUNNING
2015-11-09 10:43:51,814 INFO capacity.CapacityScheduler
(CapacityScheduler.java:addNode(1193)) - Added node 10.0.0.64:30050
clusterResource: <memory:5632, vCores:8>
2015-11-09 10:44:37,878 INFO util.RackResolver
(RackResolver.java:coreResolve(109)) - Resolved 10.0.0.63 to /default-rack
2015-11-09 10:44:37,879 INFO resourcemanager.ResourceTrackerService
(ResourceTrackerService.java:registerNodeManager(345)) - NodeManager from node
10.0.0.63(cmPort: 30050 httpPort: 30060) registered with capability:
<memory:10240, vCores:4>, assigned nodeId 10.0.0.63:30050
2015-11-09 10:44:37,879 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(434)) -
10.0.0.63:30050 Node Transitioned from NEW to RUNNING
2015-11-09 10:44:37,882 INFO capacity.CapacityScheduler
(CapacityScheduler.java:addNode(1193)) - Added node 10.0.0.63:30050
clusterResource: <memory:15872, vCores:12>
2015-11-09 10:44:39,307 INFO util.RackResolver
(RackResolver.java:coreResolve(109)) - Resolved 10.0.0.64 to /default-rack
2015-11-09 10:44:39,309 INFO resourcemanager.ResourceTrackerService
(ResourceTrackerService.java:registerNodeManager(313)) - Reconnect from the
node at: 10.0.0.64
2015-11-09 10:44:39,312 INFO resourcemanager.ResourceTrackerService
(ResourceTrackerService.java:registerNodeManager(345)) - NodeManager from node
10.0.0.64(cmPort: 30050 httpPort: 30060) registered with capability:
<memory:10240, vCores:4>, assigned nodeId 10.0.0.64:30050
2015-11-09 10:44:39,314 INFO capacity.CapacityScheduler
(CapacityScheduler.java:removeNode(1247)) - Removed node 10.0.0.64:30050
clusterResource: <memory:5632, vCores:8>
2015-11-09 10:44:39,315 INFO capacity.CapacityScheduler
(CapacityScheduler.java:addNode(1193)) - Added node 10.0.0.64:30050
clusterResource: <memory:15872, vCores:12>
{code}
In this case - NM's from 10.0.0.64 and 10.0.0.63 registered leading to a total
cluster resource of clusterResource: <memory:15872, vCores:12>. After that
10.0.0.64 re-connected with changed capabilities(from <memory:5632, vCores:8>
to <memory:10240, vCores:4>). This should have led to the cluster resources
becoming <memory:20480, vcores:8> but instead it is calculated to be
<memory:15872, vCores:12>.
The root cause is this piece of code from RMNodeImpl -
{code}
rmNode.context.getDispatcher().getEventHandler().handle(
new NodeRemovedSchedulerEvent(rmNode));
if (!rmNode.getTotalCapability().equals(
newNode.getTotalCapability())) {
rmNode.totalCapability = newNode.getTotalCapability();
{code}
If the dispatcher is delayed in its processing of the event, by the time the
remove node is processed, rmNode.totalCapability =
newNode.getTotalCapability() has already been executed and the resources that
are removed are the changed capabilities and not the older capabilities of the
node.
> NMs reconnecting with changed capabilities can lead to wrong cluster resource
> calculations
> ------------------------------------------------------------------------------------------
>
> Key: YARN-4344
> URL: https://issues.apache.org/jira/browse/YARN-4344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.7.1, 2.6.2
> Reporter: Varun Vasudev
> Assignee: Varun Vasudev
> Priority: Critical
>
> After YARN-3802, if an NM re-connects to the RM with changed capabilities,
> there can arise situations where the overall cluster resource calculation for
> the cluster will be incorrect leading to inconsistencies in scheduling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
