[
https://issues.apache.org/jira/browse/YARN-4311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14990565#comment-14990565
]
Jason Lowe commented on YARN-4311:
----------------------------------
Thanks for the patch, Kuhu! Test failures are related, please look into them.
In general the patch seems like a reasonable approach. There needs to be some
way for admins to remove nodes that are no longer relevant to the cluster, and
AFAIK there's no supported way to do this short of restarting the
resourcemanager. As nodes churn in and out of the cluster, they will simply
accumulate in the decommissioned or lost nodes buckets until the next
resourcemanager restart.
My main concern is the behavior when someone botches the include list (e.g.:
accidentally truncates the includes list file and refreshes). At that point
all of the cluster nodes will disappear from the resourcemanager with no
indication of what happened (except potentially the shutdown metric will
increment by the number of nodes lost). Today they will all go into the
decommissioned bucket, but with this patch they'll simply disappear. This
either needs to be "as designed" behavior, or we'd have to implement a separate
mechanism outside of the include/exclude lists to direct the RM to "forget" a
node. I believe HDFS recently was changed to behave this was as well wrt. the
include/exclude lists and forgetting nodes (see HDFS-8950), so I'm inclined to
be consistent with that and say it's "as designed."
Some comments on the patch itself:
isInvalidAndAbsent doesn't have the same handling of IP's as isValidNode does.
It might also be clearer if isInvalidAndAbsent were just named isUntracked or
isUntrackedNode indicating those are nodes we aren't tracking in any way.
isInvalidAndAbsent doesn't lock hostsReader like isValidNode does.
What about refreshNodesGracefully? That also refreshes the host
include/exclude lists and arguably needs similar logic. We need to discuss
what it means to gracefully refresh the list when the node completely
disappears from both the include and exclude list. Should it still gracefully
decommission, and how do we make sure that node is properly tracked? If
graceful, does it automatically disappear when the decommission completes since
it's not in either list?
Nit: While looping over the nodes, if the node is valid then there's no reason
to check if its not valid and absent. So it could be simplified to the
following:
{code}
for (NodeId nodeId: rmContext.getRMNodes().keySet()) {
if (!isValidNode(nodeId.getHost())) {
RMNodeEventType nodeEventType = isInvalidAndAbsent(nodeId.getHost()) ?
RMNodeEventType.SHUTDOWN : RMNodeEventType.DECOMMISSION;
this.rmContext.getDispatcher().getEventHandler().handle(
new RMNodeEvent(nodeId, nodeEventType));
}
}
{code}
> Removing nodes from include and exclude lists will not remove them from
> decommissioned nodes list
> -------------------------------------------------------------------------------------------------
>
> Key: YARN-4311
> URL: https://issues.apache.org/jira/browse/YARN-4311
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.6.1
> Reporter: Kuhu Shukla
> Assignee: Kuhu Shukla
> Attachments: YARN-4311-v1.patch
>
>
> In order to fully forget about a node, removing the node from include and
> exclude list is not sufficient. The RM lists it under Decomm-ed nodes. The
> tricky part that [~jlowe] pointed out was the case when include lists are not
> used, in that case we don't want the nodes to fall off if they are not active.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
