[
https://issues.apache.org/jira/browse/MAPREDUCE-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784910#comment-13784910
]
Zhijie Shen commented on MAPREDUCE-5489:
----------------------------------------
bq. By the time we clear, could new entries have been added to the sets (and
which will be lost upon clear()) ?
Seems to be impossible, because blacklistAdditions an blacklistedRemovals are
only referred in two synchronized methods: heartbeat() and handleEvent()
bq. I think the logic should be to not blacklist additional nodes if the
threshold is crossed.
Yes, if the threshold is reached, the blacklist will be be checked for the
following assigned containers. That's why we need to notify the RM to remove
the blacklist. Otherwise, the RM will continue to avoid assigning the
containers on the blacklisted nodes to the AM; therefore, the nodes will
actually still blacklisted.
bq. Will the change be simpler if whenever the blacklisted nodes change then
the entire list is sent to the RM.
We need to wait until next heatbeat to notify RM, such that we need to remember
all the changes that's why blacklistAdditions is there to track the increased
blacklisted nodes.
> MR jobs hangs as it does not use the node-blacklisting feature in RM requests
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-5489
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5489
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Yesha Vora
> Assignee: Zhijie Shen
> Attachments: MAPREDUCE-5489.1.patch
>
>
> When RM restarted, if during restart one NM went bad (bad disk), NM got
> blacklisted by AM and RM keeps giving the containers on the same node even
> though AM doesn't want it there.
> Need to change AM to specifically blacklist node in the RM requests.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
