[
https://issues.apache.org/jira/browse/MAPREDUCE-5489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13784898#comment-13784898
]
Bikas Saha commented on MAPREDUCE-5489:
---------------------------------------
By the time we clear, could new entries have been added to the sets (and which
will be lost upon clear()) ?
{code}
release.clear();
+
+ if (blacklistAdditions.size() > 0 || blacklistRemovals.size() > 0) {
+ LOG.info("Update the blacklist for " + applicationId +
+ ": blacklistAdditions=" + blacklistAdditions.size() +
+ " blacklistRemovals=" + blacklistRemovals.size());
+ }
+ blacklistAdditions.clear();
+ blacklistRemovals.clear();
{code}
I think the logic should be to not blacklist additional nodes if the threshold
is crossed. Instead the logic in the patch (and probably before it too) seems
to be to completely ignore all blacklisted nodes if the threshold is crossed.
That seems incorrect.
If we choose to not blacklist additional new nodes then we should stop sending
blacklist additions to the RM once threshold is reached. ie when the ignore
flag becomes true.
{code}
+ // notify RM to ignore all the blacklisted nodes
+ blacklistAdditions.clear();
+ blacklistRemovals.addAll(blacklistedNodes);
}
} else {
if (ignoreBlacklisting.compareAndSet(true, false)) {
LOG.info("Ignore blacklisting set to false. Known: " + clusterNmCount
+ ", Blacklisted: " + blacklistedNodeCount + ", " + val + "%");
+ // notify RM of all the blacklisted nodes
+ blacklistAdditions.add
{code}
Will the change be simpler if whenever the blacklisted nodes change then the
entire list is sent to the RM. Additionally RM is not updated if the ignore
flag is set.
> MR jobs hangs as it does not use the node-blacklisting feature in RM requests
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-5489
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5489
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Yesha Vora
> Assignee: Zhijie Shen
> Attachments: MAPREDUCE-5489.1.patch
>
>
> When RM restarted, if during restart one NM went bad (bad disk), NM got
> blacklisted by AM and RM keeps giving the containers on the same node even
> though AM doesn't want it there.
> Need to change AM to specifically blacklist node in the RM requests.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
