[
https://issues.apache.org/jira/browse/MAPREDUCE-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14965197#comment-14965197
]
chong chen commented on MAPREDUCE-6513:
---------------------------------------
How to re-schedule failure/killed tasks vs how to count task exit reason are
two different things.
For your case, node is not healthy, it is a typical abnormal case for task
failure. And this is a small probability event in a healthy cluster
environment. And for a small set of map task rerun, we should be smarter to let
it complete quickly rather than bother going through this heavy reducer ramp up
and down flow. Because this not only slowing down overall job scheduling
throughput, but also adding unnecessary loads on YARN core scheduler. Workload
requests (over 600 reducers) have been submitted to the system, for a small set
of map tasks, AM ramps down all reducer and put those small mapper into the
front of queue in order to get scheduling, then later, has to gradually
re-submit them is not an efficient way to handle things. It generates
unnecessary load on core scheduler. Since YARN is the central brain of big data
system, it manages large scale multi-tenant cluster. The design philosophy
should always keep that in mind and try to reduce unnecessary loads on core.
I think what you discover later is a problem, we need to correct them. But for
this particular case, I still prefer treating them abnormal failure and bump up
task priority.
> MR job got hanged forever when one NM unstable for some time
> ------------------------------------------------------------
>
> Key: MAPREDUCE-6513
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6513
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster, resourcemanager
> Affects Versions: 2.7.0
> Reporter: Bob
> Assignee: Varun Saxena
> Priority: Critical
>
> when job is in-progress which is having more tasks,one node became unstable
> due to some OS issue.After the node became unstable, the map on this node
> status changed to KILLED state.
> Currently maps which were running on unstable node are rescheduled, and all
> are in scheduled state and wait for RM assign container.Seen ask requests for
> map till Node is good (all those failed), there are no ask request after
> this. But AM keeps on preempting the reducers (it's recycling).
> Finally reducers are waiting for complete mappers and mappers did n't get
> container..
> My Question Is:
> ============
> why map requests did not sent AM ,once after node recovery.?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
