[
https://issues.apache.org/jira/browse/MAPREDUCE-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15274023#comment-15274023
]
Varun Saxena commented on MAPREDUCE-6689:
-----------------------------------------
Just to give a background, MAPREDUCE-6514 came up while analyzing issue in
MAPREDUCE-6513.
During our analysis of the issue, there were 2 areas which we found susceptible
to problems.
One was decreasing and updating container requests when we clear pending reduce
requests, something which has been handled in MAPREDUCE-6514.
And other one was whether we should ramp up reducers at all if there are
hanging maps since a certain period of time.
We actually wanted to take feedback of community members who have worked
extensively in MapReduce on these potential issues.
Refer to comments below(made on MAPREDUCE-6513) :
[comment1 |
https://issues.apache.org/jira/browse/MAPREDUCE-6513?focusedCommentId=14960484&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14960484],
[comment2 |
https://issues.apache.org/jira/browse/MAPREDUCE-6513?focusedCommentId=14961762&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14961762]
Basically this point got lost in between as we went ahead with rescheduling map
requests with priority 5 in MAPREDUCE-6513.
But I just thought to put it out there again.
In RMContainerAllocator#scheduleReduces we ramp up reduces. The calculations in
this method are such that this is done too aggressively with default
configurations.
Configuration of yarn.app.mapreduce.am.job.reduce.rampup.limit has the default
value of 0.5. And if headroom is limited(like in MAPREDUCE-6513 scenario) i.e.
barely enough to launch one mapper/reducer, because of this config value, it
thinks that there is sufficient room to launch one more mapper and therefore
there is no need to ramp down and reducers are ramped up. However, if this
continues forever then this does not seem correct.
Should we really be ramping up if we have hanging map requests irrespective of
configuration value of reduce rampup limit ?
We can probably use configuration introduced in MAPREDUCE-6302 to determine if
maps are hanging(i.e. maps are stuck in scheduled state) and do not ramp up
reduces if maps are hanging for a while. The config value of how long to wait
however would depend on the kind of job being run though.
We also have MAPREDUCE-6541 as well which adjusts headroom received from RM to
find out if we have enough resources for a map task to run.
> MapReduce job can infinitely increase number of reducer resource requests
> -------------------------------------------------------------------------
>
> Key: MAPREDUCE-6689
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6689
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Priority: Blocker
> Attachments: MAPREDUCE-6689.1.patch
>
>
> We have seen this issue from one of our clusters: when running terasort
> map-reduce job, some mappers failed after reducer started, and then MR AM
> tries to preempt reducers to schedule these failed mappers.
> After that, MR AM enters an infinite loop, for every
> RMContainerAllocator#heartbeat run, it:
> - In {{preemptReducesIfNeeded}}, it cancels all scheduled reducer requests.
> (total scheduled reducers = 1024)
> - Then, in {{scheduleReduces}}, it ramps up all reducers (total = 1024).
> As a result, we can see total #requested-containers increased 1024 for every
> MRAM-RM heartbeat (1 sec per heartbeat). The AM is hanging for 18+ hours, so
> we get 18 * 3600 * 1024 ~ 66M+ requested containers in RM side.
> And this bug also triggered YARN-4844, which makes RM stop scheduling
> anything.
> Thanks to [~sidharta-s] for helping with analysis.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
