[
https://issues.apache.org/jira/browse/MAPREDUCE-5507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15431740#comment-15431740
]
Karthik Kambatla commented on MAPREDUCE-5507:
---------------------------------------------
[~varun_saxena] - this appears to be very similar to issues fixed in
MAPREDUCE-6513 and MAPREDUCE-6514. Can this be closed as a duplicate?
> MapReduce reducer ramp down is suboptimal with potential job-hanging issues
> ---------------------------------------------------------------------------
>
> Key: MAPREDUCE-5507
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5507
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
> Reporter: Omkar Vinit Joshi
> Assignee: Omkar Vinit Joshi
> Priority: Critical
> Attachments: MAPREDUCE-5507.20130922.1.patch
>
>
> Today if we are setting "yarn.app.mapreduce.am.job.reduce.rampup.limit" and
> "mapreduce.job.reduce.slowstart.completedmaps" then reducers are launched
> more aggressively. However the calculation to either Ramp up or Ramp down
> reducer is not done in most optimal way.
> * If MR AM at any point sees situation something like
> ** scheduledMaps : 30
> ** scheduledReducers : 10
> ** assignedMaps : 0
> ** assignedReducers : 11
> ** finishedMaps : 120
> ** headroom : 756 ( when your map /reduce task needs only 512mb)
> * then today it simply hangs because it thinks that there is sufficient room
> to launch one more mapper and therefore there is no need to ramp down.
> However, if this continues forever then this is not the correct way / optimal
> way.
> * Ideally for MR AM when it sees that assignedMaps drops have dropped to 0
> and there are running reducers around then it should wait for certain time (
> upper limited by average map task completion time ... for heuristic
> sake)..but after that if still it doesn't get new container for map task then
> it should preempt the reducer one by one with some interval and should ramp
> up slowly...
> ** Preemption of reducers can be done in little smarter way
> *** preempt reducer on a node manager for which there is any pending map
> request.
> *** otherwise preempt any other reducer. MR AM will contribute to getting new
> mapper by releasing such a reducer / container because it will reduce its
> cluster consumption and thereby may become candidate for an allocation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
