[
https://issues.apache.org/jira/browse/MAPREDUCE-5507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omkar Vinit Joshi reassigned MAPREDUCE-5507:
--------------------------------------------
Assignee: Omkar Vinit Joshi
> MapReduce reducer preemption gets hanged
> ----------------------------------------
>
> Key: MAPREDUCE-5507
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5507
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Omkar Vinit Joshi
> Assignee: Omkar Vinit Joshi
>
> Today if we are setting "yarn.app.mapreduce.am.job.reduce.rampup.limit" and
> "mapreduce.job.reduce.slowstart.completedmaps" then reducer are launched more
> aggressively. However the calculation to either Ramp up or Ramp down reducer
> is not down in most optimal way.
> * If MR AM at any point sees situation something like
> ** scheduledMaps : 30
> ** scheduledReducers : 10
> ** assignedMaps : 0
> ** assignedReducers : 11
> ** finishedMaps : 120
> ** headroom : 756 ( when your map /reduce task needs only 512mb)
> * then today it simply hangs because it thinks that there is sufficient room
> to launch one more mapper and therefore there is no need to ramp down.
> However, if this continues forever then this is not the correct way / optimal
> way.
> * Ideally for MR AM when it sees that assignedMaps drops have dropped to 0
> and there are running reducers around should wait for certain time ( upper
> limited by average map task completion time ... for heuristic sake)..but
> after that if still it doesn't get new container for map task then should
> preempt the reducer one by one with some interval and should ramp up slowly...
> ** Preemption of reducer can be done in little smarter way
> *** preempt reducer on a node manager for which there is any pending map
> request.
> *** otherwise preempt any other reducer. MR AM will contribute to getting new
> mapper by releasing such a reducer / container because it will reduce its
> cluster consumption and thereby may become candidate for an allocation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
