[
https://issues.apache.org/jira/browse/MAPREDUCE-5507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Omkar Vinit Joshi updated MAPREDUCE-5507:
-----------------------------------------
Description:
Today if we are setting "yarn.app.mapreduce.am.job.reduce.rampup.limit" and
"mapreduce.job.reduce.slowstart.completedmaps" then reducers are launched more
aggressively. However the calculation to either Ramp up or Ramp down reducer is
not done in most optimal way.
* If MR AM at any point sees situation something like
** scheduledMaps : 30
** scheduledReducers : 10
** assignedMaps : 0
** assignedReducers : 11
** finishedMaps : 120
** headroom : 756 ( when your map /reduce task needs only 512mb)
* then today it simply hangs because it thinks that there is sufficient room to
launch one more mapper and therefore there is no need to ramp down. However, if
this continues forever then this is not the correct way / optimal way.
* Ideally for MR AM when it sees that assignedMaps drops have dropped to 0 and
there are running reducers around then it should wait for certain time ( upper
limited by average map task completion time ... for heuristic sake)..but after
that if still it doesn't get new container for map task then it should preempt
the reducer one by one with some interval and should ramp up slowly...
** Preemption of reducers can be done in little smarter way
*** preempt reducer on a node manager for which there is any pending map
request.
*** otherwise preempt any other reducer. MR AM will contribute to getting new
mapper by releasing such a reducer / container because it will reduce its
cluster consumption and thereby may become candidate for an allocation.
was:
Today if we are setting "yarn.app.mapreduce.am.job.reduce.rampup.limit" and
"mapreduce.job.reduce.slowstart.completedmaps" then reducer are launched more
aggressively. However the calculation to either Ramp up or Ramp down reducer is
not down in most optimal way.
* If MR AM at any point sees situation something like
** scheduledMaps : 30
** scheduledReducers : 10
** assignedMaps : 0
** assignedReducers : 11
** finishedMaps : 120
** headroom : 756 ( when your map /reduce task needs only 512mb)
* then today it simply hangs because it thinks that there is sufficient room to
launch one more mapper and therefore there is no need to ramp down. However, if
this continues forever then this is not the correct way / optimal way.
* Ideally for MR AM when it sees that assignedMaps drops have dropped to 0 and
there are running reducers around should wait for certain time ( upper limited
by average map task completion time ... for heuristic sake)..but after that if
still it doesn't get new container for map task then should preempt the reducer
one by one with some interval and should ramp up slowly...
** Preemption of reducer can be done in little smarter way
*** preempt reducer on a node manager for which there is any pending map
request.
*** otherwise preempt any other reducer. MR AM will contribute to getting new
mapper by releasing such a reducer / container because it will reduce its
cluster consumption and thereby may become candidate for an allocation.
> MapReduce reducer preemption gets hanged
> ----------------------------------------
>
> Key: MAPREDUCE-5507
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5507
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Omkar Vinit Joshi
> Assignee: Omkar Vinit Joshi
>
> Today if we are setting "yarn.app.mapreduce.am.job.reduce.rampup.limit" and
> "mapreduce.job.reduce.slowstart.completedmaps" then reducers are launched
> more aggressively. However the calculation to either Ramp up or Ramp down
> reducer is not done in most optimal way.
> * If MR AM at any point sees situation something like
> ** scheduledMaps : 30
> ** scheduledReducers : 10
> ** assignedMaps : 0
> ** assignedReducers : 11
> ** finishedMaps : 120
> ** headroom : 756 ( when your map /reduce task needs only 512mb)
> * then today it simply hangs because it thinks that there is sufficient room
> to launch one more mapper and therefore there is no need to ramp down.
> However, if this continues forever then this is not the correct way / optimal
> way.
> * Ideally for MR AM when it sees that assignedMaps drops have dropped to 0
> and there are running reducers around then it should wait for certain time (
> upper limited by average map task completion time ... for heuristic
> sake)..but after that if still it doesn't get new container for map task then
> it should preempt the reducer one by one with some interval and should ramp
> up slowly...
> ** Preemption of reducers can be done in little smarter way
> *** preempt reducer on a node manager for which there is any pending map
> request.
> *** otherwise preempt any other reducer. MR AM will contribute to getting new
> mapper by releasing such a reducer / container because it will reduce its
> cluster consumption and thereby may become candidate for an allocation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
