Varun Saxena commented on MAPREDUCE-5507:

I was initially thinking of having a configuration to ramp up reducers if maps 
are hanging for a while but as per discussion on MAPREDUCE-6689, this may lead 
to suboptimal job performance at it will be very hard to decide a right 
configuration value for this.

We haven't encountered any job hang issues in our deployments since 
MAPREDUCE-6513, MAPREDUCE-6514 has gone in our branch.
So I am fine with closing it. Maybe we can check with defect reporter too. cc 

> MapReduce reducer ramp down is suboptimal with potential job-hanging issues
> ---------------------------------------------------------------------------
>                 Key: MAPREDUCE-5507
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5507
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: applicationmaster
>            Reporter: Omkar Vinit Joshi
>            Assignee: Omkar Vinit Joshi
>            Priority: Critical
>         Attachments: MAPREDUCE-5507.20130922.1.patch
> Today if we are setting "yarn.app.mapreduce.am.job.reduce.rampup.limit" and 
> "mapreduce.job.reduce.slowstart.completedmaps" then reducers are launched 
> more aggressively. However the calculation to either Ramp up or Ramp down 
> reducer is not done in most optimal way. 
> * If MR AM at any point sees situation something like 
> ** scheduledMaps : 30
> ** scheduledReducers : 10
> ** assignedMaps : 0
> ** assignedReducers : 11
> ** finishedMaps : 120
> ** headroom : 756 ( when your map /reduce task needs only 512mb)
> * then today it simply hangs because it thinks that there is sufficient room 
> to launch one more mapper and therefore there is no need to ramp down. 
> However, if this continues forever then this is not the correct way / optimal 
> way.
> * Ideally for MR AM when it sees that assignedMaps drops have dropped to 0 
> and there are running reducers around then it should wait for certain time ( 
> upper limited by average map task completion time ... for heuristic 
> sake)..but after that if still it doesn't get new container for map task then 
> it should preempt the reducer one by one with some interval and should ramp 
> up slowly...
> ** Preemption of reducers can be done in little smarter way
> *** preempt reducer on a node manager for which there is any pending map 
> request.
> *** otherwise preempt any other reducer. MR AM will contribute to getting new 
> mapper by releasing such a reducer / container because it will reduce its 
> cluster consumption and thereby may become candidate for an allocation.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org

Reply via email to