[
https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069602#comment-13069602
]
Robert Joseph Evans commented on MAPREDUCE-2723:
------------------------------------------------
I have been looking at MR-279 and I want to do something similar to MR-2324 it
is just not really set up to do it easily. The scheduling is split up between
the resource manager and the application master. And in fact the resource
manager/application master are completely ignoring disk utilization at this
point.
The plan is to add in disk utilization to the resources that the RM uses, and
then have AM request both disk and RAM space for reduces with disk space based
off of the size estimate currently used. Then inside the scheduler, which is
the right place in my opinion to decide if a request is being starved or not,
it would do just what MR-2324 does now but more generalized for all resource
constraints, not just disk. This means that all schedulers would have to be
modified to support this, but I can make the code generic so it should be
fairly simple to do. I just need to dig into the MR-279 code to decide exactly
how I want to insert this in. I should hopefully have a patch by mid next week.
> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
> Key: MAPREDUCE-2723
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to
> schedule a container. For feature parity with the original map reduce it
> should be extended to allow for disk space requests within containers along
> with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler
> to allow it to avoid starvation of containers that might never get the
> resources that they need.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
