[
https://issues.apache.org/jira/browse/MAPREDUCE-3210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13178780#comment-13178780
]
Robert Joseph Evans commented on MAPREDUCE-3210:
------------------------------------------------
Your concern #1 is already happening. With MRV2 right now all the requests,
global, rack local, and node specific are made at once. This results in the
possibility that on an underused cluster all of them might be fulfilled and
returned to the AM. If the AM can make use of one of the containers it will,
otherwise it will release it.
Perhaps the better way to do this is to have the AM be responsible for making
the requests at different times. So for example on the first heartbeat after a
container is needed only the node local request is made. If it does not get it
after a specific timeout (1 heartbeat by default) then a rack local request is
added, and finally the global request is added after another timeout.
It would be nice to have it be more generic so that some how the requests are
tied together, but that would require an API change and may not be simple to do
in the short term.
> Support delay scheduling for node locality in MR2's capacity scheduler
> ----------------------------------------------------------------------
>
> Key: MAPREDUCE-3210
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3210
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Todd Lipcon
>
> The capacity scheduler in MR2 doesn't support delay scheduling for achieving
> node-level locality. So, jobs exhibit poor data locality even if they have
> good rack locality. Especially on clusters where disk throughput is much
> better than network capacity, this hurts overall job performance. We should
> optionally support node-level delay scheduling heuristics similar to what the
> fair scheduler implements in MR1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
