[
https://issues.apache.org/jira/browse/MAPREDUCE-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13406830#comment-13406830
]
Junping Du commented on MAPREDUCE-4310:
---------------------------------------
Hi Nicholas,
There are two ways to get nodegroup info in plugin:
1. Adding subclass of RackResolver, then add new interfaces something like
resolveNodeGroup() to resolve nodegroup info in plugin.
2. Keep RackResolver as current, but change comments to resolve() method so
that it resolve a networkLocality instead of rack. Then in plugin, it handle
getNodeGroup from networkLocality string itself by using
NetworkTopology.GetLastHalf(...).
Which way do you suggest here?
BTW, this patch depends on "pluggable" patch MAPREDUCE-4309.
Thanks,
Junping
> 4-layer topology (with NodeGroup layer) implementation of Container
> Assignment and Task Scheduling (for YARN)
> -------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments:
> HADOOP-8475-ContainerAssignmentTaskScheduling-withNodeGroup.patch
>
>
> There are several classes in YARN’s container assignment and task scheduling
> algorithms that related to data locality which were updated to give
> preference to running a container on the same nodegroup. This section
> summarized the changes in the patch that provides a new implementation to
> support a four-layer hierarchy.
> When the ApplicationMaster makes a resource allocation request to the
> scheduler of ResourceManager, it will add the node group to the list of
> attributes in the ResourceRequest. The parameters of the resource request
> will change from <priority, (host, rack, *), memory, #containers> to
> <priority, (host, nodegroup, rack, *), memory, #containers>.
> After receiving the ResoureRequest the RM scheduler will assign containers
> for requests in the sequence of data-local, nodegroup-local, rack-local and
> off-switch.Then, ApplicationMaster schedules tasks on allocated containers in
> sequence of data- local, nodegroup-local, rack-local and off-switch.
> In terms of code changes made to YARN task scheduling, we updated the class
> ContainerRequestEvent so that applications can requests for containers can
> include anodegroup. In RM schedulers, FifoScheduler and CapacityScheduler
> were updated. For the FifoScheduler, the changes were in the method
> assignContainers. For the Capacity Scheduler the method
> assignContainersOnNode in the class of LeafQueue was updated. In both changes
> a new method, assignNodeGroupLocalContainers() was added in between the
> assignment data-local and rack-local.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
