[ https://issues.apache.org/jira/browse/MAPREDUCE-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Junping Du moved HADOOP-8475 to MAPREDUCE-4310: ----------------------------------------------- Affects Version/s: (was: 2.0.0-alpha) (was: 1.0.0) 1.0.0 2.0.0-alpha Key: MAPREDUCE-4310 (was: HADOOP-8475) Project: Hadoop Map/Reduce (was: Hadoop Common) > 4-layer topology (with NodeGroup layer) implementation of Container > Assignment and Task Scheduling (for YARN) > ------------------------------------------------------------------------------------------------------------- > > Key: MAPREDUCE-4310 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4310 > Project: Hadoop Map/Reduce > Issue Type: Bug > Affects Versions: 2.0.0-alpha, 1.0.0 > Reporter: Junping Du > Assignee: Junping Du > Attachments: > HADOOP-8475-ContainerAssignmentTaskScheduling-withNodeGroup.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that related to data locality which were updated to give > preference to running a container on the same nodegroup. This section > summarized the changes in the patch that provides a new implementation to > support a four-layer hierarchy. > When the ApplicationMaster makes a resource allocation request to the > scheduler of ResourceManager, it will add the node group to the list of > attributes in the ResourceRequest. The parameters of the resource request > will change from <priority, (host, rack, *), memory, #containers> to > <priority, (host, nodegroup, rack, *), memory, #containers>. > After receiving the ResoureRequest the RM scheduler will assign containers > for requests in the sequence of data-local, nodegroup-local, rack-local and > off-switch.Then, ApplicationMaster schedules tasks on allocated containers in > sequence of data- local, nodegroup-local, rack-local and off-switch. > In terms of code changes made to YARN task scheduling, we updated the class > ContainerRequestEvent so that applications can requests for containers can > include anodegroup. In RM schedulers, FifoScheduler and CapacityScheduler > were updated. For the FifoScheduler, the changes were in the method > assignContainers. For the Capacity Scheduler the method > assignContainersOnNode in the class of LeafQueue was updated. In both changes > a new method, assignNodeGroupLocalContainers() was added in between the > assignment data-local and rack-local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira