[
https://issues.apache.org/jira/browse/MAPREDUCE-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Junping Du updated MAPREDUCE-4310:
----------------------------------
Attachment: MAPREDUCE-4310.patch
Update patch as recently YARN scheduler change. "Patch Available" still pending
on MAPREDUCE-4309 to checked in first.
> 4-layer topology (with NodeGroup layer) implementation of Container
> Assignment and Task Scheduling (for YARN)
> -------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4310
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4310
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 1.0.0, 2.0.0-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments:
> HADOOP-8475-ContainerAssignmentTaskScheduling-withNodeGroup.patch,
> MAPREDUCE-4310.patch
>
>
> There are several classes in YARN’s container assignment and task scheduling
> algorithms that related to data locality which were updated to give
> preference to running a container on the same nodegroup. This section
> summarized the changes in the patch that provides a new implementation to
> support a four-layer hierarchy.
> When the ApplicationMaster makes a resource allocation request to the
> scheduler of ResourceManager, it will add the node group to the list of
> attributes in the ResourceRequest. The parameters of the resource request
> will change from <priority, (host, rack, *), memory, #containers> to
> <priority, (host, nodegroup, rack, *), memory, #containers>.
> After receiving the ResoureRequest the RM scheduler will assign containers
> for requests in the sequence of data-local, nodegroup-local, rack-local and
> off-switch.Then, ApplicationMaster schedules tasks on allocated containers in
> sequence of data- local, nodegroup-local, rack-local and off-switch.
> In terms of code changes made to YARN task scheduling, we updated the class
> ContainerRequestEvent so that applications can requests for containers can
> include anodegroup. In RM schedulers, FifoScheduler and CapacityScheduler
> were updated. For the FifoScheduler, the changes were in the method
> assignContainers. For the Capacity Scheduler the method
> assignContainersOnNode in the class of LeafQueue was updated. In both changes
> a new method, assignNodeGroupLocalContainers() was added in between the
> assignment data-local and rack-local.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
