[
https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506529#comment-13506529
]
Junping Du commented on YARN-18:
--------------------------------
Hi Arun, Thanks for your review on this patch. This patch works with YARN-19 to
make YARN can aware "node-group" level (the layer between host and rack which
is mostly for cloud/virtualization use case) data locality. In previous work on
Common and HDFS part for "node-group", Nicholas suggest me to divide changes
into "pluggable" and "plug-in" patches which make previous 2 layers (host/rack)
more pluggable and support new layer in plug-in which can be enabled with
specific configuration.
So for this patch, it tries to make locality related interface/class can be
extensible with node-group layer like: ScheduledRequests, RMNodeImpl,
SchedulerNode, etc. Also, it make some methods can be overridden in sub-class
like scheduler's assignContainer() method. In sub class, we can make it
allocate container for nodegroup-local tasks before allocating rack-local
tasks. Also, this patch add more unit tests. Please let me know if anything
here make you confusing. Thanks!
> Make locatlity in YARN's container assignment and task scheduling pluggable
> for other deployment topology
> ---------------------------------------------------------------------------------------------------------
>
> Key: YARN-18
> URL: https://issues.apache.org/jira/browse/YARN-18
> Project: Hadoop YARN
> Issue Type: New Feature
> Affects Versions: 2.0.3-alpha
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: features
> Attachments:
> HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch,
> MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch,
> MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch,
> MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch
>
>
> There are several classes in YARN’s container assignment and task scheduling
> algorithms that relate to data locality which were updated to give preference
> to running a container on other locality besides node-local and rack-local
> (like nodegroup-local). This propose to make these data structure/algorithms
> pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class
> ScheduledRequests was made a package level class to it would be easier to
> create a subclass, ScheduledRequestsWithNodeGroup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
