[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13679544#comment-13679544 ] Junping Du commented on YARN-18: The findbugs warning seems to be unrelated. [~vicaya] and [~acmurthy], would you help to review it? Thanks! > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.3.patch, YARN-18-v6.4.patch, YARN-18-v6.patch, > YARN-18-v7.1.patch, YARN-18-v7.2.patch, YARN-18-v7.3.patch, YARN-18-v7.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678858#comment-13678858 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586900/YARN-18-v7.2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1172//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1172//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1172//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.3.patch, YARN-18-v6.4.patch, YARN-18-v6.patch, > YARN-18-v7.1.patch, YARN-18-v7.2.patch, YARN-18-v7.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678843#comment-13678843 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586893/YARN-18-v7.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/1171//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/1171//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1171//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.3.patch, YARN-18-v6.4.patch, YARN-18-v6.patch, > YARN-18-v7.1.patch, YARN-18-v7.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13678823#comment-13678823 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12586885/YARN-18-v7.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/1170//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.3.patch, YARN-18-v6.4.patch, YARN-18-v6.patch, > YARN-18-v7.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13666427#comment-13666427 ] Junping Du commented on YARN-18: [~acmurthy] As our discussion yesterday, let's schedule a call to discuss the specific implementation details of this JIRA and YARN-19. If you think in RM part, xSchedulerWithNodeGroup (x= Capacity, Fifo, Fair) don't make sense, we can get rid of this by abstracting locality related logic to a delegate class which can be pluggable to extend. Does that make sense to you? > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.3.patch, YARN-18-v6.4.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650739#comment-13650739 ] Hadoop QA commented on YARN-18: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582077/YARN-18-v6.4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/882//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/882//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.3.patch, YARN-18-v6.4.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650728#comment-13650728 ] Junping Du commented on YARN-18: In previous patch (v6.3), forget to include changes (to factory class name) in yarn-default.xml which cause some tests failure in creation of factory. Now fix it (and comments) in v6.4 patch. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.3.patch, YARN-18-v6.4.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650474#comment-13650474 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12582025/YARN-18-v6.3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing org.apache.hadoop.yarn.server.resourcemanager.TestResourceManager org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.TestRM org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue org.apache.hadoop.yarn.server.resourcemanager.security.TestClientTokens org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSSchedulerApp org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/878//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/878//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, Y
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650362#comment-13650362 ] Hadoop QA commented on YARN-18: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581846/YARN-18-v6.2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/871//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/871//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650346#comment-13650346 ] Junping Du commented on YARN-18: [~acmurthy]Yes. All implementation details can be found in this doc. It is dedicated for patch reviewers on YARN-18 and YARN-19. Hope it helps for review and welcome comment. [~vicaya]Agree. Will replace "Elements" with "Aware" in 6.3 patch. Thanks for reviewing carefully. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13650054#comment-13650054 ] Luke Lu commented on YARN-18: - Please find the summary/writeup in the attached "Pluggable topologies with NodeGroup for YARN.pdf" > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13649947#comment-13649947 ] Arun C Murthy commented on YARN-18: --- Junping, there are lot of changes to digest here. Can you please provide a summary/writeup of your patch? This will help me review it. Thanks! > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13649862#comment-13649862 ] Luke Lu commented on YARN-18: - The latest patch is getting close. I think the naming of Abstract(App)?TopologyElementsFactory is both long and imprecise, as a ContainerRequest is not a topology element but a topology aware object. How about we simply call it (App)?TopologyAwareFactory? This a straightforward patch to make (hierarchical) topology pluggable. I'll commit this patch this week if there is no further objections. This is needed for YARN-19. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.2.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13647249#comment-13647249 ] Hadoop QA commented on YARN-18: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581468/YARN-18-v6.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/857//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/857//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.1.patch, > YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646659#comment-13646659 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12581362/YARN-18-v6.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3 warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/854//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/854//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13646652#comment-13646652 ] Junping Du commented on YARN-18: Thanks Luke for explanation. Hey Arun ([~acmurthy]), I just attached a doc with implementation details for this patch and YARN-19. Hopefully that will be helpful for your review. Your thought to abstract notion of topology logic in scheduler make great sense to me. We already did this for each scheduler, but you mean that need to abstract common part for all schedulers which is a little complicated but still doable. Can we do this code refactoring work in a separated jira and I am glad to work on it? For a new NodeGroup Scheduler, I think it may not be necessary as it address different topologies rather than different algorithm to isolate/prioritise job. So even under topology with NodeGroup layer, different user still need different scheduler like: Fair, Capacity. etc. Thoughts? > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, Pluggable topologies with NodeGroup for YARN.pdf, > YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, YARN-18-v3.2.patch, > YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, YARN-18-v4.3.patch, > YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch, YARN-18-v6.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618525#comment-13618525 ] Luke Lu commented on YARN-18: - bq. I see a lot of factories and then no changes to actually support node-group.. YARN-18 is explicitly for making the topology pluggable based on previous review requests. So there is little node group specific changes, which are implemented in YARN-19. bq. we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology. This is a goal of the JIRA. You should be able to add another layer without having to modify scheduler code after this patch is committed. I'm not sure that we could make arbitrary topology pluggable without scheduler change. However, it should work for common class of topologies, e.g., hierarchical network topology. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618446#comment-13618446 ] Arun C Murthy commented on YARN-18: --- [~vicaya] / [~djp] - Can you please describe what is happening here? I see a lot of factories and then no changes to actually support node-group... What do I need to read-up to understand what is going on here? One thought - seems to me, (maybe longer term), we need to abstract the notion of topology in the Scheduler without having to make changes every time we need to add another layer in the topology... think of complex star-topologies etc. Thoughts? Another thing to consider is that, for now, it might be better to implement a new NodeGroup Scheduler rather than change all existing Schedulers? > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614675#comment-13614675 ] Junping Du commented on YARN-18: Thanks Arun and Luke for comments and review. [~acmurthy], YARN-18 and YARN-19 are still part of HADOOP-8468. In that umbrella JIRA, I had a proposal to describe how these two changes (P6, P7) works in detail level which may helps you to review this patch. If necessary, I can reflect the current changes (with addressing lots of comments above) and attach again in this JIRA. Thx! > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614383#comment-13614383 ] Luke Lu commented on YARN-18: - The original patch were already broken up into YARN-18 and YARN-19. YARN-18 makes topology aware classes pluggable. The patch itself is fairly straightforward: server side factory and app side factory to create topology aware classes. There is no behavior change by default. The latest looks big but fair amount of changes were trivial amendments due to the timeout requirement in unit tests. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13614355#comment-13614355 ] Arun C Murthy commented on YARN-18: --- Sorry, I'm just getting to this. This is a lot to digest. Can we consider breaking this down to a couple of smaller patches? Tx. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613604#comment-13613604 ] Hadoop QA commented on YARN-18: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575487/YARN-18-v5.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/601//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/601//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.1.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613561#comment-13613561 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12575479/YARN-18-v5.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/600//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/600//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613549#comment-13613549 ] Junping Du commented on YARN-18: Incorporate Luke's comments in latest (v5) patch. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch, YARN-18-v5.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13607066#comment-13607066 ] Junping Du commented on YARN-18: Thanks Luke for good suggestion. "yarn.app.topology.aware.factory.impl" seems better. Will address your comments in next patch. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13607021#comment-13607021 ] Luke Lu commented on YARN-18: - Juping, you're right that ScheduledRequests is an app side construct. My reaction to exposing the class was that it was a private class (hence implementation detail) of RMContainerAllocator. I was/am worried about exposing this detail (class name) to end-user config. We shouldn't have to change an end user config simply because we change the detail of implementation. How about we introduce a similar yarn.app.topology.aware.factory.impl which would be a more stable config property, so that the user-facing configuration stay the same if we later rename or add new topology aware classes for apps? I think we're getting close. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604946#comment-13604946 ] Hadoop QA commented on YARN-18: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12574124/YARN-18-v4.3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 tests included appear to have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/528//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/528//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.3.patch, YARN-18-v4.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13600849#comment-13600849 ] Hadoop QA commented on YARN-18: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573476/YARN-18-v4.2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 tests included appear to have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/511//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/511//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.2.patch, > YARN-18-v4.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13600824#comment-13600824 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573471/YARN-18-v4.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:green}+1 tests included appear to have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1 warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/510//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/510//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.1.patch, YARN-18-v4.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13600171#comment-13600171 ] Junping Du commented on YARN-18: Thanks Luke for your review and comments! I addressed almost all your points in V4 patch except for putting ScheduledRequests in topology related factory class for YARN as I think this is the object created within MRAppMaster but not YARN ResourceManager like other objects so in different package. Am I missing something? > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13600180#comment-13600180 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12573355/YARN-18-v4.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified test files. {color:red}-1 one of tests included doesn't have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 9 warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/505//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/505//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch, YARN-18-v4.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13599383#comment-13599383 ] Luke Lu commented on YARN-18: - Preliminary comments: # We should use net.topology.nodegroup.aware instead of net.topology.with.nodegroup for configuration to be consistent with branch-1 code. # We should probably use yarn.resourcemanager.topology.aware.factory.impl to specify factory class for topology aware objects. # The topologoy aware objects should include ScheduledRequests and AppSchedulingInfo, which are currently not managed by the factory. # Pluggable interface/class should be public with the InterfaceAudience.Private annotation. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593056#comment-13593056 ] Hadoop QA commented on YARN-18: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572023/YARN-18-v3.2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 tests included appear to have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/467//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/467//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.2.patch, YARN-18-v3.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13592986#comment-13592986 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572014/YARN-18-v3.1.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:red}-1 one of tests included doesn't have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/466//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/466//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.1.patch, > YARN-18-v3.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13592517#comment-13592517 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12571919/YARN-18-v3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:red}-1 one of tests included doesn't have a timeout.{color} {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/459//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/459//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch, YARN-18-v3.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506529#comment-13506529 ] Junping Du commented on YARN-18: Hi Arun, Thanks for your review on this patch. This patch works with YARN-19 to make YARN can aware "node-group" level (the layer between host and rack which is mostly for cloud/virtualization use case) data locality. In previous work on Common and HDFS part for "node-group", Nicholas suggest me to divide changes into "pluggable" and "plug-in" patches which make previous 2 layers (host/rack) more pluggable and support new layer in plug-in which can be enabled with specific configuration. So for this patch, it tries to make locality related interface/class can be extensible with node-group layer like: ScheduledRequests, RMNodeImpl, SchedulerNode, etc. Also, it make some methods can be overridden in sub-class like scheduler's assignContainer() method. In sub class, we can make it allocate container for nodegroup-local tasks before allocating rack-local tasks. Also, this patch add more unit tests. Please let me know if anything here make you confusing. Thanks! > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506499#comment-13506499 ] Arun C Murthy commented on YARN-18: --- Junping, sorry, I've missed this. Can you please provide a simple writeup of how you are trying to accomplish this? That will help me review the patch. Thanks. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506494#comment-13506494 ] Junping Du commented on YARN-18: Thanks Tom! > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505686#comment-13505686 ] Hadoop QA commented on YARN-18: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/187//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/187//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500276#comment-13500276 ] Hadoop QA commented on YARN-18: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12554120/YARN-18-v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.mapreduce.v2.app.TestAMInfos org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler org.apache.hadoop.mapreduce.v2.app.TestRecovery {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/154//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/154//console This message is automatically generated. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500260#comment-13500260 ] Thomas Graves commented on YARN-18: --- I kicked the pre-commit build. > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-18) Make locatlity in YARN's container assignment and task scheduling pluggable for other deployment topology
[ https://issues.apache.org/jira/browse/YARN-18?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500044#comment-13500044 ] Junping Du commented on YARN-18: Looks like pre-commit test (https://builds.apache.org/job/PreCommit-YARN-Build/) is not automatically triggered. Is there anything I should do besides set this issue to Patch Available? > Make locatlity in YARN's container assignment and task scheduling pluggable > for other deployment topology > - > > Key: YARN-18 > URL: https://issues.apache.org/jira/browse/YARN-18 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 2.0.3-alpha >Reporter: Junping Du >Assignee: Junping Du > Labels: features > Attachments: > HADOOP-8474-ContainerAssignmentTaskScheduling-pluggable.patch, > MAPREDUCE-4309.patch, MAPREDUCE-4309-v2.patch, MAPREDUCE-4309-v3.patch, > MAPREDUCE-4309-v4.patch, MAPREDUCE-4309-v5.patch, MAPREDUCE-4309-v6.patch, > MAPREDUCE-4309-v7.patch, YARN-18.patch, YARN-18-v2.patch > > > There are several classes in YARN’s container assignment and task scheduling > algorithms that relate to data locality which were updated to give preference > to running a container on other locality besides node-local and rack-local > (like nodegroup-local). This propose to make these data structure/algorithms > pluggable, like: SchedulerNode, RMNodeImpl, etc. The inner class > ScheduledRequests was made a package level class to it would be easier to > create a subclass, ScheduledRequestsWithNodeGroup. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira