[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505129#comment-14505129 ] Hudson commented on YARN-3463: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2120 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2120/]) YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) (wangda: rev 44872b76fcc0ddfbc7b0a4e54eef50fe8708e0f5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/OrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicy.java * hadoop-yarn-project/CHANGES.txt > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch, YARN-3463.70.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14505077#comment-14505077 ] Hudson commented on YARN-3463: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #171 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/171/]) YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) (wangda: rev 44872b76fcc0ddfbc7b0a4e54eef50fe8708e0f5) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/OrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch, YARN-3463.70.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504813#comment-14504813 ] Hudson commented on YARN-3463: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #161 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/161/]) YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) (wangda: rev 44872b76fcc0ddfbc7b0a4e54eef50fe8708e0f5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/OrderingPolicy.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch, YARN-3463.70.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504806#comment-14504806 ] Hudson commented on YARN-3463: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2102 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2102/]) YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) (wangda: rev 44872b76fcc0ddfbc7b0a4e54eef50fe8708e0f5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/OrderingPolicy.java > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch, YARN-3463.70.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504753#comment-14504753 ] Hudson commented on YARN-3463: -- FAILURE: Integrated in Hadoop-Yarn-trunk #904 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/904/]) YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) (wangda: rev 44872b76fcc0ddfbc7b0a4e54eef50fe8708e0f5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/OrderingPolicy.java > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch, YARN-3463.70.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504746#comment-14504746 ] Hudson commented on YARN-3463: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #170 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/170/]) YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) (wangda: rev 44872b76fcc0ddfbc7b0a4e54eef50fe8708e0f5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/OrderingPolicy.java > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch, YARN-3463.70.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14504022#comment-14504022 ] Hudson commented on YARN-3463: -- SUCCESS: Integrated in Hadoop-trunk-Commit #7620 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7620/]) YARN-3463. Integrate OrderingPolicy Framework with CapacityScheduler. (Craig Welch via wangda) (wangda: rev 44872b76fcc0ddfbc7b0a4e54eef50fe8708e0f5) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/FifoOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/ProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TestProportionalCapacityPreemptionPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/AbstractComparatorOrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/OrderingPolicy.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Fix For: 2.8.0 > > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch, YARN-3463.70.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503552#comment-14503552 ] Wangda Tan commented on YARN-3463: -- Great! I will commit this today if no opposite opinions. > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch, YARN-3463.70.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503496#comment-14503496
]
Hadoop QA commented on YARN-3463:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726616/YARN-3463.70.patch
against trunk revision f967fd2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7399//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7399//console
This message is automatically generated.
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch,
> YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch, YARN-3463.70.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503305#comment-14503305 ] Jian He commented on YARN-3463: --- patch looks good to me. bq. as getStatusMessage suggests to me a transient nature which may change (metrics, etc), whereas this is information about policy type and configuration which is effectively static. sounds reasonable. let's just use getInfo then. > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503302#comment-14503302 ] Wangda Tan commented on YARN-3463: -- bq. I have to say I prefer getInfo to getStatusMessage myself, as getStatusMessage suggests to me a transient nature which may change (metrics, etc), whereas this is information about policy type and configuration which is effectively static, it is just generic info, and isn't particularly transient I don't have strong preference for them: getStatusMessage is clear but looks complex, getInfo is simple but "info" is too generic. Please feel free to choose any of them. [~jianhe], please let me know if your idea about latest patch. > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502260#comment-14502260
]
Hadoop QA commented on YARN-3463:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726489/YARN-3463.69.patch
against trunk revision 8511d80.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7393//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7393//console
This message is automatically generated.
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch,
> YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502207#comment-14502207
]
Craig Welch commented on YARN-3463:
---
bq. ... I think we can just initialize this.comparator and
this.schedulableEntities inside FifoOrderingPolicy constructor and remove the
setComparator method
Done
bq. this should be inside the {removed} ...
Done
bq. getStatusMessage -> getInfo ?
Originally, it was getInfo -
https://issues.apache.org/jira/browse/YARN-3318?focusedCommentId=14494396&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14494396
I have to say I prefer getInfo to getStatusMessage myself, as getStatusMessage
suggests to me a transient nature which may change (metrics, etc), whereas this
is information about policy type and configuration which is effectively static,
it is just generic info, and isn't particularly transient
If you feel strongly that it should be getInfo, let me know and I'll change it
back.
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch,
> YARN-3463.67.patch, YARN-3463.68.patch, YARN-3463.69.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500937#comment-14500937
]
Jian He commented on YARN-3463:
---
- this.schedulableEntities != null can never happen? since each time it’s
re-creating a new Ordering class. I think we can just initialize
{{this.comparator}} and {{this.schedulableEntities}} inside FifoOrderingPolicy
constructor and remove the setComparator method
{code}
if (this.schedulableEntities != null) {
schedulableEntities.addAll(this.schedulableEntities);
}
{code}
- this should be inside the {removed} check ? otherwise any unknown container
completed will cause reordering
{code}
orderingPolicy.containerReleased(application, rmContainer);
{code}
- getStatusMessage -> getInfo ?
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch,
> YARN-3463.67.patch, YARN-3463.68.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500184#comment-14500184 ] Wangda Tan commented on YARN-3463: -- [~cwelch], Thanks for update, patch LGTM, +1. > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499285#comment-14499285
]
Hadoop QA commented on YARN-3463:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726059/YARN-3463.68.patch
against trunk revision bb6dde6.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7372//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7372//console
This message is automatically generated.
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch,
> YARN-3463.67.patch, YARN-3463.68.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499205#comment-14499205 ] Craig Welch commented on YARN-3463: --- btw, the tests pass on my box with the change, failures not related to the patch > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499204#comment-14499204 ] Craig Welch commented on YARN-3463: --- bq. how about change it to Map to explicitly pass option_key=value pairs to configure OrderingPolicy signature changed, will add configuration to pass in sizeBasedWeight as part of the FairOrderingPolicy patch, as that's where it would belong... bq. you can suppress them to avoid javac warning suppressed > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, > YARN-3463.67.patch, YARN-3463.68.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498661#comment-14498661
]
Hadoop QA commented on YARN-3463:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725950/YARN-3463.67.patch
against trunk revision 80a2a12.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1154 javac
compiler warnings (more than the trunk's current 1153 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7363//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7363//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7363//console
This message is automatically generated.
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, YARN-3463.67.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498603#comment-14498603 ] Wangda Tan commented on YARN-3463: -- bq. No, applications still start as they did before, no reason to change it May not, for example, when priority comparator enabled, how to activate apps could be determined by priority of the app instead of submission order. But it's a future requirement, we can postpone the changes to when they really needed. bq. No, this comes from the interface definition and it's needed to enable another scheduler, say FS, be able to use the same code with their derived application types of choice I don't think FS will use the CapacitySchedulerConfiguration, it's not a big deal, let's keep it as-is. bq. So this configuration is not for defining the policy, that is "ORDERING_POLICY" (which is where you would have "fair", "fifo", etc), this is for configuration elements which may modify the behavior of the policy, such as sizeBasedWeight, & it's needed for that purpose. Hmm.. That could be a problem after take a look at YARN-3319. IMO, Configure OrderingPolicy should follow the style of how we configure other modules, which is using capacity-scheduler.xml in CS and using fair-scheduler.xml in FS. For individual options such as sizeBasedWeight, it's better to make it to be a separated option instead of putting all of them together. And the configure(String config) for OrderingPolicy could be problematic, I missed this point while reviewing YARN-3318, how about change it to Map to explicitly pass option_key=value pairs to configure OrderingPolicy? bq. I'm actually not getting any there, I think because this is used for mocking I can see while using eclipse, you can suppress them to avoid javac warning. > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch, YARN-3463.67.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498492#comment-14498492 ] Craig Welch commented on YARN-3463: --- bq. should we make pendingApplications to use customized comparator No, applications still start as they did before, no reason to change it bq. can we make it not use generic type now for simpler No, this comes from the interface definition and it's needed to enable another scheduler, say FS, be able to use the same code with their derived application types of choice bq. I think we may carefully add ORDERING_POLICY_CONFIG, since this will be a public config. I understand the reason to add the policy_config is to support policy=fair, config=fair+fifo usecase So this configuration is not for defining the policy, that is "ORDERING_POLICY" (which is where you would have "fair", "fifo", etc), this is for configuration elements which may modify the behavior of the policy, such as sizeBasedWeight, & it's needed for that purpose. bq. Why change disposableLeafQueue.getApplications().size() I think this is left over from earlier changes & is no longer needed, will remove. bq. Suppress generic warning? Javac warning? I'm actually not getting any there, I think because this is used for mocking > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3463.61.patch, > YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498326#comment-14498326
]
Wangda Tan commented on YARN-3463:
--
Some comments:
LeafQueue:
- activeApplications now becomes orderingPolicy, should we make
pendingApplications to use customized comparator?
- getOrderingPolicy is located in CapacityScheduler, so SchedulerEntity can be
FicaSchedulerAppp, can we make it not use generic type now for simpler?
CapacitySchedulerConfiguration:
- I think we may carefully add ORDERING_POLICY_CONFIG, since this will be a
public config. I understand the reason to add the policy_config is to support
policy=fair, config=fair+fifo usecase, but I'm not sure if we really need that.
For now, to satisfy fair scheduling only, I suggest to support policy only, and
the config="fair+fifo" will be set directly, we can discuss and add
configurable items in the future. And since the ordering_policy potentially
could change (the ordering_policy is a new feature and isn't under extensive
review), it's better to add a note to indicate it's a experimental option in
the upcoming doc.
CapacityScheduler:
- Why change this?
{code}
- + " is not empty " + disposableLeafQueue.getApplications().size()
+ + " is not empty " + disposableLeafQueue.getNumActiveApplications()
{code}
TestProportional...Policy:
- Suppress generic warning?
{code}
OrderingPolicy so = mock(OrderingPolicy.class);
when(so.getPreemptionIterator()).thenAnswer(new Answer() {
{code}
Javac warning?
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497451#comment-14497451
]
Hadoop QA commented on YARN-3463:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725744/YARN-3463.66.patch
against trunk revision 1b89a3e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1148 javac
compiler warnings (more than the trunk's current 1147 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7353//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7353//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7353//console
This message is automatically generated.
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch, YARN-3463.66.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497354#comment-14497354
]
Hadoop QA commented on YARN-3463:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725714/YARN-3463.65.patch
against trunk revision 1b89a3e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1149 javac
compiler warnings (more than the trunk's current 1147 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7351//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7351//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7351//console
This message is automatically generated.
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14497311#comment-14497311
]
Hadoop QA commented on YARN-3463:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725702/YARN-3463.64.patch
against trunk revision 1b89a3e.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1149 javac
compiler warnings (more than the trunk's current 1147 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7350//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7350//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7350//console
This message is automatically generated.
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3463.50.patch, YARN-3463.61.patch,
> YARN-3463.64.patch, YARN-3463.65.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494924#comment-14494924 ] Craig Welch commented on YARN-3463: --- I did, but it has the right stuff in it :-) I'll fix in a minute. I looked and it appears that the rest api will show the ordering policy, it appears that the UI and the rest api are joined this way - I'll see if it can be suppressed in a reasonable way. If it can't, though, I think we need to leave it in - it's simply not acceptable to leave users blind/unable to determine what policy is in effect. > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.61.patch, YARN-3463.50.patch, > YARN-3643.58.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[
https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494920#comment-14494920
]
Wangda Tan commented on YARN-3463:
--
bq. Updated, matches/should apply and work with YARN-3318 .61.patch
Did you upload the wrong filename? {{YARN-3319.61.patch}}
> Integrate OrderingPolicy Framework with CapacityScheduler
> -
>
> Key: YARN-3463
> URL: https://issues.apache.org/jira/browse/YARN-3463
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Reporter: Craig Welch
>Assignee: Craig Welch
> Attachments: YARN-3319.61.patch, YARN-3463.50.patch,
> YARN-3643.58.patch
>
>
> Integrate the OrderingPolicy Framework with the CapacityScheduler
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494908#comment-14494908 ] Wangda Tan commented on YARN-3463: -- bq. But I don't believe the concern is correct / there is any behavioral change for FIFO. Since it's just a one line change, and it related to changes in CS, let's keep it here. bq. There are no REST API or CLI changes in the patch anymore IIUC, Changes of CapacitySchedulerLeafQueueInfo will expose orderingPolicy to REST API, is there any approach to exclude it from REST API? Changes of web UI are manageable, let's keep it here too. > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3319.61.patch, YARN-3463.50.patch, > YARN-3643.58.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494892#comment-14494892 ] Craig Welch commented on YARN-3463: --- bq. 1) Preemption policy changes seems not correct to me... So, I believe the behavior for FIFO should be exactly as it was before - and all of the preemption tests were passing with the combined patch, so I think this is the case. Fairness preemption would be handled on [YARN-3319]. I don't mind moving the final integration for preemption into another jira, but I don't believe the concern is correct / there is any behavioral change for FIFO. bq. 2) WebUI, REST API and CLI changes are public APIs and related to core changes in CS... There are no REST API or CLI changes in the patch anymore, we agreed on [YARN-3318] https://issues.apache.org/jira/browse/YARN-3318?focusedCommentId=14393347&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14393347 that the WebUI changes should stay with the initial integration - it's very important/needed to be able to confirm that configuration was accomplished properly, without it there is no way to tell what policy is active. bq. So I suggest only leave core changes for CS including configuration So, I think given the WebUI bit above, this is already the case, with the possible exception of preemption which, again, I think has not seen any behavior change for FIFO, which is all we have at this time. > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3643.58.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3463) Integrate OrderingPolicy Framework with CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-3463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494858#comment-14494858 ] Wangda Tan commented on YARN-3463: -- [~cwelch], did a quick look at this, some quick comments about patch scope: 1) Preemption policy changes seems not correct to me, for example, 4 apps in a queue, usages for app1-app4 are: 4/6/3/7, and need claim 8 resources back. With this change, it will get 6 from app4, 2 from app2. After preemption is: 4/4/3/1. And actually it should get 1 from app1, 3 from app2, 4 from app 4, after preemption is: 3/3/3/3. Since the preemption policy changes potentially big, let's move it to a separated patch. 2) WebUI, REST API and CLI changes are public APIs and related to core changes in CS. I suggest to move it to a separated patch. So I suggest only leave core changes for CS including configuration and scheduler in the patch for easier review as well as less reworks needed. Agree? > Integrate OrderingPolicy Framework with CapacityScheduler > - > > Key: YARN-3463 > URL: https://issues.apache.org/jira/browse/YARN-3463 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Reporter: Craig Welch >Assignee: Craig Welch > Attachments: YARN-3463.50.patch, YARN-3643.58.patch > > > Integrate the OrderingPolicy Framework with the CapacityScheduler -- This message was sent by Atlassian JIRA (v6.3.4#6332)
