[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695522#comment-14695522 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2232 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2232/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695483#comment-14695483 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #283 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/283/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695458#comment-14695458 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #275 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/275/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/CHANGES.txt > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695391#comment-14695391 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2213 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2213/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695127#comment-14695127 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1016 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1016/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695120#comment-14695120 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #286 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/286/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/CHANGES.txt > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694341#comment-14694341 ] Hudson commented on YARN-4026: -- FAILURE: Integrated in Hadoop-trunk-Commit #8289 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8289/]) YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan (jianhe: rev e5003be907acef87c2770e3f2914953f62017b0e) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/CHANGES.txt > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: 2.8.0 > > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14693023#comment-14693023 ] Hadoop QA commented on YARN-4026: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 16m 21s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 44s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 0m 49s | The applied patch generated 3 new checkstyle issues (total was 128, now 128). | | {color:red}-1{color} | whitespace | 0m 5s | The patch has 30 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 20s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 29s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | yarn tests | 53m 24s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 91m 53s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12750004/YARN-4026.3.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 3ae716f | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8829/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8829/artifact/patchprocess/whitespace.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8829/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8829/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8829/console | This message was automatically generated. > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14692827#comment-14692827 ] Hadoop QA commented on YARN-4026: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 17m 16s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 8m 18s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 10m 17s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 0m 54s | The applied patch generated 3 new checkstyle issues (total was 128, now 128). | | {color:red}-1{color} | whitespace | 0m 5s | The patch has 30 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 27s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 36s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 34s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | yarn tests | 53m 47s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 94m 41s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.yarn.server.resourcemanager.TestRMAdminService | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12750004/YARN-4026.3.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 7c796fd | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8827/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8827/artifact/patchprocess/whitespace.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8827/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8827/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8827/console | This message was automatically generated. > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4026.1.patch, YARN-4026.2.patch, YARN-4026.3.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14687398#comment-14687398 ] Jian He commented on YARN-4026: --- - why "assignment.setFulfilledReservation(true);" is called in Reserved state ? {code} if (result.getAllocationState() == AllocationState.RESERVED) { // This is a reserved container LOG.info("Reserved container " + " application=" + application.getApplicationId() + " resource=" + allocatedResource + " queue=" + this.toString() + " cluster=" + clusterResource); assignment.getAssignmentInformation().addReservationDetails( updatedContainer.getId(), application.getCSLeafQueue().getQueuePath()); assignment.getAssignmentInformation().incrReservations(); Resources.addTo(assignment.getAssignmentInformation().getReserved(), allocatedResource); assignment.setFulfilledReservation(true); } else { {code} - I think here can always return ContainerAllocation.LOCALITY_SKIPPED as the semantics of this method is to try to allocate a container for certain locality. {code} return type == NodeType.OFF_SWITCH ? ContainerAllocation.APP_SKIPPED : ContainerAllocation.LOCALITY_SKIPPED; {code} The caller here can choose to return APP_SKIPPED if it sees the LOCALITY_SKIPPED {code} assigned = assignOffSwitchContainers(clusterResource, offSwitchResourceRequest, node, priority, reservedContainer, schedulingMode, currentResoureLimits); assigned.requestNodeType = requestType; return assigned; } {code} > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4026.1.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4026) FiCaSchedulerApp: ContainerAllocator should be able to choose how to order pending resource requests
[ https://issues.apache.org/jira/browse/YARN-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14661459#comment-14661459 ] Hadoop QA commented on YARN-4026: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 15m 11s | Findbugs (version ) appears to be broken on trunk. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 41s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 42s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 0m 27s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 0m 6s | The patch has 27 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 22s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 1m 31s | The patch appears to introduce 1 new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | yarn tests | 52m 40s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 89m 39s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-resourcemanager | | Failed unit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | | | hadoop.yarn.server.resourcemanager.TestRMAdminService | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12749203/YARN-4026.1.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / b6265d3 | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8786/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/8786/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8786/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8786/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8786/console | This message was automatically generated. > FiCaSchedulerApp: ContainerAllocator should be able to choose how to order > pending resource requests > > > Key: YARN-4026 > URL: https://issues.apache.org/jira/browse/YARN-4026 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-4026.1.patch > > > After YARN-3983, we have an extensible ContainerAllocator which can be used > by FiCaSchedulerApp to decide how to allocate resources. > While working on YARN-1651 (allocate resource to increase container), I found > one thing in existing logic not flexible enough: > - ContainerAllocator decides what to allocate for a given node and priority: > To support different kinds of resource allocation, for example, priority as > weight / skip priority or not, etc. It's better to let ContainerAllocator to > choose how to order pending resource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)