[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660265#comment-14660265 ] Hudson commented on YARN-3983: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2206 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2206/]) YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic. Contributed by Wangda Tan (jianhe: rev ba2313d6145a1234777938a747187373f4cd58d9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AllocationState.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.8.0 Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch, YARN-3983.4.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14659994#comment-14659994 ] Hudson commented on YARN-3983: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #1009 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1009/]) YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic. Contributed by Wangda Tan (jianhe: rev ba2313d6145a1234777938a747187373f4cd58d9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AllocationState.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.8.0 Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch, YARN-3983.4.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660300#comment-14660300 ] Hudson commented on YARN-3983: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #268 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/268/]) YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic. Contributed by Wangda Tan (jianhe: rev ba2313d6145a1234777938a747187373f4cd58d9) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AllocationState.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.8.0 Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch, YARN-3983.4.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660511#comment-14660511 ] Hudson commented on YARN-3983: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #276 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/276/]) YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic. Contributed by Wangda Tan (jianhe: rev ba2313d6145a1234777938a747187373f4cd58d9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AllocationState.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * hadoop-yarn-project/CHANGES.txt Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.8.0 Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch, YARN-3983.4.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660528#comment-14660528 ] Hudson commented on YARN-3983: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2225 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2225/]) YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic. Contributed by Wangda Tan (jianhe: rev ba2313d6145a1234777938a747187373f4cd58d9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AllocationState.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.8.0 Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch, YARN-3983.4.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660027#comment-14660027 ] Hudson commented on YARN-3983: -- SUCCESS: Integrated in Hadoop-Yarn-trunk-Java8 #279 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/279/]) YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic. Contributed by Wangda Tan (jianhe: rev ba2313d6145a1234777938a747187373f4cd58d9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AllocationState.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.8.0 Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch, YARN-3983.4.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658899#comment-14658899 ] Jian He commented on YARN-3983: --- failed test pass locally, not related committing Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.8.0 Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch, YARN-3983.4.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14658955#comment-14658955 ] Hudson commented on YARN-3983: -- FAILURE: Integrated in Hadoop-trunk-Commit #8266 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8266/]) YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic. Contributed by Wangda Tan (jianhe: rev ba2313d6145a1234777938a747187373f4cd58d9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AllocationState.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Fix For: 2.8.0 Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch, YARN-3983.4.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652931#comment-14652931 ] Hadoop QA commented on YARN-3983: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 17m 40s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 9m 31s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 10m 53s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 1m 1s | The applied patch generated 29 new checkstyle issues (total was 207, now 207). | | {color:red}-1{color} | whitespace | 0m 5s | The patch has 17 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 36s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 38s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 38s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | yarn tests | 54m 16s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 97m 46s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12748559/YARN-3983.4.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 0306d90 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8759/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8759/artifact/patchprocess/whitespace.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8759/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8759/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8759/console | This message was automatically generated. Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch, YARN-3983.4.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14652711#comment-14652711 ] Jian He commented on YARN-3983: --- - {{AllocationState}} format - applicationContainerAllocator- containerAllocator - simplify a bit {code} // Only reset opportunities when we FIRST allocate the container. (When // reservedContainer != null, it's not the first time) if ((allocationResult.state == AllocationState.Allocated || allocationResult.state == AllocationState.Reserved) reservedContainer == null) { {code} - schedulingMode and clusterResource parameters not used {code} private ContainerAllocation handleNewContainerAllocation( ContainerAllocation allocationResult, Resource clusterResource, FiCaSchedulerNode node, SchedulingMode schedulingMode, Priority priority, RMContainer reservedContainer, Container container) { {code} Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650128#comment-14650128 ] Hadoop QA commented on YARN-3983: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 15m 37s | Findbugs (version ) appears to be broken on trunk. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 47s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 10m 4s | There were no new javadoc warning messages. | | {color:red}-1{color} | release audit | 0m 20s | The applied patch generated 1 release audit warnings. | | {color:green}+1{color} | checkstyle | 0m 26s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 0m 4s | The patch has 17 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 25s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 27s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | yarn tests | 52m 51s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 90m 39s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12748251/YARN-3983.3.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / d311a38 | | Release Audit | https://builds.apache.org/job/PreCommit-YARN-Build/8745/artifact/patchprocess/patchReleaseAuditProblems.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8745/artifact/patchprocess/whitespace.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8745/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8745/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8745/console | This message was automatically generated. Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3983.1.patch, YARN-3983.2.patch, YARN-3983.3.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648178#comment-14648178 ] Hadoop QA commented on YARN-3983: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 16m 11s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 7m 44s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 42s | There were no new javadoc warning messages. | | {color:red}-1{color} | release audit | 0m 19s | The applied patch generated 1 release audit warnings. | | {color:red}-1{color} | checkstyle | 0m 47s | The applied patch generated 30 new checkstyle issues (total was 54, now 55). | | {color:red}-1{color} | whitespace | 0m 2s | The patch has 16 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 21s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 24s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | yarn tests | 49m 29s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 87m 37s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA | | | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | | | hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter | | | hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService | | Timed out tests | org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12748038/YARN-3983.2.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 91b42e7 | | Release Audit | https://builds.apache.org/job/PreCommit-YARN-Build/8720/artifact/patchprocess/patchReleaseAuditProblems.txt | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8720/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8720/artifact/patchprocess/whitespace.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8720/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8720/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8720/console | This message was automatically generated. Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3983.1.patch, YARN-3983.2.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648785#comment-14648785 ] Jian He commented on YARN-3983: --- - I suggest not set the state implicitly in the constructor. It’s quite confusing which constructor indicates which state. Setting it explicitly by caller makes code easier to read. {code} public ContainerAllocation(RMContainer containerToBeUnreserved) { this(containerToBeUnreserved, null, AllocationState.QUEUE_SKIPPED); } public ContainerAllocation(RMContainer containerToBeUnreserved, Resource resourceToBeAllocated) { this(containerToBeUnreserved, resourceToBeAllocated, AllocationState.SUCCEEDED); } {code} - reserved container returns SUCCEEDED state? {code} ContainerAllocation result = new ContainerAllocation(null, request.getCapability()); result.reserved = true; result.containerNodeType = type; {code} Earlier below code will not be invoked for reserved container, now it gets invoked {code} if (allocationResult.state == AllocationState.SUCCEEDED) { // Don't reset scheduling opportunities for offswitch assignments // otherwise the app will be delayed for each non-local assignment. // This helps apps with many off-cluster requests schedule faster. if (allocationResult.containerNodeType != NodeType.OFF_SWITCH) { if (LOG.isDebugEnabled()) { LOG.debug(Resetting scheduling opportunities); } application.resetSchedulingOpportunities(priority); } // Non-exclusive scheduling opportunity is different: we need reset // it every time to make sure non-labeled resource request will be // most likely allocated on non-labeled nodes first. application.resetMissedNonPartitionedRequestSchedulingOpportunity(priority); } {code} - AllocationState#SUCCEEDED - ALLOCTED. - reserved boolean flag can be changed to be Allocateion#RESERVED state. - CSAssignment#NULL_ASSIGNMENT not used, remove - comments does not match method name {code} * doAllocation needs to handle following stuffs: {code} . - Below code was originally outside of the priorities loop {code} if (SchedulerAppUtils.isBlacklisted(application, node, LOG)) { return ContainerAllocation.APP_SKIPPED; } {code} - below code can be changed to use null check ? {code} if (Resources.greaterThan(rc, clusterResource, assigned.getResourceToBeAllocated(), Resources.none())) { {code} - move the for loop into {{applicationContainerAllocator.allocate}} method {code} for (Priority priority : getPriorities()) { {code} - return ContainerAllocation.QUEUE_SKIPPED, though logically correct, semantically is incorrect, it should return Priority_Skipped {code} private ContainerAllocation assignNodeLocalContainers( Resource clusterResource, ResourceRequest nodeLocalResourceRequest, FiCaSchedulerNode node, Priority priority, RMContainer reservedContainer, SchedulingMode schedulingMode, ResourceLimits currentResoureLimits) { if (canAssign(priority, node, NodeType.NODE_LOCAL, reservedContainer)) { return assignContainer(clusterResource, node, priority, nodeLocalResourceRequest, NodeType.NODE_LOCAL, reservedContainer, schedulingMode, currentResoureLimits); } return ContainerAllocation.QUEUE_SKIPPED; } // check if the resource request can access the label if (!SchedulerUtils.checkResourceRequestMatchingNodePartition(request, node.getPartition(), schedulingMode)) { // this is a reserved container, but we cannot allocate it now according // to label not match. This can be caused by node label changed // We should un-reserve this container. return new ContainerAllocation(rmContainer); } {code} - APP_SKIPPED? original code seems skipping the priority {code} // Does the application need this resource? if (allocatedContainer == null) { // Skip this app if we failed to allocate. ContainerAllocation ret = new ContainerAllocation(allocationResult.containerToBeUnreserved); ret.state = AllocationState.APP_SKIPPED; return ret; } {code} Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3983.1.patch, YARN-3983.2.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14646835#comment-14646835 ] Hadoop QA commented on YARN-3983: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 18m 54s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 1 new or modified test files. | | {color:green}+1{color} | javac | 9m 41s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 12m 3s | There were no new javadoc warning messages. | | {color:red}-1{color} | release audit | 0m 24s | The applied patch generated 1 release audit warnings. | | {color:red}-1{color} | checkstyle | 1m 8s | The applied patch generated 41 new checkstyle issues (total was 54, now 66). | | {color:red}-1{color} | whitespace | 0m 2s | The patch has 16 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 2m 0s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 39s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 51s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | yarn tests | 54m 36s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 101m 24s | | \\ \\ || Reason || Tests || | Failed unit tests | hadoop.yarn.server.resourcemanager.rmapp.TestRMAppTransitions | | Timed out tests | org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12747851/YARN-3983.1.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / c020b62 | | Release Audit | https://builds.apache.org/job/PreCommit-YARN-Build/8709/artifact/patchprocess/patchReleaseAuditProblems.txt | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8709/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8709/artifact/patchprocess/whitespace.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8709/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8709/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8709/console | This message was automatically generated. Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3983.1.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14647150#comment-14647150 ] Jian He commented on YARN-3983: --- thanks Wangda ! some comments on the patch - ApplicationResourceAllocator - ContainerAllocator - NewContainerAllocator - RegularContainerAllocator - internalPreAllocation - preAllocate - move the assingContainersOnNode into internalApplyAllocation - internalApplyAllocation - doAllocation - doAllocation - allocate - AllocatorAllocationResult - ContainerAllocation - SKIPPED_APP - SKIP_APP; similarly for others - this.resourceToBeAllocated can be set null; the caller can check whether null or not {code} if (resourceToBeAllocated == null) { this.resourceToBeAllocated = Resources.none(); } else { this.resourceToBeAllocated = resourceToBeAllocated; } {code} - AllocatorAllocationResult#allocateNodeType - AllocatorAllocationResult#containerNodeType - Fix FiCaSchedulerApp#assignContainer method format and remove the unused createdContainer parameter - handleNewContainerReservation does not need be a separate method; - getCSAssignmentFromAllocateResult can be part of doAllocation. Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan Attachments: YARN-3983.1.patch While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3983) Make CapacityScheduler to easier extend application allocation logic
[ https://issues.apache.org/jira/browse/YARN-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14643448#comment-14643448 ] Wangda Tan commented on YARN-3983: -- Start working on this JIRA, will upload patch for review shortly. Make CapacityScheduler to easier extend application allocation logic Key: YARN-3983 URL: https://issues.apache.org/jira/browse/YARN-3983 Project: Hadoop YARN Issue Type: Bug Reporter: Wangda Tan Assignee: Wangda Tan While working on YARN-1651 (resource allocation for increasing container), I found it is very hard to extend existing CapacityScheduler resource allocation logic to support different types of resource allocation. For example, there's a lot of differences between increasing a container and allocating a container: - Increasing a container doesn't need to check locality delay. - Increasing a container doesn't need to build/modify a resource request tree (ANY-RACK/HOST). - Increasing a container doesn't need to check allocation/reservation starvation (see {{shouldAllocOrReserveNewContainer}}). - After increasing a container is approved by scheduler, it need to update an existing container token instead of creating new container. And there're lots of similarities when allocating different types of resources. - User-limit/queue-limit will be enforced for both of them. - Both of them needs resource reservation logic. (Maybe continuous reservation looking is needed for both of them). The purpose of this JIRA is to make easier extending CapacityScheduler resource allocation logic to support different types of resource allocation, make common code reusable, and also better code organization. -- This message was sent by Atlassian JIRA (v6.3.4#6332)