[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493934#comment-14493934 ] Hudson commented on YARN-3443: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #163 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/163/]) YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. (junping_du: rev 838b06ac87339494cea706d4a97e7f5383bdc442) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerChain.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerException.java * hadoop-yarn-project/CHANGES.txt Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Fix For: 2.8.0 Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493951#comment-14493951 ] Hudson commented on YARN-3443: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2095 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2095/]) YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. (junping_du: rev 838b06ac87339494cea706d4a97e7f5383bdc442) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationException.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerChain.java Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Fix For: 2.8.0 Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following : 1)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493981#comment-14493981 ] Hudson commented on YARN-3443: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #154 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/154/]) YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. (junping_du: rev 838b06ac87339494cea706d4a97e7f5383bdc442) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationException.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerChain.java Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Fix For: 2.8.0 Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493967#comment-14493967 ] Hudson commented on YARN-3443: -- FAILURE: Integrated in Hadoop-Yarn-trunk #897 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/897/]) YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. (junping_du: rev 838b06ac87339494cea706d4a97e7f5383bdc442) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerChain.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerException.java Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Fix For: 2.8.0 Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following : 1)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494202#comment-14494202 ] Hudson commented on YARN-3443: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #164 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/164/]) YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. (junping_du: rev 838b06ac87339494cea706d4a97e7f5383bdc442) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerChain.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Fix For: 2.8.0 Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494271#comment-14494271 ] Hudson commented on YARN-3443: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2113 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2113/]) YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. (junping_du: rev 838b06ac87339494cea706d4a97e7f5383bdc442) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerChain.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationException.java Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Fix For: 2.8.0 Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[
https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492937#comment-14492937
]
Hadoop QA commented on YARN-3443:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725010/YARN-3443.005.patch
against trunk revision 7fc50e2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
43 warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/7318//artifact/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7318//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7318//console
This message is automatically generated.
Create a 'ResourceHandler' subsystem to ease addition of support for new
resource types on the NM
-
Key: YARN-3443
URL: https://issues.apache.org/jira/browse/YARN-3443
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Sidharta Seethana
Assignee: Sidharta Seethana
Attachments: YARN-3443.001.patch, YARN-3443.002.patch,
YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch
Today, support for CPU and memory as resources (on linux) are implemented in
a way that cannot be easily extended to other new resource types (e.g
network/disk). For example, some functionality cgroups functionality is
implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler.
CPU specific functionality is also implemented in CgroupsLCEResourcesHandler
- using this handler automatically enables CPU as a resource. Some cgroups
functionality requires elevated/super-user privileges and needs to be
implemented via the container-executor binary. Implementing support for a new
resource type in linux using the existing classes/mechanisms would be messy
(for example, we might have to significantly modify/bloat
CgroupsLCEResourceHandler). As an alternative, we have implemented a new
‘ResourceHandler’ mechanism that makes things cleaner and enables easier
addition of new resource types. When adding support for a new resource type
in the NM (from an isolation/enforcement perspective), there are three
different pieces required :
1) generic cgroups utilities that can be re-used across multiple resource
handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be
able to create new cgroups, update cgroup params, read cgroup params etc.
2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires
super-user privileges and is implemented by container-executor binary
3) Implementation that is specific to a resource type ( i.e network, disk
would each have an implementation that provides isolation/enforcement for
that resource type)
Corresponding to the three pieces listed above, the patch for YARN-3443
provides the following :
1) cgroups functionality that can be used across different resource types.
CGroupsHandler.java specifies the interface and implementation is in
CGroupsHandlerImpl.java . New cgroups controller types can be easily added to
CGroupsHandler.java as and when necessary
2) PrivilegedOperation.java and PrivilegedOperationExecutor.java wrap the
container-executor binary and provide a way of executing operations that
require elevated privileges. There are also utility functions that help
‘batching’ of certain kinds of operations in order to avoid multiple
invocations of the container-executor binary
3) ResourceHandler.java specifies an interface that custom resource handlers
are expected to implement. This interface provides hooks for various
operations during a container lifecyle - bootstrap, preStart, postComplete,
reAcquire, teardown. Each of these hooks return a list of privileged
operations - this is done so that the
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492978#comment-14492978 ] Sidharta Seethana commented on YARN-3443: - The javadoc warnings are unrelated to this patch - I don't find any mention of the classes included in the patch. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following : 1) cgroups functionality that can be used across different resource types. CGroupsHandler.java specifies the interface and implementation is in CGroupsHandlerImpl.java . New cgroups controller types can be easily added to CGroupsHandler.java as and when necessary 2) PrivilegedOperation.java and PrivilegedOperationExecutor.java wrap the container-executor binary and provide a way of executing operations that require elevated privileges. There are also utility functions that help ‘batching’ of certain kinds of operations in order to avoid multiple invocations of the container-executor binary 3) ResourceHandler.java specifies an interface that custom resource handlers are expected to implement. This interface provides hooks for various operations during a container lifecyle - bootstrap, preStart, postComplete, reAcquire, teardown. Each of these hooks return a list of privileged operations - this is done so that the resulting set of privileged operations can be batched for performance reasons, if necessary. ResourceHandlerChain.java provides a simple chaining mechanism across multiple resource handlers. This is useful when multiple resource handlers are in place. They can be chained in sequence - e.g cpu, network, disk . A resource handler chain would hook in directly into LCE at various points in the container life cycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493401#comment-14493401 ] Junping Du commented on YARN-3443: -- +1. Latest patch LGTM. Double check all javadoc warnings under hadoop-yarn-server-nodemanager component, nothing is related to this patch (all under hadoop-yarn-server-nodemanager). Will commit it shortly. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following : 1) cgroups functionality that can be used across different resource types. CGroupsHandler.java specifies the interface and implementation is in CGroupsHandlerImpl.java . New cgroups controller types can be easily added to CGroupsHandler.java as and when necessary 2) PrivilegedOperation.java and PrivilegedOperationExecutor.java wrap the container-executor binary and provide a way of executing operations that require elevated privileges. There are also utility functions that help ‘batching’ of certain kinds of operations in order to avoid multiple invocations of the container-executor binary 3) ResourceHandler.java specifies an interface that custom resource handlers are expected to implement. This interface provides hooks for various operations during a container lifecyle - bootstrap, preStart, postComplete, reAcquire, teardown. Each of these hooks return a list of privileged operations - this is done so that the resulting set of privileged operations can be batched for performance reasons, if necessary. ResourceHandlerChain.java provides a simple chaining mechanism across multiple resource handlers. This is useful when multiple resource handlers are in place. They can be chained in sequence - e.g cpu, network, disk . A resource handler chain would hook in directly into LCE at various points in the container life cycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14493411#comment-14493411 ] Hudson commented on YARN-3443: -- FAILURE: Integrated in Hadoop-trunk-Commit #7579 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7579/]) YARN-3443. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM. Contributed by Sidharta Seethana. (junping_du: rev 838b06ac87339494cea706d4a97e7f5383bdc442) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/TestPrivilegedOperationExecutor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerChain.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/TestCGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/privileged/PrivilegedOperationException.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/CGroupsHandlerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/ResourceHandlerException.java * hadoop-yarn-project/CHANGES.txt Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Fix For: 2.8.0 Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch, YARN-3443.005.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following : 1)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492643#comment-14492643 ] Vinod Kumar Vavilapalli commented on YARN-3443: --- bq. I would prefer we separate interfaces with implementation. I've tried to get this done this way offline, but we unfortunately came to the conclusion that this may not be straightforward given the tight coupling between LCE, CgroupsHandler, container-executor binary etc. I think this is a reasonable first cut at cleaning up things for Linux, with the goal of adding more resource types like disk to Linux less painful. Existing resource-types added to Windows (YARN-2190) are already laid out in some fashion, I'd not do another refactor before carefully looking at how to structure them together. Let's keep the focus of this JIRA to extend resources for Linux and branch the cross-platform refactoring off to its own JIRA. IAC, let's make sure that none of this internal packaging structure leaks through to the end-users via configuration property names or documentation. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following : 1) cgroups functionality that can be used across different resource types. CGroupsHandler.java specifies the interface and implementation is in CGroupsHandlerImpl.java . New cgroups controller types can be easily added to CGroupsHandler.java as and when necessary 2) PrivilegedOperation.java and PrivilegedOperationExecutor.java wrap the container-executor binary and provide a way of executing operations that require elevated privileges. There are also utility functions that help ‘batching’ of certain kinds of operations in order to avoid multiple invocations of the container-executor binary 3) ResourceHandler.java specifies an interface that custom resource handlers are expected to implement. This interface provides hooks for various operations during a container lifecyle - bootstrap, preStart, postComplete, reAcquire, teardown. Each of these hooks return a list of privileged operations - this is done so that the resulting set of privileged operations can be batched for performance reasons, if necessary. ResourceHandlerChain.java provides a simple chaining mechanism across multiple resource handlers. This is useful when multiple resource handlers are in place. They can be chained in sequence - e.g cpu, network, disk . A resource handler chain would hook in directly into LCE at various points in the container life cycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492669#comment-14492669 ] Junping Du commented on YARN-3443: -- Thanks [~vinodkv] for comments and confirmation on this. I just want to make sure we don't miss other platforms/implementations in our previous discussions/design. I am OK with keep current package structure for now - if needed, we can do more refactor later when Windows is ready in future. [~sidharta-s], would you address my other comments in new patch? Thx! Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch Today, support for CPU and memory as resources (on linux) are implemented in a way that cannot be easily extended to other new resource types (e.g network/disk). For example, some functionality cgroups functionality is implemented in LCE (mountCgroups) and the rest in CgroupsLCEResourcesHandler. CPU specific functionality is also implemented in CgroupsLCEResourcesHandler - using this handler automatically enables CPU as a resource. Some cgroups functionality requires elevated/super-user privileges and needs to be implemented via the container-executor binary. Implementing support for a new resource type in linux using the existing classes/mechanisms would be messy (for example, we might have to significantly modify/bloat CgroupsLCEResourceHandler). As an alternative, we have implemented a new ‘ResourceHandler’ mechanism that makes things cleaner and enables easier addition of new resource types. When adding support for a new resource type in the NM (from an isolation/enforcement perspective), there are three different pieces required : 1) generic cgroups utilities that can be re-used across multiple resource handler ( e.g for CPU, Network, Disk). For example for net_cls we want to be able to create new cgroups, update cgroup params, read cgroup params etc. 2) A mechanism to execute ‘PrivilegedOperation’s whose functionality requires super-user privileges and is implemented by container-executor binary 3) Implementation that is specific to a resource type ( i.e network, disk would each have an implementation that provides isolation/enforcement for that resource type) Corresponding to the three pieces listed above, the patch for YARN-3443 provides the following : 1) cgroups functionality that can be used across different resource types. CGroupsHandler.java specifies the interface and implementation is in CGroupsHandlerImpl.java . New cgroups controller types can be easily added to CGroupsHandler.java as and when necessary 2) PrivilegedOperation.java and PrivilegedOperationExecutor.java wrap the container-executor binary and provide a way of executing operations that require elevated privileges. There are also utility functions that help ‘batching’ of certain kinds of operations in order to avoid multiple invocations of the container-executor binary 3) ResourceHandler.java specifies an interface that custom resource handlers are expected to implement. This interface provides hooks for various operations during a container lifecyle - bootstrap, preStart, postComplete, reAcquire, teardown. Each of these hooks return a list of privileged operations - this is done so that the resulting set of privileged operations can be batched for performance reasons, if necessary. ResourceHandlerChain.java provides a simple chaining mechanism across multiple resource handlers. This is useful when multiple resource handlers are in place. They can be chained in sequence - e.g cpu, network, disk . A resource handler chain would hook in directly into LCE at various points in the container life cycle. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[
https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486481#comment-14486481
]
Junping Du commented on YARN-3443:
--
Thanks [~sidharta-s] for delivering the patch and [~vvasudev] for reviewing.
Just quickly walk though the patch, the patch looks good in overall. Some minor
comments below, but more comments may come later:
{code}
+if (LOG.isWarnEnabled()) {
+ LOG.warn(Invalid argument: + arg);
+}
{code}
We typically log message directly on INFO or above level (like: warn, error),
not necessary for additional check for LOG.isWarnEnabled(). Also there are some
other places for checking LOG.isWarnEnabled() or even LOG.isErrorEnabled()
which should be unnecessary.
{code}
+@InterfaceAudience.Private
+@InterfaceStability.Unstable class CGroupsHandlerImpl
+implements CGroupsHandler {
{code}
We should separate annotation from the line of calss definition. Also, it
should be a public class?
In ResourceHandler.java (and other places, like: ResourceHandlerChain)
{code}
+ ListPrivilegedOperation teardown() throws ResourceHandlerException;
{code}
teardown = tearDown
Create a 'ResourceHandler' subsystem to ease addition of support for new
resource types on the NM
-
Key: YARN-3443
URL: https://issues.apache.org/jira/browse/YARN-3443
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Sidharta Seethana
Assignee: Sidharta Seethana
Attachments: YARN-3443.001.patch, YARN-3443.002.patch,
YARN-3443.003.patch, YARN-3443.004.patch
The current cgroups implementation is closely tied to supporting CPU as a
resource . We need to separate out CGroups support as well a provide a simple
ResourceHandler subsystem that will enable us to add support for new resource
types on the NM - e.g Network, Disk etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14486499#comment-14486499 ] Sidharta Seethana commented on YARN-3443: - Thanks for the review, [~djp] . Responses below : 1. I was trying to keep the 'enabled' check consistent between various log levels as a general practice. I could remove it, though. 2. We should separate annotation from the line of calss definition. - this was by accident. I'll fix it. Its not a public class by design, it is meant to be used only within the resources package (Custom resource handlers are added to this package). 3. teardown is one word, hence no camel-case :) I'll wait for further review comments from you before uploading a new patch, thanks! Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch The current cgroups implementation is closely tied to supporting CPU as a resource . We need to separate out CGroups support as well a provide a simple ResourceHandler subsystem that will enable us to add support for new resource types on the NM - e.g Network, Disk etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[
https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482688#comment-14482688
]
Hadoop QA commented on YARN-3443:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12723549/YARN-3443.004.patch
against trunk revision 3fb5abf.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7234//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7234//console
This message is automatically generated.
Create a 'ResourceHandler' subsystem to ease addition of support for new
resource types on the NM
-
Key: YARN-3443
URL: https://issues.apache.org/jira/browse/YARN-3443
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Sidharta Seethana
Assignee: Sidharta Seethana
Attachments: YARN-3443.001.patch, YARN-3443.002.patch,
YARN-3443.003.patch, YARN-3443.004.patch
The current cgroups implementation is closely tied to supporting CPU as a
resource . We need to separate out CGroups support as well a provide a simple
ResourceHandler subsystem that will enable us to add support for new resource
types on the NM - e.g Network, Disk etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483489#comment-14483489 ] Varun Vasudev commented on YARN-3443: - +1, lgtm for the latest patch. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch, YARN-3443.004.patch The current cgroups implementation is closely tied to supporting CPU as a resource . We need to separate out CGroups support as well a provide a simple ResourceHandler subsystem that will enable us to add support for new resource types on the NM - e.g Network, Disk etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14481471#comment-14481471 ] Sidharta Seethana commented on YARN-3443: - Thanks for the review, [~vvasudev] . Responses below : 1. I could change such log lines to use StringBuffer everywhere. However, I think a metric based on calls to LOG.warn()/LOG.error() does not accurately reflect the warn/error 'even't count. 2. I'll add an info log line to the if block. I had added this block for better readability since the behavior here is a little different from the current implementation in CgroupsLCEResourcesHandler 3. - 4. Same comment as 1. 5. Sure, I can make this change. 6. This is by design. Otherwise, every resource handler implementation that uses cgroups will have to check if cgroup mounting is enabled or not (which is error-prone). It seemed better to instead ignore a mount request when cgroup mounting is disabled. 7. I'll fix it. 8. I'll fix it. 9. I'll fix it. 10. Yikes. Yes, I'll fix it. Not sure how this one got through. 11. I had added it for clarity, but maybe it isn't necessary. I'll remove it. 12. I'll fix it. 13. I applied a formatter to all file before creating the patch - but, I'll verify. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Attachments: YARN-3443.001.patch, YARN-3443.002.patch The current cgroups implementation is closely tied to supporting CPU as a resource . We need to separate out CGroups support as well a provide a simple ResourceHandler subsystem that will enable us to add support for new resource types on the NM - e.g Network, Disk etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[
https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482026#comment-14482026
]
Hadoop QA commented on YARN-3443:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12723436/YARN-3443.003.patch
against trunk revision 28bebc8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7228//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7228//console
This message is automatically generated.
Create a 'ResourceHandler' subsystem to ease addition of support for new
resource types on the NM
-
Key: YARN-3443
URL: https://issues.apache.org/jira/browse/YARN-3443
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Sidharta Seethana
Assignee: Sidharta Seethana
Attachments: YARN-3443.001.patch, YARN-3443.002.patch,
YARN-3443.003.patch
The current cgroups implementation is closely tied to supporting CPU as a
resource . We need to separate out CGroups support as well a provide a simple
ResourceHandler subsystem that will enable us to add support for new resource
types on the NM - e.g Network, Disk etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[ https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14482591#comment-14482591 ] Varun Vasudev commented on YARN-3443: - Minor documentation fixes, everything else looks good. # In PrivilegedOperationExecutor.java, getPrivilegedOperationExecutionCommand is documented as throwing ExitCodeException but it doesn't throw it. # In CGroupsHandler.java, the documentation for createCGroup is missing descriptions for controller and path. Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM - Key: YARN-3443 URL: https://issues.apache.org/jira/browse/YARN-3443 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Reporter: Sidharta Seethana Assignee: Sidharta Seethana Attachments: YARN-3443.001.patch, YARN-3443.002.patch, YARN-3443.003.patch The current cgroups implementation is closely tied to supporting CPU as a resource . We need to separate out CGroups support as well a provide a simple ResourceHandler subsystem that will enable us to add support for new resource types on the NM - e.g Network, Disk etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[
https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14480958#comment-14480958
]
Varun Vasudev commented on YARN-3443:
-
Thanks for uploading the patch [~sidharta-s]!
Some comments
# In PrivilegedOperationExecutor.java
{noformat}
catch (ExitCodeException e) {
if (LOG.isWarnEnabled()) {
LOG.warn(Shell execution returned exit code: + exec.getExitCode());
LOG.warn(Privileged Execution Operation Output:);
LOG.warn(exec.getOutput());
}
}
{noformat}
Can you merge the warnings into one warn message and add the exception to the
log message? This is so that alerts based on the number of errors/warnings
reflect the actual number.
# In CGroupHandlerImpl.java
{noformat}
if (enableCGroupMount) {
//nothing to do here - we support 'deferred' mounting of specific
//controllers - we'll populate the path for a given controller when an
//explicit mountCGroupController request is issued.
} else {
//cluster admins are expected to have mounted controllers in specific
//locations - we'll attempt to figure out mount points
initializeControllerPathsFromMtab();
}
{noformat}
Can we remove the empty if block?
# In CGroupHandlerImpl.java
{noformat}
LOG.error(Mount point Based on mtab file : + MTAB_FILE);
LOG.error(Controller mount point not writable for: + name);
{noformat}
Merge the log messages
# In CGroupHandlerImpl.java
{noformat}
LOG.warn(Failed to initialize controller paths!);
LOG.warn(e);
{noformat}
Merge the log messages
# In CGroupHandlerImpl.java
In initializeControllerPathsFromMtab, when the error and warnings are logged,
should we throw an exception, since the functionality being requested will not
work?
# In CGroupHandlerImpl.java
In mountCGroupController
{noformat}
if (!enableCGroupMount) {
if (LOG.isWarnEnabled()) {
LOG.warn(CGroup mounting is disabled - ignoring mount request for: +
controller.getName());
return;
}
}
{noformat}
Should this be an error with an exception being thrown since the functionality
being requested won't be possible?
# In CGroupHandlerImpl.java
{noformat}
@param file object referring to the cgroup to be deleted mislabelled
{noformat}
The param name is mislabeled. It should be cgf instead of file
# In ResourceHandler.java
{noformat}
* @param containerId if of the container being reacquired.
* @return
* @throws ResourceHandlerException
{noformat}
return is missing description
# In ResourceHandler.java
{noformat}
ListPrivilegedOperation postComplete(ContainerId containerId) throws
ResourceHandlerException;
;
{noformat}
There's an extra semi-colon.
# In TestPrivilegedOperationExecutor.java
{noformat}
expectedPath = customExecutorPath;
containerExePath = PrivilegedOperationExecutor
.getContainerExecutorExecutablePath(confWithExecutorPath);
Assert.assertEquals(expectedPath, customExecutorPath);
{noformat}
containerExePath is not used. The assert will always succeed since you set them
to be equal before calling the function. Did you mean to compare
containerExePath and expectedPath?
# In TestPrivilegedOperationExecutor.java
{noformat}
prefixLength + 0
{noformat}
Remove the 0?
# In TestPrivilegedOperationExecutor.java
{noformat}
try {
PrivilegedOperationExecutor.squashCGroupOperations(ops);
} catch (PrivilegedOperationException e) {
if (LOG.isInfoEnabled()) {
LOG.info(Caught expected exception : + e);
}
caughtPrivilegedOperationException = true;
}
Assert.assertTrue(Expected squash operation to fail!,
caughtPrivilegedOperationException);
{noformat}
Instead of using the boolean to check if the exception was thrown, you can use
Assert.fail in the try block. This pattern occurs multiple times.
# I think the formatting is a bit off in a couple of files. Can you apply the
formatter and check?
Create a 'ResourceHandler' subsystem to ease addition of support for new
resource types on the NM
-
Key: YARN-3443
URL: https://issues.apache.org/jira/browse/YARN-3443
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Sidharta Seethana
Assignee: Sidharta Seethana
Attachments: YARN-3443.001.patch, YARN-3443.002.patch
The current cgroups implementation is closely tied to supporting CPU as a
resource . We need to separate out CGroups support as well a provide a simple
ResourceHandler subsystem that will enable us to add support for new resource
types on the NM - e.g Network, Disk etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[
https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394200#comment-14394200
]
Hadoop QA commented on YARN-3443:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12709194/YARN-3443.002.patch
against trunk revision 72f6bd4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1150 javac
compiler warnings (more than the trunk's current 1148 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7211//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7211//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7211//console
This message is automatically generated.
Create a 'ResourceHandler' subsystem to ease addition of support for new
resource types on the NM
-
Key: YARN-3443
URL: https://issues.apache.org/jira/browse/YARN-3443
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Sidharta Seethana
Assignee: Sidharta Seethana
Attachments: YARN-3443.001.patch, YARN-3443.002.patch
The current cgroups implementation is closely tied to supporting CPU as a
resource . We need to separate out CGroups support as well a provide a simple
ResourceHandler subsystem that will enable us to add support for new resource
types on the NM - e.g Network, Disk etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3443) Create a 'ResourceHandler' subsystem to ease addition of support for new resource types on the NM
[
https://issues.apache.org/jira/browse/YARN-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394003#comment-14394003
]
Hadoop QA commented on YARN-3443:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12709150/YARN-3443.001.patch
against trunk revision bad070f.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1150 javac
compiler warnings (more than the trunk's current 1148 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7210//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7210//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7210//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7210//console
This message is automatically generated.
Create a 'ResourceHandler' subsystem to ease addition of support for new
resource types on the NM
-
Key: YARN-3443
URL: https://issues.apache.org/jira/browse/YARN-3443
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Reporter: Sidharta Seethana
Assignee: Sidharta Seethana
Attachments: YARN-3443.001.patch
The current cgroups implementation is closely tied to supporting CPU as a
resource . We need to separate out CGroups support as well a provide a simple
ResourceHandler subsystem that will enable us to add support for new resource
types on the NM - e.g Network, Disk etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
