[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716031#comment-14716031 ] Srikanth Kandula commented on YARN-3534: [~elgoiri], [~kasha], could you comment on extending this to also take in network and disk usage information? > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.8.0 > > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699678#comment-14699678 ] Hudson commented on YARN-3534: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2236 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2236/]) YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.8.0 > > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699661#comment-14699661 ] Hudson commented on YARN-3534: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #279 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/279/]) YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.8.0 > > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699640#comment-14699640 ] Hudson commented on YARN-3534: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2217 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2217/]) YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.8.0 > > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699363#comment-14699363 ] Hudson commented on YARN-3534: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #290 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/290/]) YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.8.0 > > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14699360#comment-14699360 ] Hudson commented on YARN-3534: -- FAILURE: Integrated in Hadoop-Yarn-trunk #1020 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1020/]) YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.8.0 > > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698806#comment-14698806 ] Inigo Goiri commented on YARN-3534: --- Thank you [~kasha] for taking care of the review and the commits. I'll be moving to propagating this info to the scheduler. > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.8.0 > > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698674#comment-14698674 ] Hudson commented on YARN-3534: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #286 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/286/]) YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.8.0 > > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698667#comment-14698667 ] Hudson commented on YARN-3534: -- FAILURE: Integrated in Hadoop-trunk-Commit #8311 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8311/]) YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha) (kasha: rev def12933b38efd5e47c5144b729c1a1496f09229) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/TestContainersMonitor.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitor.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeResourceMonitor.java > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: 2.8.0 > > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698663#comment-14698663 ] Karthik Kambatla commented on YARN-3534: +1 > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698442#comment-14698442
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 35s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 39s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 46s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 21s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 55s | Tests failed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 10s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 53m 33s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.util.TestRackResolver |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12750666/YARN-3534-18.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8dfec7a |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8851/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8851/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8851/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch,
> YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch,
> YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch,
> YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch,
> YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch,
> YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698419#comment-14698419 ] Inigo Goiri commented on YARN-3534: --- I uploaded 17 twice instead of 18... sorry. > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-18.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14698162#comment-14698162 ] Karthik Kambatla commented on YARN-3534: The latest patches (v17) don't seem to have the updates. [~elgoiri] - could you please verify? > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-2.patch, YARN-3534-3.patch, > YARN-3534-3.patch, YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch, > YARN-3534-7.patch, YARN-3534-8.patch, YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14663186#comment-14663186
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 52s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 45s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 51s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 20s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 55s | Tests failed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 8s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 53m 54s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.util.TestRackResolver |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749435/YARN-3534-17.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8802/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8802/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8802/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch,
> YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch,
> YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-2.patch, YARN-3534-3.patch,
> YARN-3534-3.patch, YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch,
> YARN-3534-7.patch, YARN-3534-8.patch, YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14663179#comment-14663179
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 45s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 41s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 45s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:green}+1{color} | whitespace | 0m 3s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 22s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 22s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 54s | Tests failed in
hadoop-yarn-common. |
| {color:red}-1{color} | yarn tests | 6m 7s | Tests failed in
hadoop-yarn-server-nodemanager. |
| | | 53m 35s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.util.TestRackResolver |
| | hadoop.yarn.server.nodemanager.TestDeletionService |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12749432/YARN-3534-17.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8f73bdd |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8801/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8801/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8801/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch,
> YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch,
> YARN-3534-17.patch, YARN-3534-17.patch, YARN-3534-2.patch, YARN-3534-3.patch,
> YARN-3534-3.patch, YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch,
> YARN-3534-7.patch, YARN-3534-8.patch, YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14663167#comment-14663167 ] Karthik Kambatla commented on YARN-3534: In yarn-default.xml, for the container-mon configs, we should add info to say - "If not set, the value for nm-resource-monitor configs will be used." > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-17.patch, YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, > YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, > YARN-3534-8.patch, YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14663147#comment-14663147 ] Inigo Goiri commented on YARN-3534: --- # I missed the Deprecated for the calculator... my bad. Fixed. # That's good; fixed. # The behavior for the CONTAINER_MON hasn't really changed, it's the node ones that are used a backups right? I fixed the default value and extended the resource monitor; not sure about the best wording though. > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch, > YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch, > YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch, > YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14663100#comment-14663100
]
Karthik Kambatla commented on YARN-3534:
Hopefully, last set of comments:
# We should also not deprecate NM_CONTAINER_MON_RESOURCE_CALCULATOR. Should
have been more explicit in my previous comment.
# ContainersMonitorImpl should prioritize container-specific configs and
fallback if they are not present. Something along these lines:
{code}
this.monitoringInterval =
conf.getLong(YarnConfiguration.NM_CONTAINER_MON_INTERVAL_MS,
conf.getLong(YarnConfiguration.NM_RESOURCE_MON_INTERVAL_MS,
YarnConfiguration.DEFAULT_NM_RESOURCE_MON_INTERVAL_MS));
Class clazz =
conf.getClass(YarnConfiguration.NM_CONTAINER_MON_RESOURCE_CALCULATOR,
conf.getClass(
YarnConfiguration.NM_MON_RESOURCE_CALCULATOR, null,
ResourceCalculatorPlugin.class),
ResourceCalculatorPlugin.class);
{code}
# yarn-default.xml needs to be updated for the CONTAINER_MON configs we are
changing the behavior of, and any default values there should be removed.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch,
> YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch,
> YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch,
> YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch,
> YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14650130#comment-14650130
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 36s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 37s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 48s | The applied patch generated 1
new checkstyle issues (total was 211, now 211). |
| {color:red}-1{color} | checkstyle | 2m 33s | The applied patch generated 2
new checkstyle issues (total was 28, now 27). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 20s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 22s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 10s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 53m 36s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748290/YARN-3534-16.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d311a38 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8747/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8747/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8747/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch,
> YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-16.patch,
> YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch,
> YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch,
> YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14650092#comment-14650092
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 46s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 46s | The applied patch generated 4
new checkstyle issues (total was 211, now 214). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 20s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 9s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 53m 43s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748272/YARN-3534-16.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / d311a38 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8742/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8742/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8742/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch,
> YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-2.patch, YARN-3534-3.patch,
> YARN-3534-3.patch, YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch,
> YARN-3534-7.patch, YARN-3534-8.patch, YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14649993#comment-14649993 ] Inigo Goiri commented on YARN-3534: --- I guess in #2 you meant yarn-default.xml, right? I left #4 as it was but I'm not sure if you were referring to leave it as it was in trunk; let me know. The fallback is kind of complicated; we can give it a try if you want. > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Attachments: YARN-3534-1.patch, YARN-3534-10.patch, > YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch, > YARN-3534-15.patch, YARN-3534-16.patch, YARN-3534-2.patch, YARN-3534-3.patch, > YARN-3534-3.patch, YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch, > YARN-3534-7.patch, YARN-3534-8.patch, YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642141#comment-14642141
]
Karthik Kambatla commented on YARN-3534:
Thanks for updating the patch, [~elgoiri]. The latest patch looks pretty good.
Few minor comments:
# Let us not deprecate {{NM_CONTAINER_MON_INTERVAL_MS}}. Deprecating the
default value seems reasonable. One might want to specify a different interval
for node monitoring and container monitoring.
# Make corresponding changes to yarn-site.xml
# {{NodeResourceMonitorImpl#serviceStop}}: the log message should say couldn't
wait for the thread to join, and may be we should log it as WARN instead of INFO
# ContainersMonitorImpl: Users could continue to use the deprecated configs. If
those are specified and the new ones are not, we should fallback to the
deprecated ones. If this becomes too complicated, I am comfortable with leaving
the configs for container-monitoring as is. We could file a follow-up JIRA to
clean these configs for Hadoop-3.x.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch,
> YARN-3534-15.patch, YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch,
> YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch,
> YARN-3534-8.patch, YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621621#comment-14621621
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 26s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 7m 46s | The applied patch generated 2
additional warning messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 19s | The applied patch generated 5
new checkstyle issues (total was 211, now 215). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 24s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 44s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 6m 6s | Tests failed in
hadoop-yarn-server-nodemanager. |
| | | 48m 0s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
|
| |
hadoop.yarn.server.nodemanager.containermanager.localizer.TestResourceLocalizationService
|
| | hadoop.yarn.server.nodemanager.containermanager.container.TestContainer |
| | hadoop.yarn.server.nodemanager.TestDeletionService |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12744609/YARN-3534-15.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 1a0752d |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8488/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8488/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8488/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch,
> YARN-3534-15.patch, YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch,
> YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch,
> YARN-3534-8.patch, YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621023#comment-14621023
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 13s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 2m 59s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12744528/YARN-3534-14.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / ac60483 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8483/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-14.patch,
> YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch,
> YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch,
> YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14585162#comment-14585162
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 30s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 19s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 41s | The applied patch generated 1
new checkstyle issues (total was 213, now 212). |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 20s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 3s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 53m 3s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12739505/YARN-3534-12.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / b8341f1 |
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8247/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8247/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8247/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-12.patch, YARN-3534-2.patch, YARN-3534-3.patch,
> YARN-3534-3.patch, YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch,
> YARN-3534-7.patch, YARN-3534-8.patch, YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571462#comment-14571462
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 18s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 18s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 44s | The applied patch generated 1
new checkstyle issues (total was 212, now 211). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 15s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 21s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 13s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 52m 52s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737322/YARN-3534-11.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / c59e745 |
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8180/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8180/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8180/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-11.patch, YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch,
> YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch,
> YARN-3534-8.patch, YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14571318#comment-14571318
]
Hadoop QA commented on YARN-3534:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 42s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 7m 34s | The applied patch generated 1
additional warning messages. |
| {color:red}-1{color} | javadoc | 9m 39s | The applied patch generated 3
additional warning messages. |
| {color:red}-1{color} | release audit | 0m 18s | The applied patch generated
1 release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 24s | The applied patch generated 9
new checkstyle issues (total was 212, now 220). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 18s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 21s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 1m 56s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 4s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 50m 53s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12737047/YARN-3534-10.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / c59e745 |
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavacWarnings.txt
|
| javadoc |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffJavadocWarnings.txt
|
| Release Audit |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8176/console |
This message was automatically generated.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-10.patch,
> YARN-3534-2.patch, YARN-3534-3.patch, YARN-3534-3.patch, YARN-3534-4.patch,
> YARN-3534-5.patch, YARN-3534-6.patch, YARN-3534-7.patch, YARN-3534-8.patch,
> YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[ https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566728#comment-14566728 ] Inigo Goiri commented on YARN-3534: --- I'll do the changes you propose but I have a quesiton about 1: right now, I don't have ResourceUtilization in thos patch, should I added to this one? I wouldn't add any of the PBImpl stuff here. > Collect memory/cpu usage on the node > > > Key: YARN-3534 > URL: https://issues.apache.org/jira/browse/YARN-3534 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Affects Versions: 2.7.0 >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Attachments: YARN-3534-1.patch, YARN-3534-2.patch, YARN-3534-3.patch, > YARN-3534-3.patch, YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch, > YARN-3534-7.patch, YARN-3534-8.patch, YARN-3534-9.patch > > Original Estimate: 336h > Remaining Estimate: 336h > > YARN should be aware of the resource utilization of the nodes when scheduling > containers. For this, this task will implement the collection of memory/cpu > usage on the node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3534) Collect memory/cpu usage on the node
[
https://issues.apache.org/jira/browse/YARN-3534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14566283#comment-14566283
]
Karthik Kambatla commented on YARN-3534:
Thanks for working on this, Inigo.
Few comments:
# Given these stats are to be sent to the RM in heartbeat, should we capture
the aggregate node resource usage in {{ResourceUtilization}}?
# Instead of adding a separate {{DEFAULT_NM_NODE_MON_INTERVAL_MS}}, we should
probably just re-use the default for container-monitor?
# Also, should we add another config
{{yarn.nodemanager.usage-monitor.interval-ms}} that both container-monitor and
node-monitor inherit unless specified otherwise? If that seems reasonable, we
should deprecate the default value for container-monitor-interval.
# For the monitoring thread,
## set the thread name?
## make it a daemon thread?
## on {{monitoringThread.join()}}, specify a timeout as well.
## in the corresponding catch-block, at least log that we couldn't wait until
the monitoring-thread is interrupted.
> Collect memory/cpu usage on the node
>
>
> Key: YARN-3534
> URL: https://issues.apache.org/jira/browse/YARN-3534
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Affects Versions: 2.7.0
>Reporter: Inigo Goiri
>Assignee: Inigo Goiri
> Attachments: YARN-3534-1.patch, YARN-3534-2.patch, YARN-3534-3.patch,
> YARN-3534-3.patch, YARN-3534-4.patch, YARN-3534-5.patch, YARN-3534-6.patch,
> YARN-3534-7.patch, YARN-3534-8.patch, YARN-3534-9.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> YARN should be aware of the resource utilization of the nodes when scheduling
> containers. For this, this task will implement the collection of memory/cpu
> usage on the node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
