[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623343#comment-14623343 ] Hudson commented on YARN-3445: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #253 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/253/]) YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) (mingma: rev 08244264c0583472b9c4e16591cfde72c6db62a2) * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * hadoop-yarn-project/CHANGES.txt Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Fix For: 2.8.0 Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch, YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445-v5.1.patch, YARN-3445-v5.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623347#comment-14623347 ] Hudson commented on YARN-3445: -- FAILURE: Integrated in Hadoop-Yarn-trunk #983 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/983/]) YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) (mingma: rev 08244264c0583472b9c4e16591cfde72c6db62a2) * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/CHANGES.txt Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Fix For: 2.8.0 Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch, YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445-v5.1.patch, YARN-3445-v5.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623384#comment-14623384 ] Hudson commented on YARN-3445: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #251 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/251/]) YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) (mingma: rev 08244264c0583472b9c4e16591cfde72c6db62a2) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Fix For: 2.8.0 Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch, YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445-v5.1.patch, YARN-3445-v5.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623457#comment-14623457 ] Hudson commented on YARN-3445: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2199 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2199/]) YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) (mingma: rev 08244264c0583472b9c4e16591cfde72c6db62a2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Fix For: 2.8.0 Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch, YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445-v5.1.patch, YARN-3445-v5.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623390#comment-14623390 ] Hudson commented on YARN-3445: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2180 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2180/]) YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) (mingma: rev 08244264c0583472b9c4e16591cfde72c6db62a2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/CHANGES.txt * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Fix For: 2.8.0 Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch, YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445-v5.1.patch, YARN-3445-v5.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14623414#comment-14623414 ] Hudson commented on YARN-3445: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #241 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/241/]) YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) (mingma: rev 08244264c0583472b9c4e16591cfde72c6db62a2) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Fix For: 2.8.0 Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch, YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445-v5.1.patch, YARN-3445-v5.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14622487#comment-14622487 ] Hudson commented on YARN-3445: -- FAILURE: Integrated in Hadoop-trunk-Commit #8148 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8148/]) YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) (mingma: rev 08244264c0583472b9c4e16591cfde72c6db62a2) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMNodeTransitions.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Fix For: 2.8.0 Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch, YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445-v5.1.patch, YARN-3445-v5.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620814#comment-14620814
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 41s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 4s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 20s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 14s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 0m 52s | Tests passed in
hadoop-sls. |
| {color:green}+1{color} | yarn tests | 51m 2s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 91m 24s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12744509/YARN-3445-v5.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fffb15b |
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8480/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8480/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8480/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8480/console |
This message was automatically generated.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch,
YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch,
YARN-3445-v5.1.patch, YARN-3445-v5.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14620855#comment-14620855 ] Ming Ma commented on YARN-3445: --- +1 on the latest patch. Thanks Junping. I will wait until tomorrow to commit in case [~vinodkv] or others have comments. Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch, YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445-v5.1.patch, YARN-3445-v5.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616906#comment-14616906
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 1s | Findbugs (version ) appears to
be broken on trunk. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 46s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 41s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 15s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 0m 52s | Tests passed in
hadoop-sls. |
| {color:red}-1{color} | yarn tests | 62m 54s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 102m 39s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestNodeLabelContainerAllocation
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12743963/YARN-3445-v5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e0febce |
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8448/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8448/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8448/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8448/console |
This message was automatically generated.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch,
YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch,
YARN-3445-v5.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617794#comment-14617794
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 6s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 48s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 53s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 6s | The applied patch generated 1
new checkstyle issues (total was 115, now 115). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 16s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 0m 51s | Tests passed in
hadoop-sls. |
| {color:red}-1{color} | yarn tests | 48m 6s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 89m 41s | |
\\
\\
|| Reason || Tests ||
| Timed out tests |
org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens |
| |
org.apache.hadoop.yarn.server.resourcemanager.ahs.TestRMApplicationHistoryWriter
|
| |
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12743963/YARN-3445-v5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / c9dd2ca |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8453/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8453/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8453/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8453/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8453/console |
This message was automatically generated.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch,
YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch,
YARN-3445-v5.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614979#comment-14614979
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 1s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 56s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 7s | The applied patch generated 2
new checkstyle issues (total was 115, now 116). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 17s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 0m 51s | Tests passed in
hadoop-sls. |
| {color:green}+1{color} | yarn tests | 51m 4s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 92m 42s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12743708/YARN-3445-v4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fc92d3e |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8437/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8437/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8437/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8437/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8437/console |
This message was automatically generated.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch,
YARN-3445-v3.patch, YARN-3445-v4.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615517#comment-14615517
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 50s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 53s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 24s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 26s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 16s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 57s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 39s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 52s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 0m 57s | Tests passed in
hadoop-sls. |
| {color:red}-1{color} | yarn tests | 54m 46s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 98m 4s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12743748/YARN-3445-v4.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fc92d3e |
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8442/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8442/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8442/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8442/console |
This message was automatically generated.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch,
YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616036#comment-14616036
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 25s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 8m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 11m 26s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 10s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 57s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 38s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 46s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 0m 55s | Tests passed in
hadoop-sls. |
| {color:green}+1{color} | yarn tests | 53m 32s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 98m 56s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12743748/YARN-3445-v4.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / bf89ddb |
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8445/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8445/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8445/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8445/console |
This message was automatically generated.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch,
YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616120#comment-14616120
]
Ming Ma commented on YARN-3445:
---
Thanks Junping. Can you please check if it really needs to take write lock for
the private methods {{addAppToRunningApps}} and {{removeAppFromRunningApps}}?
It looks like {{RMNodeImpl}}'s {{handle}} function already takes write lock for
any state transition, and these private methods will only be invoked from state
transition functions.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch,
YARN-3445-v3.patch, YARN-3445-v4.1.patch, YARN-3445-v4.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612543#comment-14612543 ] Junping Du commented on YARN-3445: -- Thanks for review and comments, [~mingma]! bq. That is around 10M entries. So it should be ok for RM. ApplicationId only contains int (4 bytes) and long (8 bytes) field. Even consider java object header, padding and PB object overhead, should be far less than 100 bytes. Agree that it should be fine even in large scale as mentioned scenario. bq. Do you need synchronizedList in the following list? It looks like the access of runningApplications are protected by RMNodeImpl's readLock and writeLock. Nice catch! Will replace synchronizedList will ArrayList and add some writeLocks (missing in previous patch). Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch, YARN-3445-v3.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606766#comment-14606766
]
Ming Ma commented on YARN-3445:
---
Thanks [~djp]. Quick questions:
* Regarding the extra memory this might take. Say you have 10K NM, 1000 apps
per NM ( an application will remain in a RMNodeImpl list even though there are
no more containers running on that NM, but the app still run on another NM.).
That is around 10M entries. So it should be ok for RM.
* Do you need synchronizedList in the following list? It looks like the access
of runningApplications are protected by RMNodeImpl's readLock and writeLock.
{noformat}
private final ListApplicationId runningApplications =
Collections.synchronizedList(new ArrayListApplicationId());
{noromat}
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch,
YARN-3445-v3.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14605139#comment-14605139
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 19s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 5s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 14s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 0m 53s | Tests passed in
hadoop-sls. |
| {color:green}+1{color} | yarn tests | 50m 48s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 92m 8s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12742445/YARN-3445-v3.1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / aad6a7d |
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8371/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8371/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8371/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8371/console |
This message was automatically generated.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.1.patch,
YARN-3445-v3.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554591#comment-14554591
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 13s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 48s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 53s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 6s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 57s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 0m 51s | Tests passed in
hadoop-sls. |
| {color:green}+1{color} | yarn tests | 52m 18s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 91m 44s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734462/YARN-3445-v3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a5def58 |
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8042/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8042/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8042/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8042/console |
This message was automatically generated.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Attachments: YARN-3445-v2.patch, YARN-3445-v3.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554623#comment-14554623 ] Junping Du commented on YARN-3445: -- [~vinodkv], can you help to review it again? Thanks! Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445-v3.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14534861#comment-14534861
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 52s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 3s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 5s | The applied patch generated 1
new checkstyle issues (total was 111, now 112). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 16 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 2m 1s | The patch appears to introduce 1
new Findbugs (version 2.0.3) warnings. |
| {color:red}-1{color} | tools/hadoop tests | 0m 20s | Tests failed in
hadoop-sls. |
| {color:red}-1{color} | yarn tests | 52m 30s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 91m 8s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-sls |
| | Unwritten field:NodeInfo.java:[line 140] |
| Failed unit tests | hadoop.yarn.sls.nodemanager.TestNMSimulator |
| | hadoop.yarn.sls.TestSLSRunner |
| | hadoop.yarn.sls.appmaster.TestAMSimulator |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12728844/YARN-3445-v2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6f62267 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/7807/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/7807/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-YARN-Build/7807/artifact/patchprocess/newPatchFindbugsWarningshadoop-sls.html
|
| hadoop-sls test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7807/artifact/patchprocess/testrun_hadoop-sls.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/7807/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/7807/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7807/console |
This message was automatically generated.
Cache runningApps in RMNode for getting running apps on given NodeId
Key: YARN-3445
URL: https://issues.apache.org/jira/browse/YARN-3445
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager, resourcemanager
Affects Versions: 2.7.0
Reporter: Junping Du
Assignee: Junping Du
Labels: BB2015-05-TBR
Attachments: YARN-3445-v2.patch, YARN-3445.patch
Per discussion in YARN-3334, we need filter out unnecessary collectors info
from RM in heartbeat response. Our propose is to add cache for runningApps in
RMNode, so RM only send collectors for local running apps back. This is also
needed in YARN-914 (graceful decommission) that if no running apps in NM
which is in decommissioning stage, it will get decommissioned immediately.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14530659#comment-14530659 ] Junping Du commented on YARN-3445: -- Thanks [~vinodkv] for review and comments, I just deliver a quick demo patch in YARN-3586 (target for YARN-2928) to demonstrate how patch here helps to improve RM get back NM with limited number of collector addresses. Can you help to take a look? BTW, YARN-3212 for gracefully decommission need this patch too, so our target branch here is still for trunk/branch-2. Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Labels: BB2015-05-TBR Attachments: YARN-3445-v2.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527480#comment-14527480 ] Vinod Kumar Vavilapalli commented on YARN-3445: --- With those comments out of the way, the patch looks good to me. But I'd still like to see a follow up JIRA and a patch that depend on this. Just to make sure we have it the right way. Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521578#comment-14521578 ] Junping Du commented on YARN-3445: -- Thanks for comments, [~vinodkv]! bq. logAggregationReportsForApps itself is a map of ApplicationID with a nested LogAggregationReport.ApplicationID - duplicate AppID information Are u suggest we should replace map with list in NodeHeartbeatRequest? I fully agree and I will suggest to do so in YARN-3505. bq. runningApplications in this patch In v2 patch, runningApplications is already removed. Kindly check the v2 patch again? bq. NodeStatus.keepAliveApplications I agree. This shouldn't be needed anymore after YARN-1402. I had the similar idea before in synced with Xuan but forget to put it on JIRA. May be we should file a separated JIRA to fix it? CC [~xgong]. Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14521893#comment-14521893 ] Jian He commented on YARN-3445: --- One other thing, the LogAggregationReport#(get/set)getNodeId can also be removed as it's not used anywhere. I'm also unsure about the usage of LogAggregationReport#(get/set)DiagnosticMessage as it's only set with an empty string. agree we can have a separate jira to fix this, preferably in the same 2.8 release. Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522058#comment-14522058 ] Vinod Kumar Vavilapalli commented on YARN-3445: --- bq. I agree. This shouldn't be needed anymore after YARN-1402. I had the similar idea before in synced with Xuan but forget to put it on JIRA. May be we should file a separated JIRA to fix it? keepAliveApplications cannot be removed as we need to support protocol compatibility. But the new ones you added for logs can be removed as they are new. Can you take this forward also on YARN-3505? bq. .. LogAggregationReport#(get/set)getNodeId .. LogAggregationReport#(get/set)DiagnosticMessage .. [~jianhe], can you file a ticket please? Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522060#comment-14522060 ] Vinod Kumar Vavilapalli commented on YARN-3445: --- bq. Jian He, can you file a ticket please? Actually, we can do this too on YARN-3505 as that is related to LogAggregationReport. Please leave a comment there.. Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520396#comment-14520396 ] Vinod Kumar Vavilapalli commented on YARN-3445: --- There is a too much of duplicate information already in NodeHeartbeatRequest, albeit for slightly different purposes. We need to consolidate the following (without breaking compatibility of previous releases), lest the heartbeat will become heavier and heavier. - logAggregationReportsForApps added, but not released yet -- logAggregationReportsForApps itself is a map of ApplicationID with a nested LogAggregationReport.ApplicationID - duplicate AppID information - runningApplications in this patch - NodeStatus.keepAliveApplications /cc [~jianhe] [~leftnoteasy] Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[
https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519367#comment-14519367
]
Hadoop QA commented on YARN-3445:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 15 line(s) that
end in whitespace. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 40s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 5m 20s | The applied patch generated 4
additional checkstyle issues. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 1m 59s | The patch appears to introduce
14 new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 0m 52s | Tests passed in
hadoop-sls. |
| {color:green}+1{color} | yarn tests | 52m 26s | Tests passed in
hadoop-yarn-server-resourcemanager. |
| | | 95m 0s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-sls |
| | Found reliance on default encoding in
org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String,
String):in
org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String,
String): new java.io.FileReader(String) At RumenToSLSConverter.java:[line 122]
|
| | Found reliance on default encoding in
org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String,
String):in
org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSLoadFile(String,
String): new java.io.FileWriter(String) At RumenToSLSConverter.java:[line 124]
|
| | Found reliance on default encoding in
org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSNodeFile(String):in
org.apache.hadoop.yarn.sls.RumenToSLSConverter.generateSLSNodeFile(String): new
java.io.FileWriter(String) At RumenToSLSConverter.java:[line 145] |
| | Found reliance on default encoding in
org.apache.hadoop.yarn.sls.SLSRunner.startAMFromSLSTraces(Resource, int):in
org.apache.hadoop.yarn.sls.SLSRunner.startAMFromSLSTraces(Resource, int): new
java.io.FileReader(String) At SLSRunner.java:[line 280] |
| | Unwritten field:NodeInfo.java:[line 140] |
| | Found reliance on default encoding in
org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.initMetrics():in
org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper.initMetrics():
new java.io.FileWriter(String) At ResourceSchedulerWrapper.java:[line 490] |
| | Found reliance on default encoding in new
org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper$MetricsLogRunnable(ResourceSchedulerWrapper):in
new
org.apache.hadoop.yarn.sls.scheduler.ResourceSchedulerWrapper$MetricsLogRunnable(ResourceSchedulerWrapper):
new java.io.FileWriter(String) At ResourceSchedulerWrapper.java:[line 695] |
| | Found reliance on default encoding in
org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.initMetrics():in
org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.initMetrics(): new
java.io.FileWriter(String) At SLSCapacityScheduler.java:[line 493] |
| | Found reliance on default encoding in new
org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler$MetricsLogRunnable(SLSCapacityScheduler):in
new
org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler$MetricsLogRunnable(SLSCapacityScheduler):
new java.io.FileWriter(String) At SLSCapacityScheduler.java:[line 698] |
| | Found reliance on default encoding in
org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromNodeFile(String):in
org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromNodeFile(String): new
java.io.FileReader(String) At SLSUtils.java:[line 119] |
| | Found reliance on default encoding in
org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromSLSTrace(String):in
org.apache.hadoop.yarn.sls.utils.SLSUtils.parseNodesFromSLSTrace(String): new
java.io.FileReader(String) At SLSUtils.java:[line 92] |
| | Class org.apache.hadoop.yarn.sls.web.SLSWebApp defines non-transient
non-serializable instance field handleOperTimecostHistogramMap In
SLSWebApp.java:instance field handleOperTimecostHistogramMap In SLSWebApp.java
|
| | Class org.apache.hadoop.yarn.sls.web.SLSWebApp defines non-transient
non-serializable
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14517649#comment-14517649 ] Hadoop QA commented on YARN-3445: - (!) The patch artifact directory on has been removed! This is a fatal error for test-patch.sh. Aborting. Jenkins (node H3) information at https://builds.apache.org/job/PreCommit-YARN-Build/7525/ may provide some hints. Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445-v2.patch, YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3445) Cache runningApps in RMNode for getting running apps on given NodeId
[ https://issues.apache.org/jira/browse/YARN-3445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505333#comment-14505333 ] Vinod Kumar Vavilapalli commented on YARN-3445: --- Better than before, will comment once see an updated patch. Cache runningApps in RMNode for getting running apps on given NodeId Key: YARN-3445 URL: https://issues.apache.org/jira/browse/YARN-3445 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Affects Versions: 2.7.0 Reporter: Junping Du Assignee: Junping Du Attachments: YARN-3445.patch Per discussion in YARN-3334, we need filter out unnecessary collectors info from RM in heartbeat response. Our propose is to add cache for runningApps in RMNode, so RM only send collectors for local running apps back. This is also needed in YARN-914 (graceful decommission) that if no running apps in NM which is in decommissioning stage, it will get decommissioned immediately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
