[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[
https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559496#comment-14559496
]
Hadoop QA commented on YARN-1012:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 53s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 35s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 35s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 5s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 48s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 22s | Tests passed in
hadoop-yarn-api. |
| {color:green}+1{color} | yarn tests | 1m 58s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 17s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 49m 10s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735357/YARN-1012-7.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 022f49d |
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8088/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8088/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8088/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8088/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8088/console |
This message was automatically generated.
NM should report resource utilization of running containers to RM in heartbeat
--
Key: YARN-1012
URL: https://issues.apache.org/jira/browse/YARN-1012
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.7.0
Reporter: Arun C Murthy
Assignee: Inigo Goiri
Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch,
YARN-1012-4.patch, YARN-1012-5.patch, YARN-1012-6.patch, YARN-1012-7.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559351#comment-14559351 ] Inigo Goiri commented on YARN-1012: --- I checked the issue with testContainerStatusPBImpl and I cannot figure out what's wrong there. Am I missing any method in ResourceUtilization? I also updated the interfaces and made it Unstable and Private (which I think matches our scope). Regarding the unit test, how would you check? Would you check to context.getContainers()? This related to your original quesiton of where should we store this information (ContainerMetrics or ContainerStatus). NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559770#comment-14559770 ] Vinod Kumar Vavilapalli commented on YARN-1012: --- Quick comments on the patch - ContainerStatus is a user-facing record, so you are also sending this to the AMs now. Was that intentional? I was assuming we are decoupling information sent to apps completely from what gets reported from NM to RM. - NodeManagers can send the raw information instead of dealing with vcores? Also, send both pmem and vmem usage? NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch, YARN-1012-6.patch, YARN-1012-7.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559787#comment-14559787 ] Inigo Goiri commented on YARN-1012: --- *I didn't realize about the ContainerStatus being sent to the AM. How would you send the information to the RM then? *I can send pmem and vmem with no problems; I'll add it right away. Regarding the CPU utilization, [~kasha] suggested to send millivcores as an int; we would have to agree on what to send there, a float? an int? NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch, YARN-1012-6.patch, YARN-1012-7.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559825#comment-14559825 ] Karthik Kambatla commented on YARN-1012: For CPU utilization, the other alternative is to send the number of physical cores on the machine along with percentage usage (like the one top shows). [~vinodkv] - does the need to avoid vcores stem from our intention of moving all the utilization information collection to a separate service that might not be specific to Yarn? NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch, YARN-1012-6.patch, YARN-1012-7.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14560105#comment-14560105 ] Karthik Kambatla commented on YARN-1012: I am not sure we want to send per-container utilization. We want to send (1) aggregate usage across all Yarn containers, and the (2) node utilization. NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch, YARN-1012-6.patch, YARN-1012-7.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14560085#comment-14560085 ] Vinod Kumar Vavilapalli commented on YARN-1012: --- bq. I didn't realize about the ContainerStatus being sent to the AM. How would you send the information to the RM then? We will have to create a new record somewhere in the hierarchy of NodeHeartbeatRequest. bq. does the need to avoid vcores stem from our intention of moving all the utilization information collection to a separate service that might not be specific to Yarn? Only partly. The bigger reason is vcore is more of a construct created for scheduling. We can have a vcore-millis which represents more of the resources reserved as opposed to real utilization. Real utilization can be cpu-wall-time, avg cpu utilization etc. bq. For CPU utilization, the other alternative is to send the number of physical cores on the machine along with percentage usage (like the one top shows). This reminds me of another thing that I missed mentioning before. We need to send per-container utilization as well as the the host-level utilization. May be we should first agree here on what exactly we are sending across. The patch should be relatively straigh-forward after that. NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch, YARN-1012-6.patch, YARN-1012-7.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14560102#comment-14560102 ] Karthik Kambatla commented on YARN-1012: bq. The bigger reason is vcore is more of a construct created for scheduling. We can have a vcore-millis which represents more of the resources reserved as opposed to real utilization. Real utilization can be cpu-wall-time, avg cpu utilization etc. I see your point. That said, NM is sending information across to RM solely because the scheduler can take it into consideration. It would definitely be simpler if the scheduler doesn't have to convert real utilization to vcores. NodeManagers are better places to do that. I would like for the heartbeat itself to have the utilization information in rounded-up vcores on milli-vcores. Should we store both? NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch, YARN-1012-6.patch, YARN-1012-7.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14560134#comment-14560134 ] Inigo Goiri commented on YARN-1012: --- For (1), the aggregated usage would be YARN-3481. (2) will depend on getting YARN-3534 first; after that, we'll need this one (or YARN-3481) to have the ResourceUtilization entity. NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch, YARN-1012-6.patch, YARN-1012-7.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[
https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559468#comment-14559468
]
Hadoop QA commented on YARN-1012:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 43s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 6s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 47s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 55s | Tests failed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 18s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 49m 6s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.api.TestPBImplRecords |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12735350/YARN-1012-6.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 022f49d |
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8087/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8087/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8087/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8087/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8087/console |
This message was automatically generated.
NM should report resource utilization of running containers to RM in heartbeat
--
Key: YARN-1012
URL: https://issues.apache.org/jira/browse/YARN-1012
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.7.0
Reporter: Arun C Murthy
Assignee: Inigo Goiri
Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch,
YARN-1012-4.patch, YARN-1012-5.patch, YARN-1012-6.patch, YARN-1012-7.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14559350#comment-14559350 ] Inigo Goiri commented on YARN-1012: --- I checked the issue with testContainerStatusPBImpl and I cannot figure out what's wrong there. Am I missing any method in ResourceUtilization? I also updated the interfaces and made it Unstable and Private (which I think matches our scope). Regarding the unit test, how would you check? Would you check to context.getContainers()? This related to your original quesiton of where should we store this information (ContainerMetrics or ContainerStatus). NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14557461#comment-14557461 ] Karthik Kambatla commented on YARN-1012: The test failure is related to the patch. [~elgoiri] - can you take a look? Other comments: # ResourceUtilization should be marked Unstable. Also, does it need to be Public? Accordingly, we need to update or get rid of annotations for the member methods. # It would be nice to add a unit test to ensure resource-utilization is actually tracked when container-monitoring is enabled. NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553688#comment-14553688 ] Inigo Goiri commented on YARN-1012: --- I don't know how I missed the missing files... I've been checking this for days. Fixed now. Agreed and fixed 1, 2, 3, and 4. I don't know what to do with 5... your call. NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[
https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553758#comment-14553758
]
Hadoop QA commented on YARN-1012:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 37s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 38s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 46s | The applied patch generated 5
new checkstyle issues (total was 1, now 6). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 48s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 29s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 55s | Tests failed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 7s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 48m 55s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.api.TestPBImplRecords |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734340/YARN-1012-4.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fb6b38d |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/diffcheckstylehadoop-yarn-api.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/whitespace.txt
|
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8038/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8038/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8038/console |
This message was automatically generated.
NM should report resource utilization of running containers to RM in heartbeat
--
Key: YARN-1012
URL: https://issues.apache.org/jira/browse/YARN-1012
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.7.0
Reporter: Arun C Murthy
Assignee: Inigo Goiri
Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch,
YARN-1012-4.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14554993#comment-14554993 ] Inigo Goiri commented on YARN-1012: --- I understand now what you're saying in 5. I actually tried to store the values in the metrics and get it from there but the interface for the metrics is not easy. Ideas? NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch, YARN-1012-4.patch, YARN-1012-5.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[
https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14555092#comment-14555092
]
Hadoop QA commented on YARN-1012:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 10s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 46s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 0s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 5s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 56s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | yarn tests | 0m 23s | Tests passed in
hadoop-yarn-api. |
| {color:red}-1{color} | yarn tests | 1m 55s | Tests failed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 17s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 50m 4s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.api.TestPBImplRecords |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12734640/YARN-1012-5.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 05e04f3 |
| hadoop-yarn-api test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8048/artifact/patchprocess/testrun_hadoop-yarn-api.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8048/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8048/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8048/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8048/console |
This message was automatically generated.
NM should report resource utilization of running containers to RM in heartbeat
--
Key: YARN-1012
URL: https://issues.apache.org/jira/browse/YARN-1012
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.7.0
Reporter: Arun C Murthy
Assignee: Inigo Goiri
Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch,
YARN-1012-4.patch, YARN-1012-5.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14553646#comment-14553646 ] Karthik Kambatla commented on YARN-1012: Looks like the patch is missing ResourceUtilizationPBImpl, and hence doesn't build. Could you please include those new files as well? Comments on the patch itself: # Given this is all new code, let us hold off on exposing it to end users just yet. Can we mark ContainerStatus#getUtilization Public-Unstable? # Is there a reason folks would want to turn off tracking utilization? If not, let us get rid of the config and always track it? # When logging at debug level, we want to check if debug logging is enabled to avoid string creation and concat. # I notice that we are using a float for virtual_cores. Do we anticipate using this value in any calculations? If yes, should we change this to be millivcores and int instead to avoid those floating point operations. Given this is just tracking utilization, I suspect we ll do any calculations. # In ContainerMonitorsImpl, we save utilization and then set container metrics. Should we leave this as is? Or, link them up so that the ContainerMonitorsImpl is aware of only one of them? NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager Affects Versions: 2.7.0 Reporter: Arun C Murthy Assignee: Inigo Goiri Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[
https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14550987#comment-14550987
]
Hadoop QA commented on YARN-1012:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 17s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 2m 23s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12733899/YARN-1012-3.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8860e35 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8004/console |
This message was automatically generated.
NM should report resource utilization of running containers to RM in heartbeat
--
Key: YARN-1012
URL: https://issues.apache.org/jira/browse/YARN-1012
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.7.0
Reporter: Arun C Murthy
Assignee: Inigo Goiri
Attachments: YARN-1012-1.patch, YARN-1012-2.patch, YARN-1012-3.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[
https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531846#comment-14531846
]
Hadoop QA commented on YARN-1012:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 2m 22s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731026/YARN-1012-2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 4c7b9b6 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7746/console |
This message was automatically generated.
NM should report resource utilization of running containers to RM in heartbeat
--
Key: YARN-1012
URL: https://issues.apache.org/jira/browse/YARN-1012
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.7.0
Reporter: Arun C Murthy
Assignee: Inigo Goiri
Attachments: YARN-1012-1.patch, YARN-1012-2.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[
https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14531795#comment-14531795
]
Hadoop QA commented on YARN-1012:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 42s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:red}-1{color} | javac | 2m 22s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731013/YARN-1012-1.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 31b627b |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/7745/console |
This message was automatically generated.
NM should report resource utilization of running containers to RM in heartbeat
--
Key: YARN-1012
URL: https://issues.apache.org/jira/browse/YARN-1012
Project: Hadoop YARN
Issue Type: Sub-task
Components: nodemanager
Affects Versions: 2.7.0
Reporter: Arun C Murthy
Assignee: Inigo Goiri
Attachments: YARN-1012-1.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1012) NM should report resource utilization of running containers to RM in heartbeat
[ https://issues.apache.org/jira/browse/YARN-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522818#comment-14522818 ] Inigo Goiri commented on YARN-1012: --- Thanks for adjusting the JIRAs [~vinodkv]. My proposal is to get the data from the ContainerMonitorImpl, aggregate it and put it in a new entity called ResourceUtilization which will contain for now CPU as a float and memory. Then, this will go into in the NodeHealthStauts. The other option is to add the utilization per container using the ResourceUtilization again but adding it per each container into the ContainerStatus. Both of them would require the new ResourceUtilization which in this case for simplicity will just be it without the whole ResourceUtilizations layer. Which one you guys prefer? NM should report resource utilization of running containers to RM in heartbeat -- Key: YARN-1012 URL: https://issues.apache.org/jira/browse/YARN-1012 Project: Hadoop YARN Issue Type: Sub-task Reporter: Arun C Murthy Assignee: Inigo Goiri -- This message was sent by Atlassian JIRA (v6.3.4#6332)
