[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14606014#comment-14606014 ] Chris Douglas commented on YARN-3819: - bq. I think that if we decide to move this to Common, we should move the whole ResourceCalculator; otherwise, just finish this one here. I'm willing to start the JIRA in Common (or reuse if anybody knows about a JIRA already pushing for that) to have the whole ResourceCalculator there. +1 Let's just do this and move on. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch, > YARN-3819-4.patch, YARN-3819-5.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605953#comment-14605953 ] Lei Guo commented on YARN-3819: --- Moving the implementation into common, it may help to reduce the development effort and maintenance of code, but still with extra overhead in a production environment because different component need collect similar information. And as the polling could occur at different time for different component, they may behave differently, Ideally, there should be one centrally place/component to collect node metrics and provide related information to different services, something like the master LIM in Platform LSF. Maybe the timeline service could take this role in the future. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch, > YARN-3819-4.patch, YARN-3819-5.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605925#comment-14605925 ] Inigo Goiri commented on YARN-3819: --- [~aw], at some point, I talked to [~jakobhoman] about the possibility of moving the whole ResourceCalculator structure to Common and it seemed like a good idea. Given that and taking advantage of the momentum of this JIRA, I would push for that solution instead of a patch just for this solution. Another thing is the resource monitoring part of the ProcessTree and friends; I would leave that alone for now but ultimately, I think the natural place would be Common too. Summarizing, I think that if we decide to move this to Common, we should move the whole ResourceCalculator; otherwise, just finish this one here. I'm willing to start the JIRA in Common (or reuse if anybody knows about a JIRA already pushing for that) to have the whole ResourceCalculator there. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch, > YARN-3819-4.patch, YARN-3819-5.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605917#comment-14605917 ] Srikanth Kandula commented on YARN-3819: >From [~chris.douglas] "@Allen Wittenauer Is there a corresponding part of the datanode already monitoring these resources? I looked, but found only the metrics. This JIRA and YARN-3819 only extend the monitoring. As Karthik pointed out in YARN-2745, refactoring for more unified resource monitoring is in YARN-3332." > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch, > YARN-3819-4.patch, YARN-3819-5.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605879#comment-14605879 ] Allen Wittenauer commented on YARN-3819: Yes, I recognize that you are building on an already established framework that, prior, only collected metrics that were specific to YARN. But now with network and disk, the collection details are such that all of the sub-projects could benefit.It's shortsighted to build something that could very easily be used by all. That said, the data collection code should be done in a generic way such that, in the future, HDFS could plug into the same collection classes so that it too may make block scheduling decisions. (This has been a discussion point around the HDFS community for a while). YARN could then call those methods that gather the data into its own framework to do whatever it needs to do. So while the framework is obviously different the actual work, of e.g. "how do I know the IO stats on file system X", should be in common. It could be argued that the previous bits that are also being collected should be in common, but that's already shipped. Let's not repeat past mistakes though. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch, > YARN-3819-4.patch, YARN-3819-5.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605861#comment-14605861 ] Srikanth Kandula commented on YARN-3819: [~aw] Allen- Could you expand a bit? Specifically, where in common would you like us to incorporate this? This is a pretty small and somewhat straightforward change. We are "plugging within" what is already there in terms of the resource monitoring harness in the NM and want to hear what may be an equivalent place in common. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch, > YARN-3819-4.patch, YARN-3819-5.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605812#comment-14605812 ] Allen Wittenauer commented on YARN-3819: -1 This needs to go into common. I don't know how much more obvious it should be that HDFS will at some point to collect this data too for its usage. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch, > YARN-3819-4.patch, YARN-3819-5.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[
https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14592246#comment-14592246
]
Hadoop QA commented on YARN-3819:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 30s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 53s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 48s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 39s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 16m 34s | Tests passed in
hadoop-gridmix. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 32s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 69m 18s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740322/YARN-3819-5.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 1c13519 |
| hadoop-gridmix test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8284/artifact/patchprocess/testrun_hadoop-gridmix.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8284/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8284/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8284/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8284/console |
This message was automatically generated.
> Collect network usage on the node
> -
>
> Key: YARN-3819
> URL: https://issues.apache.org/jira/browse/YARN-3819
> Project: Hadoop YARN
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Robert Grandl
>Assignee: Robert Grandl
> Labels: yarn-common, yarn-util
> Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch,
> YARN-3819-4.patch, YARN-3819-5.patch
>
>
> In this JIRA we propose to collect the network usage on a node. This JIRA is
> part of a larger effort of monitoring resource usages on the nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[
https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591702#comment-14591702
]
Hadoop QA commented on YARN-3819:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 13s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 57s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 25s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 52s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 2s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 36s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 16m 27s | Tests passed in
hadoop-gridmix. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 19s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 68m 40s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740322/YARN-3819-5.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 295d678 |
| hadoop-gridmix test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8283/artifact/patchprocess/testrun_hadoop-gridmix.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8283/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8283/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8283/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8283/console |
This message was automatically generated.
> Collect network usage on the node
> -
>
> Key: YARN-3819
> URL: https://issues.apache.org/jira/browse/YARN-3819
> Project: Hadoop YARN
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Robert Grandl
>Assignee: Robert Grandl
> Labels: yarn-common, yarn-util
> Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch,
> YARN-3819-4.patch, YARN-3819-5.patch
>
>
> In this JIRA we propose to collect the network usage on a node. This JIRA is
> part of a larger effort of monitoring resource usages on the nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[
https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591342#comment-14591342
]
Hadoop QA commented on YARN-3819:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 54s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 38s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 46s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 35s | The applied patch generated 7
new checkstyle issues (total was 15, now 22). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 38s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | tools/hadoop tests | 16m 20s | Tests passed in
hadoop-gridmix. |
| {color:green}+1{color} | yarn tests | 1m 57s | Tests passed in
hadoop-yarn-common. |
| {color:green}+1{color} | yarn tests | 6m 5s | Tests passed in
hadoop-yarn-server-nodemanager. |
| | | 67m 37s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740281/YARN-3819-4.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 74351af |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8275/artifact/patchprocess/diffcheckstylehadoop-yarn-common.txt
|
| hadoop-gridmix test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8275/artifact/patchprocess/testrun_hadoop-gridmix.txt
|
| hadoop-yarn-common test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8275/artifact/patchprocess/testrun_hadoop-yarn-common.txt
|
| hadoop-yarn-server-nodemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8275/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8275/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8275/console |
This message was automatically generated.
> Collect network usage on the node
> -
>
> Key: YARN-3819
> URL: https://issues.apache.org/jira/browse/YARN-3819
> Project: Hadoop YARN
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Robert Grandl
>Assignee: Robert Grandl
> Labels: yarn-common, yarn-util
> Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch,
> YARN-3819-4.patch
>
>
> In this JIRA we propose to collect the network usage on a node. This JIRA is
> part of a larger effort of monitoring resource usages on the nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[
https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591075#comment-14591075
]
Hadoop QA commented on YARN-3819:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:red}-1{color} | javac | 3m 3s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740201/YARN-3819-3.patch |
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / cc43288 |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8274/console |
This message was automatically generated.
> Collect network usage on the node
> -
>
> Key: YARN-3819
> URL: https://issues.apache.org/jira/browse/YARN-3819
> Project: Hadoop YARN
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Robert Grandl
>Assignee: Robert Grandl
> Labels: yarn-common, yarn-util
> Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch
>
>
> In this JIRA we propose to collect the network usage on a node. This JIRA is
> part of a larger effort of monitoring resource usages on the nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590600#comment-14590600 ] Lei Guo commented on YARN-3819: --- Thanks for the explanation. My concern is mainly about we will need update the code when we need new resource information for scheduling purpose. If we have a generic framework, and the integration developer may can write a script to feed information into NM, and then RM can do scheduling based on that, this is part of my comment in 3332, https://issues.apache.org/jira/browse/YARN-3332?focusedCommentId=14355923&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14355923 > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590587#comment-14590587 ] Srikanth Kandula commented on YARN-3819: [~grey] The patch does have the generic component, in that it needs /proc/net... It would be possible to expose whatever additional fields end up being needed by schedulers or monitors. We only expose a first cut of them (total read/ written). > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590555#comment-14590555 ] Robert Grandl commented on YARN-3819: - Short description of this JIRA: We process "/proc/net/dev" file which reports for every network interface present on the node, the cumulative amount of bytes read/written. We aggregate these numbers across all the interfaces except loopback. We tested the existence of these files in the following Linux kernel versions: Linux 3.2.0 Linux 2.6.32 Linux 3.13.0 Also, doing further search on the web, it seems people are using/recommending these files for extracting read/written network bytes counters. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590553#comment-14590553 ] Robert Grandl commented on YARN-3819: - [~grey], YARN-2745 is an effort to schedule multiple resources. The resources taken in account are CPU/Memory/Disk/Network. For fungible resources such as disk and network, the counters required are the total number of bytes read/written from/to disk/network. This JIRA extends the ResourceCalculatorPlugin which is able to extract the amount of available CPU and Memory on a node. YARN-1012 is already using this information and YARN-1012 is aggregating this information in a heartbeat from NM to RM. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch, YARN-3819-3.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590472#comment-14590472 ] Lei Guo commented on YARN-3819: --- For multiple resource scheduling, we may have different resource types, not just CPU/disk/network. Even for network, we may need other attributes instead of just read and write. It's better to have some generic framework in RM/NM and collect data via plug-ins. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch, YARN-3819-2.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3819) Collect network usage on the node
[ https://issues.apache.org/jira/browse/YARN-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14590217#comment-14590217 ] Robert Grandl commented on YARN-3819: - [~srikanthkandula] and I were proposing to collect the network usage on a node. This is part of a larger effort of multi-resource scheduling. Previous efforts in collecting network usage per containers is not enough for the purpose of multi-resource scheduling, as it is not able to capture other traffic activities on the node such as ingestion or evacuation. > Collect network usage on the node > - > > Key: YARN-3819 > URL: https://issues.apache.org/jira/browse/YARN-3819 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.0 >Reporter: Robert Grandl >Assignee: Robert Grandl > Labels: yarn-common, yarn-util > Attachments: YARN-3819-1.patch > > > In this JIRA we propose to collect the network usage on a node. This JIRA is > part of a larger effort of monitoring resource usages on the nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
