[jira] [Commented] (YARN-2141) [Umbrella] Capture container and node resource consumption
[ https://issues.apache.org/jira/browse/YARN-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365554#comment-14365554 ] Carlo Curino commented on YARN-2141: Karthik, it would be very useful to grab also disk and network consumption information. I believe the mechanisms are very similar, it should be easy. > [Umbrella] Capture container and node resource consumption > -- > > Key: YARN-2141 > URL: https://issues.apache.org/jira/browse/YARN-2141 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Carlo Curino >Priority: Minor > > Collecting per-container and per-node resource consumption statistics in a > fairly granular manner, and making them available to both infrastructure code > (e.g., schedulers) and users (e.g., AMs or directly users via webapps), can > facilitate several performance work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2141) [Umbrella] Capture container and node resource consumption
[ https://issues.apache.org/jira/browse/YARN-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14355590#comment-14355590 ] Vinod Kumar Vavilapalli commented on YARN-2141: --- bq. One other related effort is YARN-2928 which is also planning to obtain and send information about container resource-usage to a per-application aggregator. We should try to unify these.. Filed YARN-3332. > [Umbrella] Capture container and node resource consumption > -- > > Key: YARN-2141 > URL: https://issues.apache.org/jira/browse/YARN-2141 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Carlo Curino >Priority: Minor > > Collecting per-container and per-node resource consumption statistics in a > fairly granular manner, and making them available to both infrastructure code > (e.g., schedulers) and users (e.g., AMs or directly users via webapps), can > facilitate several performance work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2141) [Umbrella] Capture container and node resource consumption
[ https://issues.apache.org/jira/browse/YARN-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347932#comment-14347932 ] Karthik Kambatla commented on YARN-2141: [~curino] - the two sub-tasks capture memory and CPU in NM metrics. Did you have anything else in mind to be done here? Otherwise, we could mark this as resolved. > [Umbrella] Capture container and node resource consumption > -- > > Key: YARN-2141 > URL: https://issues.apache.org/jira/browse/YARN-2141 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Carlo Curino >Priority: Minor > > Collecting per-container and per-node resource consumption statistics in a > fairly granular manner, and making them available to both infrastructure code > (e.g., schedulers) and users (e.g., AMs or directly users via webapps), can > facilitate several performance work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2141) [Umbrella] Capture container and node resource consumption
[ https://issues.apache.org/jira/browse/YARN-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14271515#comment-14271515 ] Karthik Kambatla commented on YARN-2141: I would like to look at YARN-2141 as a pre-requisite of YARN-1012. Once we capture the actual resource consumption, we can use it in multiple ways - (1) Send to the RM, (2) Log it for debugging etc. > [Umbrella] Capture container and node resource consumption > -- > > Key: YARN-2141 > URL: https://issues.apache.org/jira/browse/YARN-2141 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Carlo Curino >Priority: Minor > > Collecting per-container and per-node resource consumption statistics in a > fairly granular manner, and making them available to both infrastructure code > (e.g., schedulers) and users (e.g., AMs or directly users via webapps), can > facilitate several performance work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2141) [Umbrella] Capture container and node resource consumption
[ https://issues.apache.org/jira/browse/YARN-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270535#comment-14270535 ] Vinod Kumar Vavilapalli commented on YARN-2141: --- One other related effort is YARN-2928 which is also planning to obtain and send information about container resource-usage to a per-application aggregator. We should try to unify these.. > [Umbrella] Capture container and node resource consumption > -- > > Key: YARN-2141 > URL: https://issues.apache.org/jira/browse/YARN-2141 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Carlo Curino >Priority: Minor > > Collecting per-container and per-node resource consumption statistics in a > fairly granular manner, and making them available to both infrastructure code > (e.g., schedulers) and users (e.g., AMs or directly users via webapps), can > facilitate several performance work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2141) [Umbrella] Capture container and node resource consumption
[ https://issues.apache.org/jira/browse/YARN-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14269919#comment-14269919 ] Vinod Kumar Vavilapalli commented on YARN-2141: --- Related to and very likely a dup of YARN-1012 which is part of a larger effort. > [Umbrella] Capture container and node resource consumption > -- > > Key: YARN-2141 > URL: https://issues.apache.org/jira/browse/YARN-2141 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Carlo Curino >Priority: Minor > > Collecting per-container and per-node resource consumption statistics in a > fairly granular manner, and making them available to both infrastructure code > (e.g., schedulers) and users (e.g., AMs or directly users via webapps), can > facilitate several performance work. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
