[jira] [Commented] (YARN-3881) Writing RM cluster-level metrics
[
https://issues.apache.org/jira/browse/YARN-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16919272#comment-16919272
]
Rohith Sharma K S commented on YARN-3881:
-
[~prabham] feel free to assign your self as there is not response for longer
period
> Writing RM cluster-level metrics
>
>
> Key: YARN-3881
> URL: https://issues.apache.org/jira/browse/YARN-3881
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Major
> Labels: YARN-5355
> Attachments: metrics.json
>
>
> RM has a bunch of metrics that we may want to write into the timeline backend
> to. I attached the metrics.json that I've crawled via
> {{http://localhost:8088/jmx?qry=Hadoop:*}}. IMHO, we need to pay attention to
> three groups of metrics:
> 1. QueueMetrics
> 2. JvmMetrics
> 3. ClusterMetrics
> The problem is that unlike other metrics belongs to a single application,
> these ones belongs to RM or cluster-wide. Therefore, current write path is
> not going to work for these metrics because they don't have the associated
> user/flow/app context info. We need to rethink of modeling cross-app metrics
> and the api to handle them.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3881) Writing RM cluster-level metrics
[
https://issues.apache.org/jira/browse/YARN-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16786861#comment-16786861
]
Prabha Manepalli commented on YARN-3881:
[~BINGXUE QIU] If you are not currently working on this, Can I take this up
now. Thanks.
> Writing RM cluster-level metrics
>
>
> Key: YARN-3881
> URL: https://issues.apache.org/jira/browse/YARN-3881
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>Priority: Major
> Labels: YARN-5355
> Attachments: metrics.json
>
>
> RM has a bunch of metrics that we may want to write into the timeline backend
> to. I attached the metrics.json that I've crawled via
> {{http://localhost:8088/jmx?qry=Hadoop:*}}. IMHO, we need to pay attention to
> three groups of metrics:
> 1. QueueMetrics
> 2. JvmMetrics
> 3. ClusterMetrics
> The problem is that unlike other metrics belongs to a single application,
> these ones belongs to RM or cluster-wide. Therefore, current write path is
> not going to work for these metrics because they don't have the associated
> user/flow/app context info. We need to rethink of modeling cross-app metrics
> and the api to handle them.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3881) Writing RM cluster-level metrics
[
https://issues.apache.org/jira/browse/YARN-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15676313#comment-15676313
]
Bingxue Qiu commented on YARN-3881:
---
Hi [~zjshen], I haven't find the totalVirtualCores / totalMB of cluster
metrics in the metrics.json, maybe it's necessary to show the water line
trends when the nodes changes, just like add nodes or nodes fail?
> Writing RM cluster-level metrics
>
>
> Key: YARN-3881
> URL: https://issues.apache.org/jira/browse/YARN-3881
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
> Labels: YARN-5355
> Attachments: metrics.json
>
>
> RM has a bunch of metrics that we may want to write into the timeline backend
> to. I attached the metrics.json that I've crawled via
> {{http://localhost:8088/jmx?qry=Hadoop:*}}. IMHO, we need to pay attention to
> three groups of metrics:
> 1. QueueMetrics
> 2. JvmMetrics
> 3. ClusterMetrics
> The problem is that unlike other metrics belongs to a single application,
> these ones belongs to RM or cluster-wide. Therefore, current write path is
> not going to work for these metrics because they don't have the associated
> user/flow/app context info. We need to rethink of modeling cross-app metrics
> and the api to handle them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3881) Writing RM cluster-level metrics
[
https://issues.apache.org/jira/browse/YARN-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612638#comment-14612638
]
Zhijie Shen commented on YARN-3881:
---
Once the metrics are ready, we can build YARN/timeline service builtin webUI to
show this information, as well as expose it via API, such that third party
monitoring like ambari can integrate with it. I think it should be quite
flexible.
Writing RM cluster-level metrics
Key: YARN-3881
URL: https://issues.apache.org/jira/browse/YARN-3881
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: metrics.json
RM has a bunch of metrics that we may want to write into the timeline backend
to. I attached the metrics.json that I've crawled via
{{http://localhost:8088/jmx?qry=Hadoop:*}}. IMHO, we need to pay attention to
three groups of metrics:
1. QueueMetrics
2. JvmMetrics
3. ClusterMetrics
The problem is that unlike other metrics belongs to a single application,
these ones belongs to RM or cluster-wide. Therefore, current write path is
not going to work for these metrics because they don't have the associated
user/flow/app context info. We need to rethink of modeling cross-app metrics
and the api to handle them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3881) Writing RM cluster-level metrics
[
https://issues.apache.org/jira/browse/YARN-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612429#comment-14612429
]
Zhijie Shen commented on YARN-3881:
---
IMHO, we need to add an addition API to direct write the cross app metrics (or
already aggregated metrics, if you think of these ones are actually the
aggregated data of each individual app, such as the counters of
submitted/pending/running apps) to the backend, in the separate tables, such as
cluster/queue/user tables, and these data don't need to be aggregated any more.
Writing RM cluster-level metrics
Key: YARN-3881
URL: https://issues.apache.org/jira/browse/YARN-3881
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: metrics.json
RM has a bunch of metrics that we may want to write into the timeline backend
to. I attached the metrics.json that I've crawled via
{{http://localhost:8088/jmx?qry=Hadoop:*}}. IMHO, we need to pay attention to
three groups of metrics:
1. QueueMetrics
2. JvmMetrics
3. ClusterMetrics
The problem is that unlike other metrics belongs to a single application,
these ones belongs to RM or cluster-wide. Therefore, current write path is
not going to work for these metrics because they don't have the associated
user/flow/app context info. We need to rethink of modeling cross-app metrics
and the api to handle them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3881) Writing RM cluster-level metrics
[
https://issues.apache.org/jira/browse/YARN-3881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14612432#comment-14612432
]
Lei Guo commented on YARN-3881:
---
This is an interesting topic, assuming the timeline server provides this
support, should Ambari or other monitoring tool to use this for monitoring
purpose? If not, what's the scenario to write RM related metrics?
Writing RM cluster-level metrics
Key: YARN-3881
URL: https://issues.apache.org/jira/browse/YARN-3881
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Zhijie Shen
Assignee: Zhijie Shen
Attachments: metrics.json
RM has a bunch of metrics that we may want to write into the timeline backend
to. I attached the metrics.json that I've crawled via
{{http://localhost:8088/jmx?qry=Hadoop:*}}. IMHO, we need to pay attention to
three groups of metrics:
1. QueueMetrics
2. JvmMetrics
3. ClusterMetrics
The problem is that unlike other metrics belongs to a single application,
these ones belongs to RM or cluster-wide. Therefore, current write path is
not going to work for these metrics because they don't have the associated
user/flow/app context info. We need to rethink of modeling cross-app metrics
and the api to handle them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
