[
https://issues.apache.org/jira/browse/AMBARI-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993712#comment-13993712
]
Dmytro Sen commented on AMBARI-5607:
------------------------------------
For simplicity purposes, I'll describe the issue on a single metric
yarn.NodeManagerMetrics.AllocatedContainers
According to the Ganglia configuration (gmetad.conf)
{noformat}
RRAs "RRA:AVERAGE:0.5:1:244" "RRA:AVERAGE:0.5:24:244" "RRA:AVERAGE:0.5:168:244"
"RRA:AVERAGE:0.5:672:244" "RRA:AVERAGE:0.5:5760:374"
{noformat}
5 separate RRD circular buffers stored in the
yarn.NodeManagerMetrics.AllocatedContainers.rrd database
1. Keep 1*15*244 seconds (1 hour) of metrics at 1*15 second resolution
2. Keep 24*15*244 seconds (24 hours) of metrics at 24*15 second resolution
3. Keep 168*15*244 seconds of metrics at 168*15 second resolution
4. Keep 672*15*244 seconds of metrics at 672*15 second resolution
5. Keep 5760*15*374 seconds (1 year) of metrics at 5760*15 second resolution
When a user calls
{noformat}
http://localhost:8080/api/v1/clusters/c1/services/YARN/components/NODEMANAGER?fields=host_components/metrics/yarn
{noformat}
Ambari returns the value from the RRD circular buffer #2. So it can take up to
6 minutes to have the API response updated.
> Yarn Nodemanager Metrics only update every few minutes
> ------------------------------------------------------
>
> Key: AMBARI-5607
> URL: https://issues.apache.org/jira/browse/AMBARI-5607
> Project: Ambari
> Issue Type: Bug
> Affects Versions: 1.5.1
> Environment: sles11sp1
> Reporter: Michael Harp
>
> Yarn Nodemanager Metrics take far too long between updates.
> To demonstrate:
> Run Terasort or anything that runs mapreduce:
> {code}
> hdfs dfs -mkdir -p benchmarks/terasort
> hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar teragen
> -Dmapred.map.tasks=72 -Dmapred.reduce.tasks=36 1000000
> benchmarks/terasort/input
> hadoop jar /usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples.jar terasort
> -Dmapred.map.tasks=72 -Dmapred.reduce.tasks=36 benchmarks/terasort/input
> benchmarks/terasort/output
> hdfs dfs -rm -R -skipTrash benchmarks/terasort
> {code}
> Then repeatedly probe the API at:
> {noformat}https://<server>:8081/api/v1/clusters/c1/services/YARN/components/NODEMANAGER?fields=host_components/metrics/yarn{noformat}
> It usually takes 2-3 minutes to see the metrics update, very repeatable.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
