[
https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15951228#comment-15951228
]
Varun Saxena edited comment on YARN-6375 at 3/31/17 4:37 PM:
-
Thanks [~vrushalic] for the comments.
I was primarily going by the metrics we currently report. And what can we
support at this point of time
The 2 examples you pointed out is basically the difference between how we
should treat a gauge vs a counter.
As you said for CPU and Memory, it probably makes sense in the way I have done
in the patch. This is what I had primarily in mind while raising JIRA.
But for counters the current mechanism might be better.
Counters such as HDFS bytes read can be aggregated at the application level by
AM itself as they are currently done by Mapreduce AM. However, I do see merit
though in aggregating them on our own on the collector side.
If we do say that we will sum up metrics across the lifetime of an app then we
would need to handle that on collector restart as well. Do we store all the
metrics in the state store which are read back upon restart? Or probably we
would need some kind of edit log mechanism.
Probably we need some way of identifying the metric type (gauge vs counter).
And do not clear counters but clear gauges upon aggregation. Maybe indicate it
in TimelineMetric object and rely on client reporting it in a consistent
fashion?
Or adopt the way we aggregate metrics for a flow run?
This probably needs to be discussed further.
bq. for CPU, perhaps we might want to know across this application what was the
cpu used by all the containers in the lifetime of this app?
This would be better dealt with time weighted accumulation which is in future
scope. Just summing up metrics of all entities running across different
timelines may not be very useful for metrics such as CPU / Memory. For
instance,
* App1 has 3 containers running one after the other for 10 second each i.e. app
runs for 30 seconds. And these containers while they were running, report CPU
of 20% each time.
* App2 has 3 containers but they run parallely for 30 seconds. And similar to
above these containers report CPU of 20% each time.
Now, aggregation as it is done at 15 second period would lead to a value of
120% by the end in both the cases. But both the apps and their resource
requirements are very distinct.
was (Author: varun_saxena):
Thanks [~vrushalic] for the comments.
I was primarily going by the metrics we currently report. And what can we
support at this point of time
The 2 examples you pointed out is basically the difference between how we
should treat a gauge vs a counter.
As you said for CPU and Memory, it probably makes sense in the way I have done
in the patch. This is what I had primarily in mind while raising JIRA.
But for counters the current mechanism might be better.
Counters such as HDFS bytes read can be aggregated at the application level by
AM itself as they are currently done by Mapreduce AM. However, I do see merit
though in aggregating them on our own on the collector side.
If we do say that we will sum up metrics across the lifetime of an app then we
would need to handle that on collector restart as well. Do we store all the
metrics in the state store which are read back upon restart? Or probably we
would need some kind of edit log mechanism.
Probably we need some way of identifying the metric type (gauge vs counter).
And do not clear counters but clear gauges upon aggregation. Maybe indicate it
in TimelineMetric object and rely on client reporting it in a consistent
fashion?
Or adopt the way we aggregate metrics for a flow run?
This probably needs to be discussed further.
bq. for CPU, perhaps we might want to know across this application what was the
cpu used by all the containers in the lifetime of this app?
This would be better dealt with time weighted accumulation which is in future
scope. Just summing up metrics of all entities running across different
timelines may not be very useful for metrics such as CPU / Memory. For
instance,
* App1 has 3 containers running one after the other for 10 second each i.e. app
runs for 30 seconds. And these containers while they were running, report CPU
of 20% each time.
* App2 has 3 containers but they run parallely for 30 seconds. And similar to
above these containers report CPU of 20% each time.
Now, aggregation as it is done at 15 second period would lead to a value of
120% in both the cases. But both the apps and their resource requirements are
very distinct.
> App level aggregation should not consider metric values reported in the
> previous aggregation cycle
> --
>
> Key: YARN-6375
> URL: https://issues.apache.org/jira/browse/YARN-6375
> Project: Hadoop YARN
>