Varun Saxena commented on YARN-3816:

In TimelineMetric#accumulateTo, can latestMetric be TIME_SERIES ? If not(seems 
to be the case as per current code), is the else part of the condition if 
(latestMetric.getType().equals(Type.SINGLE_VALUE)) { required ? Wont be 
handling TIME_SERIES then ?
I am not sure if I understand your comments correctly. But it definitely 
support TIME_SERIES type for latestMetrics and handle two types separately.
Actually I should have worded my query differently. accumulateTo by itself can 
handle TIME_SERIES. This is more from the context of caller. I am not sure if 
Li's patch is calling it but in TimelineCollector#aggregateMetrics, we have 
code like below. Here, I see latestTimelineMetrics.retrieveSingleDataValue() 
being called, which will throw an exception if metric type is not SINGLE_VALUE. 
Objective of throwing exception here ? As we have to get a single value for 
delta calculations, for TIME_SERIES maybe we can take value the latest 
timestamp value. 
I was getting confused by this code(calling method which throws exception for 
time series). So was wondering if we wont be handling time series.
213                 TimelineMetric latestTimelineMetrics = 
215                 Number delta = null;
216                 // new added metric for specific entityId
217                 if (latestTimelineMetrics == null) {
218                   delta = metric.retrieveSingleDataValue();
219                 } else {
220                   delta = TimelineMetricCalculator.sub(
221                       metric.retrieveSingleDataValue(),
222                       latestTimelineMetrics.retrieveSingleDataValue());
223                 }
250                   TimelineMetric newAggregatedArea = metric.accumulateTo(
251                       oldAggregatedArea, latestTimelineMetrics, 
252                       TimelineMetric.Operation.SUM);

> [Aggregation] App-level aggregation and accumulation for YARN system metrics
> ----------------------------------------------------------------------------
>                 Key: YARN-3816
>                 URL: https://issues.apache.org/jira/browse/YARN-3816
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Junping Du
>            Assignee: Junping Du
>         Attachments: Application Level Aggregation of Timeline Data.pdf, 
> YARN-3816-YARN-2928-v1.patch, YARN-3816-YARN-2928-v2.1.patch, 
> YARN-3816-YARN-2928-v2.2.patch, YARN-3816-YARN-2928-v2.3.patch, 
> YARN-3816-YARN-2928-v2.patch, YARN-3816-YARN-2928-v3.patch, 
> YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch
> We need application level aggregation of Timeline data:
> - To present end user aggregated states for each application, include: 
> resource (CPU, Memory) consumption across all containers, number of 
> containers launched/completed/failed, etc. We need this for apps while they 
> are running as well as when they are done.
> - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be 
> aggregated to show details of states in framework level.
> - Other level (Flow/User/Queue) aggregation can be more efficient to be based 
> on Application-level aggregations rather than raw entity-level data as much 
> less raws need to scan (with filter out non-aggregated entities, like: 
> events, configurations, etc.).

This message was sent by Atlassian JIRA

Reply via email to