[
https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14791775#comment-14791775
]
Varun Saxena commented on YARN-3816:
------------------------------------
[~djp],
{quote}
In TimelineMetric#accumulateTo, can latestMetric be TIME_SERIES ? If not(seems
to be the case as per current code), is the else part of the condition if
(latestMetric.getType().equals(Type.SINGLE_VALUE)) { required ? Wont be
handling TIME_SERIES then ?
I am not sure if I understand your comments correctly. But it definitely
support TIME_SERIES type for latestMetrics and handle two types separately.
{quote}
Actually I should have worded my query differently. accumulateTo by itself can
handle TIME_SERIES. This is more from the context of caller. I am not sure if
Li's patch is calling it but in TimelineCollector#aggregateMetrics, we have
code like below. Here, I see latestTimelineMetrics.retrieveSingleDataValue()
being called, which will throw an exception if metric type is not SINGLE_VALUE.
Objective of throwing exception here ? As we have to get a single value for
delta calculations, for TIME_SERIES maybe we can take value the latest
timestamp value.
I was getting confused by this code(calling method which throws exception for
time series). So was wondering if we wont be handling time series.
{code}
213 TimelineMetric latestTimelineMetrics =
entityIdMap.get(entityId);
214
215 Number delta = null;
216 // new added metric for specific entityId
217 if (latestTimelineMetrics == null) {
218 delta = metric.retrieveSingleDataValue();
219 } else {
220 delta = TimelineMetricCalculator.sub(
221 metric.retrieveSingleDataValue(),
222 latestTimelineMetrics.retrieveSingleDataValue());
223 }
...........................
250 TimelineMetric newAggregatedArea = metric.accumulateTo(
251 oldAggregatedArea, latestTimelineMetrics,
aggregatedTime,
252 TimelineMetric.Operation.SUM);
{code}
> [Aggregation] App-level aggregation and accumulation for YARN system metrics
> ----------------------------------------------------------------------------
>
> Key: YARN-3816
> URL: https://issues.apache.org/jira/browse/YARN-3816
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Junping Du
> Assignee: Junping Du
> Attachments: Application Level Aggregation of Timeline Data.pdf,
> YARN-3816-YARN-2928-v1.patch, YARN-3816-YARN-2928-v2.1.patch,
> YARN-3816-YARN-2928-v2.2.patch, YARN-3816-YARN-2928-v2.3.patch,
> YARN-3816-YARN-2928-v2.patch, YARN-3816-YARN-2928-v3.patch,
> YARN-3816-poc-v1.patch, YARN-3816-poc-v2.patch
>
>
> We need application level aggregation of Timeline data:
> - To present end user aggregated states for each application, include:
> resource (CPU, Memory) consumption across all containers, number of
> containers launched/completed/failed, etc. We need this for apps while they
> are running as well as when they are done.
> - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be
> aggregated to show details of states in framework level.
> - Other level (Flow/User/Queue) aggregation can be more efficient to be based
> on Application-level aggregations rather than raw entity-level data as much
> less raws need to scan (with filter out non-aggregated entities, like:
> events, configurations, etc.).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
