[
https://issues.apache.org/jira/browse/YARN-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15067330#comment-15067330
]
Sangjin Lee commented on YARN-3816:
-----------------------------------
Regarding the new entity type "YARN_APPLICATION_AGGREGATION", I think I have
raised this topic before and so have others, but at least I cannot find answers
in this JIRA. Is there a strong reason to introduce a separate entity type just
for this purpose, rather than reusing the existing YARN_APPLICATION type (and
the application table)? If so, could you elaborate on why?
This would create a complete separation of any normal metrics that may be
stored in the application table and these aggregated metrics handled in this
JIRA. It has a number of implications. First, if you query normally for
applications, the aggregated metrics would *not* be included in the reader
queries (I guess that's why a separate REST end point was introduced?).
Furthermore, the current app-to-flow-run aggregation looks only at the
application table, and the aggregated metrics in this manner would *not* be
rolled up to the flow run, flow, and so on unless we make an explicit change to
look at the entity table with that entity type. Making that change also sounds
like a very much a non-trivial change (cc [~vrushalic]).
> [Aggregation] App-level aggregation and accumulation for YARN system metrics
> ----------------------------------------------------------------------------
>
> Key: YARN-3816
> URL: https://issues.apache.org/jira/browse/YARN-3816
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Junping Du
> Assignee: Junping Du
> Labels: yarn-2928-1st-milestone
> Attachments: Application Level Aggregation of Timeline Data.pdf,
> YARN-3816-YARN-2928-v1.patch, YARN-3816-YARN-2928-v2.1.patch,
> YARN-3816-YARN-2928-v2.2.patch, YARN-3816-YARN-2928-v2.3.patch,
> YARN-3816-YARN-2928-v2.patch, YARN-3816-YARN-2928-v3.1.patch,
> YARN-3816-YARN-2928-v3.patch, YARN-3816-YARN-2928-v4.patch,
> YARN-3816-feature-YARN-2928.v4.1.patch, YARN-3816-poc-v1.patch,
> YARN-3816-poc-v2.patch
>
>
> We need application level aggregation of Timeline data:
> - To present end user aggregated states for each application, include:
> resource (CPU, Memory) consumption across all containers, number of
> containers launched/completed/failed, etc. We need this for apps while they
> are running as well as when they are done.
> - Also, framework specific metrics, e.g. HDFS_BYTES_READ, should be
> aggregated to show details of states in framework level.
> - Other level (Flow/User/Queue) aggregation can be more efficient to be based
> on Application-level aggregations rather than raw entity-level data as much
> less raws need to scan (with filter out non-aggregated entities, like:
> events, configurations, etc.).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
