[
https://issues.apache.org/jira/browse/YARN-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14612798#comment-14612798
]
Sangjin Lee commented on YARN-3815:
-----------------------------------
{quote}
We don't have to make it at container level I think but also not necessary for
AM to retain and aggregate these values. AM could help to forward the values to
per app timeline collector but don't have to aggregate them. Vinod got more
ideas on this in offline discussion. [~vinodkv], can you comment on this?
{quote}
Interesting. Could you or [~vinodkv] shed light on the idea? It would still
need to be captured in an entity or entities, right? I would think sending it
as part of the container entities would be simpler and more consistent (in that
the per-app collector can simply look at all container metrics as subject to
aggregation). I'd love to hear more about this.
{quote}
I think "per-container averages" is not equal to per-container resource usage.
Understanding application's real resource consumption/usage is one of the core
use cases for new timeline service at the beginning so I don't think we should
rule out anything important here.
{quote}
How is the per-container resource usage different than the per-container
average described in the summary? Could you kindly provide its definition?
No doubt understanding applications' real resource consumption/usage is
critical. Between the individual container resource usage (which are all
captured), the aggregated resource usage at the app/flow level (which the basic
real time aggregation addresses), and the running averages/max of the
aggregated resource usage at the app/flow level, I think it definitely covers
that need. What would be the gap that's not addressed by the above data?
> [Aggregation] Application/Flow/User/Queue Level Aggregations
> ------------------------------------------------------------
>
> Key: YARN-3815
> URL: https://issues.apache.org/jira/browse/YARN-3815
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Junping Du
> Assignee: Junping Du
> Priority: Critical
> Attachments: Timeline Service Nextgen Flow, User, Queue Level
> Aggregations (v1).pdf, aggregation-design-discussion.pdf,
> hbase-schema-proposal-for-aggregation.pdf
>
>
> Per previous discussions in some design documents for YARN-2928, the basic
> scenario is the query for stats can happen on:
> - Application level, expect return: an application with aggregated stats
> - Flow level, expect return: aggregated stats for a flow_run, flow_version
> and flow
> - User level, expect return: aggregated stats for applications submitted by
> user
> - Queue level, expect return: aggregated stats for applications within the
> Queue
> Application states is the basic building block for all other level
> aggregations. We can provide Flow/User/Queue level aggregated statistics info
> based on application states (a dedicated table for application states is
> needed which is missing from previous design documents like HBase/Phoenix
> schema design).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
