Sangjin Lee commented on YARN-3815:

We don't have to make it at container level I think but also not necessary for 
AM to retain and aggregate these values. AM could help to forward the values to 
per app timeline collector but don't have to aggregate them. Vinod got more 
ideas on this in offline discussion. [~vinodkv], can you comment on this?

Interesting. Could you or [~vinodkv] shed light on the idea? It would still 
need to be captured in an entity or entities, right? I would think sending it 
as part of the container entities would be simpler and more consistent (in that 
the per-app collector can simply look at all container metrics as subject to 
aggregation). I'd love to hear more about this.

I think "per-container averages" is not equal to per-container resource usage. 
Understanding application's real resource consumption/usage is one of the core 
use cases for new timeline service at the beginning so I don't think we should 
rule out anything important here.

How is the per-container resource usage different than the per-container 
average described in the summary? Could you kindly provide its definition?

No doubt understanding applications' real resource consumption/usage is 
critical. Between the individual container resource usage (which are all 
captured), the aggregated resource usage at the app/flow level (which the basic 
real time aggregation addresses), and the running averages/max of the 
aggregated resource usage at the app/flow level, I think it definitely covers 
that need. What would be the gap that's not addressed by the above data?

> [Aggregation] Application/Flow/User/Queue Level Aggregations
> ------------------------------------------------------------
>                 Key: YARN-3815
>                 URL: https://issues.apache.org/jira/browse/YARN-3815
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Junping Du
>            Assignee: Junping Du
>            Priority: Critical
>         Attachments: Timeline Service Nextgen Flow, User, Queue Level 
> Aggregations (v1).pdf, aggregation-design-discussion.pdf, 
> hbase-schema-proposal-for-aggregation.pdf
> Per previous discussions in some design documents for YARN-2928, the basic 
> scenario is the query for stats can happen on:
> - Application level, expect return: an application with aggregated stats
> - Flow level, expect return: aggregated stats for a flow_run, flow_version 
> and flow 
> - User level, expect return: aggregated stats for applications submitted by 
> user
> - Queue level, expect return: aggregated stats for applications within the 
> Queue
> Application states is the basic building block for all other level 
> aggregations. We can provide Flow/User/Queue level aggregated statistics info 
> based on application states (a dedicated table for application states is 
> needed which is missing from previous design documents like HBase/Phoenix 
> schema design). 

This message was sent by Atlassian JIRA

Reply via email to