Sangjin Lee commented on YARN-3815:

The use case here should be obviously. A quick real life example here is Google 
Borg - cluster management tools 
 which aggregate per-task resource usage information for usage-based charging, 
debugging job and long-term capacity planning.

Thanks [~djp]. What I'm looking for is a little more specific examples. That's 
why we spent some time during the discussion to define precisely what we mean 
by "averages". We discovered that there were already two different definitions 
of the average for gauges. We also ruled out per-container averages (explained 
in the summary), so per-task resource usage is not an example we're looking for.

So as for the moving (but aggregate) average, are there other examples? What we 
discussed during the meeting (also in the summary) was the total CPU 
utilization of an app/flow. Other examples, and how they might be useful, or is 
that pretty much the best example?

> [Aggregation] Application/Flow/User/Queue Level Aggregations
> ------------------------------------------------------------
>                 Key: YARN-3815
>                 URL: https://issues.apache.org/jira/browse/YARN-3815
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Junping Du
>            Assignee: Junping Du
>            Priority: Critical
>         Attachments: Timeline Service Nextgen Flow, User, Queue Level 
> Aggregations (v1).pdf, aggregation-design-discussion.pdf, 
> hbase-schema-proposal-for-aggregation.pdf
> Per previous discussions in some design documents for YARN-2928, the basic 
> scenario is the query for stats can happen on:
> - Application level, expect return: an application with aggregated stats
> - Flow level, expect return: aggregated stats for a flow_run, flow_version 
> and flow 
> - User level, expect return: aggregated stats for applications submitted by 
> user
> - Queue level, expect return: aggregated stats for applications within the 
> Queue
> Application states is the basic building block for all other level 
> aggregations. We can provide Flow/User/Queue level aggregated statistics info 
> based on application states (a dedicated table for application states is 
> needed which is missing from previous design documents like HBase/Phoenix 
> schema design). 

This message was sent by Atlassian JIRA

Reply via email to