Eric Payne updated YARN-415:

    Attachment: YARN-415.201408212033.txt

[~kkambatl], Thanks for your help in reviewing this.

A more minor comment: Can we rename ResourceUtilization to 
AggregateAppResourceUsage for clarity?
Can we update the title and the corresponding class/field names accordingly? 
Also, the values are accumulated for the duration of the app. Can we add 
aggregate in the required class/field names? 

With this patch, I changed the names of the {{ResourceUtilization}} class to 
{{AggregateAppResourceUsage}}, and modified appropriate fields, variable names, 
and method names.

[~jianhe], I appreciate the time you have put in on helping me review this 
patch and your comments.
we may need a test to test RMAppAttempt metrics is updated when container is 
finished also ? i.e. to test the code change in RMContainerFinished transition.
Doesn't this transition get tested when the container metrics are summed and 
compared to the output of RMAppImpl#getRMAppMetrics? For example, as in 
TestContainerResourceUsage#testUsageWithMultipleContainersAndRMRestart, and in 
some of the other tests in TestContainerResourceUsage, each container's 
resources are accessed and summed. Then, when RMAppImpl#getRMAppMetrics is 
called, it takes the total resource allocation for each attempt, and sums 
those. Then the 2 are compared. Perhaps there needs to be an additional Assert 
to make sure these are never 0, but other than that, I think these 2 
calculations are arrived at independently. Is that testing what you suggested?

> Capture aggregate memory allocation at the app-level for chargeback
> -------------------------------------------------------------------
>                 Key: YARN-415
>                 URL: https://issues.apache.org/jira/browse/YARN-415
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: resourcemanager
>    Affects Versions: 2.5.0
>            Reporter: Kendall Thrapp
>            Assignee: Andrey Klochkov
>         Attachments: YARN-415--n10.patch, YARN-415--n2.patch, 
> YARN-415--n3.patch, YARN-415--n4.patch, YARN-415--n5.patch, 
> YARN-415--n6.patch, YARN-415--n7.patch, YARN-415--n8.patch, 
> YARN-415--n9.patch, YARN-415.201405311749.txt, YARN-415.201406031616.txt, 
> YARN-415.201406262136.txt, YARN-415.201407042037.txt, 
> YARN-415.201407071542.txt, YARN-415.201407171553.txt, 
> YARN-415.201407172144.txt, YARN-415.201407232237.txt, 
> YARN-415.201407242148.txt, YARN-415.201407281816.txt, 
> YARN-415.201408062232.txt, YARN-415.201408080204.txt, 
> YARN-415.201408092006.txt, YARN-415.201408132109.txt, 
> YARN-415.201408150030.txt, YARN-415.201408181938.txt, 
> YARN-415.201408181938.txt, YARN-415.201408212033.txt, YARN-415.patch
> For the purpose of chargeback, I'd like to be able to compute the cost of an
> application in terms of cluster resource usage.  To start out, I'd like to 
> get the memory utilization of an application.  The unit should be MB-seconds 
> or something similar and, from a chargeback perspective, the memory amount 
> should be the memory reserved for the application, as even if the app didn't 
> use all that memory, no one else was able to use it.
> (reserved ram for container 1 * lifetime of container 1) + (reserved ram for
> container 2 * lifetime of container 2) + ... + (reserved ram for container n 
> * lifetime of container n)
> It'd be nice to have this at the app level instead of the job level because:
> 1. We'd still be able to get memory usage for jobs that crashed (and wouldn't 
> appear on the job history server).
> 2. We'd be able to get memory usage for future non-MR jobs (e.g. Storm).
> This new metric should be available both through the RM UI and RM Web 
> Services REST API.

This message was sent by Atlassian JIRA

Reply via email to