Eric Payne commented on YARN-415:
Thank you very much for reviewing my patch.
I think I understand what you are suggesting. Please let me clarify:
1) Add memory utilization to RMAppMetrics/RMAppAttemptMetrics
Since every RMAppAttemptImpl object has a reference to an RMAppAttemptMetrics
object, you are suggesting that I move the resource usage stats to
RMAppAttemptMetrics. Also, when reporting on resource usage, use the reporting
methods from RMAppAttempt and RMApp.
2) Keep running container resource utilization in SchedulerApplicationAttempt
The way the patch for YARN-415 is currently, it keeps resource usage stats for
both running and finished containers in the SchedulerApplicationAttempt object.
You're suggestion is to keep resource usage stats only for running containers.
3) Move completed container resource calculation to
For completed containers, you are suggesting that the calculation be done for
final resource usage stats within the RMContainerImpl#FinishTransition method
and have that send the resource stats as a payload within the
RMAppAttemptContainerFinishedEvent event. Then, when RMAppAttemptImpl receives
the CONTAINER_FINISHED event, it would add the resource usage stats for the
finished containers to those already collected within the RMAppAttemptMetrics
Is that correct?
> Capture memory utilization at the app-level for chargeback
> Key: YARN-415
> URL: https://issues.apache.org/jira/browse/YARN-415
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
> Affects Versions: 0.23.6
> Reporter: Kendall Thrapp
> Assignee: Andrey Klochkov
> Attachments: YARN-415--n10.patch, YARN-415--n2.patch,
> YARN-415--n3.patch, YARN-415--n4.patch, YARN-415--n5.patch,
> YARN-415--n6.patch, YARN-415--n7.patch, YARN-415--n8.patch,
> YARN-415--n9.patch, YARN-415.201405311749.txt, YARN-415.201406031616.txt,
> YARN-415.201406262136.txt, YARN-415.201407042037.txt,
> YARN-415.201407071542.txt, YARN-415.patch
> For the purpose of chargeback, I'd like to be able to compute the cost of an
> application in terms of cluster resource usage. To start out, I'd like to
> get the memory utilization of an application. The unit should be MB-seconds
> or something similar and, from a chargeback perspective, the memory amount
> should be the memory reserved for the application, as even if the app didn't
> use all that memory, no one else was able to use it.
> (reserved ram for container 1 * lifetime of container 1) + (reserved ram for
> container 2 * lifetime of container 2) + ... + (reserved ram for container n
> * lifetime of container n)
> It'd be nice to have this at the app level instead of the job level because:
> 1. We'd still be able to get memory usage for jobs that crashed (and wouldn't
> appear on the job history server).
> 2. We'd be able to get memory usage for future non-MR jobs (e.g. Storm).
> This new metric should be available both through the RM UI and RM Web
> Services REST API.
This message was sent by Atlassian JIRA