[
https://issues.apache.org/jira/browse/YARN-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14062261#comment-14062261
]
Eric Payne commented on YARN-415:
---------------------------------
Hi [~leftnoteasy].
Thank you very much for reviewing my patch.
I think I understand what you are suggesting. Please let me clarify:
{quote}
1) Add memory utilization to RMAppMetrics/RMAppAttemptMetrics
{quote}
Since every RMAppAttemptImpl object has a reference to an RMAppAttemptMetrics
object, you are suggesting that I move the resource usage stats to
RMAppAttemptMetrics. Also, when reporting on resource usage, use the reporting
methods from RMAppAttempt and RMApp.
{quote}
2) Keep running container resource utilization in SchedulerApplicationAttempt
{quote}
The way the patch for YARN-415 is currently, it keeps resource usage stats for
both running and finished containers in the SchedulerApplicationAttempt object.
You're suggestion is to keep resource usage stats only for running containers.
{quote}
3) Move completed container resource calculation to
RMContainerImpl#FinishTransition
{quote}
For completed containers, you are suggesting that the calculation be done for
final resource usage stats within the RMContainerImpl#FinishTransition method
and have that send the resource stats as a payload within the
RMAppAttemptContainerFinishedEvent event. Then, when RMAppAttemptImpl receives
the CONTAINER_FINISHED event, it would add the resource usage stats for the
finished containers to those already collected within the RMAppAttemptMetrics
object.
Is that correct?
> Capture memory utilization at the app-level for chargeback
> ----------------------------------------------------------
>
> Key: YARN-415
> URL: https://issues.apache.org/jira/browse/YARN-415
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: resourcemanager
> Affects Versions: 0.23.6
> Reporter: Kendall Thrapp
> Assignee: Andrey Klochkov
> Attachments: YARN-415--n10.patch, YARN-415--n2.patch,
> YARN-415--n3.patch, YARN-415--n4.patch, YARN-415--n5.patch,
> YARN-415--n6.patch, YARN-415--n7.patch, YARN-415--n8.patch,
> YARN-415--n9.patch, YARN-415.201405311749.txt, YARN-415.201406031616.txt,
> YARN-415.201406262136.txt, YARN-415.201407042037.txt,
> YARN-415.201407071542.txt, YARN-415.patch
>
>
> For the purpose of chargeback, I'd like to be able to compute the cost of an
> application in terms of cluster resource usage. To start out, I'd like to
> get the memory utilization of an application. The unit should be MB-seconds
> or something similar and, from a chargeback perspective, the memory amount
> should be the memory reserved for the application, as even if the app didn't
> use all that memory, no one else was able to use it.
> (reserved ram for container 1 * lifetime of container 1) + (reserved ram for
> container 2 * lifetime of container 2) + ... + (reserved ram for container n
> * lifetime of container n)
> It'd be nice to have this at the app level instead of the job level because:
> 1. We'd still be able to get memory usage for jobs that crashed (and wouldn't
> appear on the job history server).
> 2. We'd be able to get memory usage for future non-MR jobs (e.g. Storm).
> This new metric should be available both through the RM UI and RM Web
> Services REST API.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
