[
https://issues.apache.org/jira/browse/YUNIKORN-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17631930#comment-17631930
]
Weiwei Yang commented on YUNIKORN-1385:
---------------------------------------
hi [~yzhangal] thanks for putting all these together in a doc.
I want to confirm that we are on the same page for one thing: we only aggregate
the "requested" resource seconds, and we do not track the actually utilized
resources. To give an example, if an executor requests for 3 cores and runs for
10 seconds, then the resource seconds is 30. But behind this on the K8s node,
this pod may only use 1 core or use 5 cores, that's the "actual" utilization
and we do not want to track that.
K8s allows you to set a limit, that limits per pod actual usage, and it has a
metrics server (a separate component) that can track the "actual" usage. Also
some other 3rd party tools can achieve the similar goal, such as datadog. These
are not part of the scope we are discussing here.
> To provide visibility to application's aggregated resource consumption
> ----------------------------------------------------------------------
>
> Key: YUNIKORN-1385
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1385
> Project: Apache YuniKorn
> Issue Type: New Feature
> Components: core - common, core - scheduler, shim - kubernetes
> Reporter: Yongjun Zhang
> Assignee: Yongjun Zhang
> Priority: Major
> Attachments: image-2022-11-09-11-06-20-479.png,
> yunikornResourceUsageVisibility.pdf
>
>
> Currently the "Used Resource" for a given application reported at Yunikorn
> GUI is actually the snapshot of the resources currently allocated to the
> application. We need to provide visibility to how much resources an
> application has used so far, and it would be important to report how much
> total resources is used by the application.
> The unit of resources used can be memoryseconds/vcoreseconds like how Hadoop
> Yarn does in its app summary report. However, there is more complexity with
> K8s which supports different instance types (Hadoop Yarn does support
> multiple instance types but its app summary report ignored instance types).
> It would be nice to report how much resource is used for each instance types
> used by an application.
> This is a top level Jira for this new feature. Subtask jira will be created
> when we work on it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
