[
https://issues.apache.org/jira/browse/YUNIKORN-1385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17631059#comment-17631059
]
Wilfred Spiegelenburg commented on YUNIKORN-1385:
-------------------------------------------------
The next thing we would need to discuss is if we want to store history of an
application:
* Currently we do not keep history.
* Keeping history is a memory consuming job with overhead for processing.
* We do not want to install a database as part of YuniKorn
The choice to not store and track history beside counters was made consciously.
Logging the report might be nice but we are trying to get rid of all the local
logs in the pod. We are looking for long duration uptimes and local logs are
not the way to go. It would mean that you need to filter them out of the
standard logs that are generated. With the current logs that is doable.
A push service might be possible as long as it is out of band of the scheduling
cycle. History should never impact scheduler resource usage or scheduling
cycles.
One last thing: YuniKorn can only look at the scheduled values. Looking at the
extended tracking it looks like the basis for a chargeback system. You cannot
and must not rely on any of this for chargeback. You can schedule a pod on a
node with 1 core request and no limit set. If the node is empty the pod will
use the whole node. The pod in that case runs much quicker than when scheduled
on a full node. YuniKorn will not even know if the pod really uses all memory.
YuniKorn contrary to YARN will never run on each node in the cluster. The info
you thus can get is of limited value and only gives an indication.
> To provide visibility to application's aggregated resource consumption
> ----------------------------------------------------------------------
>
> Key: YUNIKORN-1385
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1385
> Project: Apache YuniKorn
> Issue Type: New Feature
> Components: core - common, core - scheduler, shim - kubernetes
> Reporter: Yongjun Zhang
> Assignee: Yongjun Zhang
> Priority: Major
> Attachments: image-2022-11-09-11-06-20-479.png,
> yunikornResourceUsageVisibility.pdf
>
>
> Currently the "Used Resource" for a given application reported at Yunikorn
> GUI is actually the snapshot of the resources currently allocated to the
> application. We need to provide visibility to how much resources an
> application has used so far, and it would be important to report how much
> total resources is used by the application.
> The unit of resources used can be memoryseconds/vcoreseconds like how Hadoop
> Yarn does in its app summary report. However, there is more complexity with
> K8s which supports different instance types (Hadoop Yarn does support
> multiple instance types but its app summary report ignored instance types).
> It would be nice to report how much resource is used for each instance types
> used by an application.
> This is a top level Jira for this new feature. Subtask jira will be created
> when we work on it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
