Hi Bowen, Maybe a strange question but what do you consider "actually used" resources? Anything the scheduler sees is used. The scheduler has no information on what the container really occupies: it asked for 100GB but it only allocated 50GB etc. If you need that YuniKorn cannot help you. If it is just a looking at allocation over time YuniKorn is capable of giving you the information.
Second point to make is that normally applications do not provide any information on what they expect to use before they use it. Let's take a Spark application. The driver creates pods as it needs new executors. The Spark config drives those requests and the limitations. The scheduler only sees the pods that are really requested. It does not know, and should not know, if that is limited by what is configured or that the job uses only part or more than what is configured. The only time the scheduler would have any idea about a "maximum" is when a gang request is made. For gang scheduling we can track if the gang request is completely used or not. We could add metrics for it on an application. We can also track the number of containers allocated for an application or queue, the time from start to finish for containers etc. We could even track the maximum resource allocation for an application or a queue over a time interval. Prometheus should give us a number of possibilities, we just need to hook them into the scheduling cycle. As far as I know we currently do not have application metrics but that can always be added. Some queue metrics are there already. I think one of those is what you are looking for to fill a number of the gaps that you see. I have added YUNIKORN-829 as a subtask to YUNIKORN-720 [1] which is already referencing a number of metrics to improve. With the release of v0.12.1 I moved that jira to v1.0.0. A major improvement to the metrics would be a nice addition for the v1.0.0. I would not see anything that is blocking enhancing metrics: it is a part that can be improved without a major impact on other functionality. We do need to make sure that we measure the impact on performance and memory usage. Wilfred [1] https://issues.apache.org/jira/browse/YUNIKORN-720 On Tue, 21 Dec 2021 at 16:18, Bowen Li <[email protected]> wrote: > Hi community, > > Reviving https://issues.apache.org/jira/browse/YUNIKORN-829 . We are > running Spark on YuniKorn, and have a requirement to provide more > observability of *actual* resource usage for our customers, data > engineers/scientists who wrote Spark jobs who may not have deep expertise > in Spark job optimization. > > - requirement: > > - have actual resource usage metrics at both job level and queue level (YK > already have requested resource usage metrics) > > - key use case: > > - as indicators of job optimization for ICs like data engineers/scientists, > to show users how much resources they requested v.s. how much resources > their jobs actually used > > - as indicator for managers on their team's resource utilization. In our > setup or a typical YK setup, each customer team has their own YuniKorn > queue in a shared, multi tenant environment. Managers of the team would > want high level (queue) metrics rather than low level (job) ones > > Currently we haven't found a good product on the market to do this, so > would be great if YuniKorn can support it. Would like your input here on > feasibility (seems feasible according Weiwei's comment in Jira), priority, > and timeline/complexity of the projects. > > Thanks, > Bowen >
