K8s dashboard did some integration with metrics-server, maybe we can investigate and see how that was done. Essentially we just need to pull these metrics somewhere.
On Tue, Dec 21, 2021 at 2:42 PM Chaoran Yu <[email protected]> wrote: > Previously when doing research on this topic, I saw that the metrics-server > documentation says:"*Metrics Server is not meant for non-autoscaling > purposes. For example, don't use it to forward metrics to monitoring > solutions, or as a source of monitoring solution metrics. In such cases > please collect metrics from Kubelet /metrics/resource endpoint directly*." > But the Kubelet APIs > < > https://github.com/kubernetes/kubernetes/blob/v1.21.5/pkg/kubelet/server/server.go#L236 > >that > the statement refers to are not documented, meaning they are hidden APIs > that can change or be deprecated at any future Kubernetes release. > Integrating with these APIs doesn't sound promising. But besides Kubelet, > the actual utilization info of workloads is not readily available anywhere > else. We'll need to explore other ideas. > > On Tue, Dec 21, 2021 at 12:51 PM Weiwei Yang <[email protected]> wrote: > > > Thank you Bowen to raise this up, this is an interesting topic. Bear with > > me this long reply : ) > > > > Like Wilfred mentioned, YK doesn't know about the actual used resources > in > > terms of CPU and memory for each pod, or application, at least not > today. I > > understand the requirements about tracking this info in order to give > users > > some feedback or even recommendations on how to tune their jobs more > > properly. It would be good to have something in our view as "Allocated" > vs > > "Used" for each app/queue. We could further introduce some penalties if > > people keep over-requesting resources. > > > > However, most likely we will need to do this outside of YK. The major > > reason is all data YK is consuming are from api-server, backed by etcd. > Non > > of such metrics will be stored in etcd, as per design in metrics-server > > <https://github.com/kubernetes-sigs/metrics-server>. Second, YK doesn't > > have any per-node agent running that we can facilitate to collect actual > > resource usages, we still need to leverage a 3rd party tool to do so. > Maybe > > we can do some integration with metrics-server, aggregating app/queue > used > > info from those fragmented metrics, and then plug that into our > > yunikorn-web UI. We have the flexibility to do this I believe, which > could > > be an option. > > > > On Mon, Dec 20, 2021 at 10:28 PM Wilfred Spiegelenburg < > > [email protected]> > > wrote: > > > > > Hi Bowen, > > > > > > Maybe a strange question but what do you consider "actually > > > used" resources? Anything the scheduler sees is used. The scheduler has > > no > > > information on what the container really occupies: it asked for 100GB > but > > > it only allocated 50GB etc. If you need that YuniKorn cannot help you. > If > > > it is just a looking at allocation over time YuniKorn is capable of > > giving > > > you the information. > > > > > > Second point to make is that normally applications do not provide any > > > information on what they expect to use before they use it. Let's take a > > > Spark application. The driver creates pods as it needs new executors. > The > > > Spark config drives those requests and the limitations. The scheduler > > only > > > sees the pods that are really requested. It does not know, and should > not > > > know, if that is limited by what is configured or that the job uses > only > > > part or more than what is configured. > > > > > > The only time the scheduler would have any idea about a "maximum" is > > when a > > > gang request is made. For gang scheduling we can track if the gang > > > request is completely used or not. We could add metrics for it on an > > > application. We can also track the number of containers allocated for > an > > > application or queue, the time from start to finish for containers etc. > > We > > > could even track the maximum resource allocation for an application or > a > > > queue over a time interval. Prometheus should give us a number of > > > possibilities, we just need to hook them into the scheduling cycle. > > > > > > As far as I know we currently do not have application metrics but that > > can > > > always be added. Some queue metrics are there already. I think one of > > those > > > is what you are looking for to fill a number of the gaps that you see. > I > > > have added YUNIKORN-829 as a subtask to YUNIKORN-720 [1] which is > already > > > referencing a number of metrics to improve. With the release of > v0.12.1 I > > > moved that jira to v1.0.0. A major improvement to the metrics would be > a > > > nice addition for the v1.0.0. > > > > > > I would not see anything that is blocking enhancing metrics: it is a > part > > > that can be improved without a major impact on other functionality. We > do > > > need to make sure that we measure the impact on performance and memory > > > usage. > > > > > > Wilfred > > > > > > [1] https://issues.apache.org/jira/browse/YUNIKORN-720 > > > > > > On Tue, 21 Dec 2021 at 16:18, Bowen Li <[email protected]> wrote: > > > > > > > Hi community, > > > > > > > > Reviving https://issues.apache.org/jira/browse/YUNIKORN-829 . We are > > > > running Spark on YuniKorn, and have a requirement to provide more > > > > observability of *actual* resource usage for our customers, data > > > > engineers/scientists who wrote Spark jobs who may not have deep > > expertise > > > > in Spark job optimization. > > > > > > > > - requirement: > > > > > > > > - have actual resource usage metrics at both job level and queue > level > > > (YK > > > > already have requested resource usage metrics) > > > > > > > > - key use case: > > > > > > > > - as indicators of job optimization for ICs like data > > > engineers/scientists, > > > > to show users how much resources they requested v.s. how much > resources > > > > their jobs actually used > > > > > > > > - as indicator for managers on their team's resource utilization. In > > our > > > > setup or a typical YK setup, each customer team has their own > YuniKorn > > > > queue in a shared, multi tenant environment. Managers of the team > would > > > > want high level (queue) metrics rather than low level (job) ones > > > > > > > > Currently we haven't found a good product on the market to do this, > so > > > > would be great if YuniKorn can support it. Would like your input here > > on > > > > feasibility (seems feasible according Weiwei's comment in Jira), > > > priority, > > > > and timeline/complexity of the projects. > > > > > > > > Thanks, > > > > Bowen > > > > > > > > > >
