[
https://issues.apache.org/jira/browse/YARN-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tao Yang updated YARN-8692:
---------------------------
Description:
The distribution of node utilization is an important healthy factor for the
YARN cluster, related metrics in SLS can be used to evaluate the scheduling
effects and optimize related configurations.
To implement this improvement, we need to do things as below:
(1) Add input configurations (contain avg and stddev for cpu/memory utilization
ratio) and generate utilization samples for tasks, not include AM container
cause I think it's negligible. (2) Simulate containers and node utilization
within node status.
(3) calculate and generate the distribution metrics and use standard deviation
metric (stddev for short) to evaluate the effects(smaller is better).
(4) show these metrics on SLS simulator page like this:
!image-2018-08-21-18-04-22-749.png!
For Node memory/CPU utilization distribution graphs, Y-axis is nodes number,
and P0 represents 0%~9% utilization ratio(containers-utilization /
node-total-resource), P1 represents 10%~19% utilization ratio, P2 represents
20%~29% utilization ratio, ..., at last P9 represents 90%~100% utilization
ratio.
was:
The distribution of node utilization is an important healthy factor for the
YARN cluster, related metrics in SLS can be used to evaluate the scheduling
effects and optimize related configurations.
To implement this improvement, we need to do things as below:
(1) Add input configurations (contain avg and stddev for cpu/memory utilization
ratio) and generate utilization samples for tasks, not include AM container
cause I think it's negligible. (2) Simulate containers and node utilization
within node status.
(3) calculate and generate the distribution metrics and use standard deviation
metric (stddev for short) to evaluate the effects(smaller is better).
(4) show these metrics on SLS simulator page like this:
!image-2018-08-21-17-50-04-011.png!
For Node memory/CPU utilization distribution graphs, Y-axis is nodes number,
and P0 represents 0%~9% utilization ratio(containers-utilization /
node-total-resource), P1 represents 10%~19% utilization ratio, P2 represents
20%~29% utilization ratio, ..., at last P9 represents 90%~100% utilization
ratio.
> Support node utilization metrics for SLS
> ----------------------------------------
>
> Key: YARN-8692
> URL: https://issues.apache.org/jira/browse/YARN-8692
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler-load-simulator
> Affects Versions: 3.2.0
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Major
> Attachments: image-2018-08-21-18-03-59-665.png,
> image-2018-08-21-18-04-22-749.png
>
>
> The distribution of node utilization is an important healthy factor for the
> YARN cluster, related metrics in SLS can be used to evaluate the scheduling
> effects and optimize related configurations.
> To implement this improvement, we need to do things as below:
> (1) Add input configurations (contain avg and stddev for cpu/memory
> utilization ratio) and generate utilization samples for tasks, not include AM
> container cause I think it's negligible. (2) Simulate containers and node
> utilization within node status.
> (3) calculate and generate the distribution metrics and use standard
> deviation metric (stddev for short) to evaluate the effects(smaller is
> better).
> (4) show these metrics on SLS simulator page like this:
> !image-2018-08-21-18-04-22-749.png!
> For Node memory/CPU utilization distribution graphs, Y-axis is nodes number,
> and P0 represents 0%~9% utilization ratio(containers-utilization /
> node-total-resource), P1 represents 10%~19% utilization ratio, P2 represents
> 20%~29% utilization ratio, ..., at last P9 represents 90%~100% utilization
> ratio.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
