[
https://issues.apache.org/jira/browse/YARN-5179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15934970#comment-15934970
]
Manikandan R commented on YARN-5179:
------------------------------------
Thanks [[email protected]] for your comments.
After my previous comment and proposals, had a offline discussion with
[[email protected]] and was able to correct my understanding of
vcores in general and especially by playing around with Default Container
Executor & Linux container executor (with cgroups as well).
Then, I ran a test with stress command (stress --cpu 4) to confirm my
understanding about vcores with various possible options and compiled results
in excel sheet. I've attached the same for reference. My box has 2 pcores and
resourceCalculatorPlugin.getNumProcessors() is 4. Various Options are
Using Linux container executor -
1. Physical cpu limit is 100, Vcores is 2, Strict resource usage is false.
cpuUsagePercentPerCore is 320 & Millivcores is 1600
2. Physical cpu limit is 100, Vcores is 4, Strict resource usage is false.
cpuUsagePercentPerCore is 360 & Millivcores is 3600
3. Physical cpu limit is 50, Vcores is 2, Strict resource usage is false.
cpuUsagePercentPerCore is 99 & Millivcores is 990
4. Physical cpu limit is 50, Vcores is 4, Strict resource usage is false.
cpuUsagePercentPerCore is 98 & Millivcores is 1960
5. Physical cpu limit is 100, Vcores is 2, Strict resource usage is true.
cpuUsagePercentPerCore is 100 & Millivcores is 500
6. Physical cpu limit is 100, Vcores is 4, Strict resource usage is true.
cpuUsagePercentPerCore is 50 & Millivcores is 500
7. Physical cpu limit is 50, Vcores is 2, Strict resource usage is true.
cpuUsagePercentPerCore is 50 & Millivcores is 500
8. Physical cpu limit is 50, Vcores is 4, Strict resource usage is true.
cpuUsagePercentPerCore is 25 & Millivcores is 500
Using Default Linux executor -
Physical cpu limit is 100, Vcores is 200. cpuUsagePercentPerCore is 380 &
Millivcores is 190000
Physical cpu limit is 50, Vcores is 200. cpuUsagePercentPerCore is 375 &
Millivcores is 375000
Verified O/p of millivcores for each above option and its correctness with
[[email protected]]. Initally, We had a doubt on our own
understanding only when physical cpu limit is 50 and thought it should be half
of the current millivcores, then we were able to reason it this way - for ex,
in option 3, max utilization can go upto 200 because
resourceCalculatorPlugin.getNumProcessors() is 2 (50% of 4 processors), but we
got 99 as utilization. It means half of vcores should have been utilized, hence
millivcores should be 1000 (approx). Based on these test results and our
understanding, we don't see any issue with current calculation (like
[[email protected]] said in his comments).
Please correct me if you see any gap.
> Issue of CPU usage of containers
> --------------------------------
>
> Key: YARN-5179
> URL: https://issues.apache.org/jira/browse/YARN-5179
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 2.7.0
> Environment: Both on Windows and Linux
> Reporter: Zhongkai Mi
> Attachments: YARN-5179.xls
>
>
> // Multiply by 1000 to avoid losing data when converting to int
> int milliVcoresUsed = (int) (cpuUsageTotalCoresPercentage * 1000
> * maxVCoresAllottedForContainers /nodeCpuPercentageForYARN);
> This formula will not get right CPU usage based vcore if vcores != physical
> cores.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
