Varun Vasudev commented on YARN-2190:

[~chuanliu] one more issue with the patch -
 if (conf.getBoolean(YarnConfiguration.NM_WINDOWS_CONTAINER_CPU_LIMIT_ENABLED,
+        int vcores = resource.getVirtualCores();
+        // cap overall usage to the number of cores allocated to YARN
+        float yarnProcessors = NodeManagerHardwareUtils.getContainersCores(
+            ResourceCalculatorPlugin.getResourceCalculatorPlugin(null, conf),
+            conf);
+        // CPU should be set to a percentage * 100, e.g. 20% cpu rate limit
+        // should be set as 20 * 100. The following setting is equal to:
+        // 100 * (100 * (vcores / Total # of cores allocated to YARN))
+        cpuRate = Math.min(10000, (int) ((vcores * 10000) / yarnProcessors));
+      }

This may not behave as users expected. The 'yarnProcessors' that you receive 
from the NodeManagerHardwareUtils is the number of physical cores allocated to 
YARN containers. However resource.getVirtualCores() returns a number that the 
user submits(and can be potentially greater than 'yarnProcessors'). For 
example, an admin sets 'yarn.nodemanager.resource.cpu-vcores' to 24 on a node 
with 4 cores(this can be done by admins who wish to oversubscribe nodes). He 
also sets 'yarn.nodemanager.resource.percentage-physical-cpu-limit' to 50 
indicating that only 2 physical cores are to be used for YARN containers. The 
RM allocates two containers with 12 vcores each on the node. According to your 
math both containers would get 100% cpu, when each container should only get 
25% cpu.

What you need to do is scale the container vcores to the number of physical 
cores and not use the value as provided.

> Provide a Windows container executor that can limit memory and CPU
> ------------------------------------------------------------------
>                 Key: YARN-2190
>                 URL: https://issues.apache.org/jira/browse/YARN-2190
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager
>            Reporter: Chuan Liu
>            Assignee: Chuan Liu
>         Attachments: YARN-2190-prototype.patch, YARN-2190.1.patch, 
> YARN-2190.10.patch, YARN-2190.2.patch, YARN-2190.3.patch, YARN-2190.4.patch, 
> YARN-2190.5.patch, YARN-2190.6.patch, YARN-2190.7.patch, YARN-2190.8.patch, 
> YARN-2190.9.patch
> Yarn default container executor on Windows does not set the resource limit on 
> the containers currently. The memory limit is enforced by a separate 
> monitoring thread. The container implementation on Windows uses Job Object 
> right now. The latest Windows (8 or later) API allows CPU and memory limits 
> on the job objects. We want to create a Windows container executor that sets 
> the limits on job objects thus provides resource enforcement at OS level.
> http://msdn.microsoft.com/en-us/library/windows/desktop/ms686216(v=vs.85).aspx

This message was sent by Atlassian JIRA

Reply via email to