Jason Lowe commented on YARN-2440:

For the case presented by [~sjlee0] the user has an 8 core system and wants to 
use at most 6 cores for YARN containers. That can be done by simply setting 
containers-cpu-percentage to 75.  I don't see why we need a separate 
containers-cpu-cores parameter here, and I think it causes more problems than 
it solves per my previous comment.  If we only want to support whole-core 
granularity then I can see containers-cpu-cores as a better choice, but 
otherwise containers-cpu-percentage is more flexible.

Also I don't see vcores being relevant for this JIRA.  The way vcores map to 
physical cores is node-dependent, but apps ask for vcores in a node-independent 
fashion.  IIUC this JIRA is focused on simply limiting the amount of CPU all 
YARN containers on the node can possibly use in aggregate.  Changing the 
vcore-to-core ratio on the node will change how many containers the node might 
run simultaneously, but it shouldn't impact how much of the physical CPU the 
user wants reserved for non-container processes.

On a related note, it's interesting to step back and see if this is really what 
most users will want in practice.  If the intent is to ensure the NM, DN, and 
other system processes get enough CPU time then I think a better approach is to 
put those system processes in a peer cgroup to the YARN containers cgroup and 
set their relative CPU shares accordingly.  Then YARN containers can continue 
to use any spare CPU if desired (i.e.: no CPU "fragmentation") but the system 
processes are guaranteed not to be starved out by the YARN containers.  Some 
users may want a hard limit and hence why this feature would be useful for 
them, but I suspect most users will not want to leave spare CPU lying around 
when containers need it.

bq. How about yarn.nodemanager.all-containers-cpu-cores and 

I'm indifferent on adding "all" as a prefix.  Something like 
yarn.nodemanager.containers-limit-cpu-percentage might be more clear that this 
is a hard limit and CPUs can go idle even if containers are demanding more from 
the machine than this limit.

> Cgroups should allow YARN containers to be limited to allocated cores
> ---------------------------------------------------------------------
>                 Key: YARN-2440
>                 URL: https://issues.apache.org/jira/browse/YARN-2440
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, 
> apache-yarn-2440.2.patch, screenshot-current-implementation.jpg
> The current cgroups implementation does not limit YARN containers to the 
> cores allocated in yarn-site.xml.

This message was sent by Atlassian JIRA

Reply via email to