Varun Vasudev commented on YARN-2440:

[~jlowe] the example provided by [~sjlee0] is the one I wanted to address when 
I added support for both percentage and absolute cores. Would it make more 
sense if I picked the lower value instead of one overriding the other. 
Something like -

1. Evaluate cores allocated by yarn.nodemanager.containers-cpu-cores and 
2. Pick the lower of the two values
3. Log a warning/info message that both were specified and that we're picking 
the lower value.

I'm not thrilled about the name template "containers-cpu-*" since it could 
easily be misinterpreted as a per-container thing as well, but I'm currently at 
a loss for a better prefix. Suggestions welcome.

How about yarn.nodemanager.all-containers-cpu-cores and 

Does getOverallLimits need to check for a quotaUS that's too low as well?

Thanks for catching this; I'll fix it in the next patch.

I think minimally we need to log a warning if we're going to ignore setting up 
cgroups to limit CPU usage across all containers if the user specified to do so.
I'll add in the logging message.

Related to the previous comment, I think it would be nice if we didn't try to 
setup any limits if none were specified. That way if there's some issue with 
correctly determining the number of cores on a particular system it can still 
work in the default, "use everything" scenario.
Will do.

NodeManagerHardwareUtils.getContainerCores should be getContainersCores (the 
per-container vs. all-containers confusion again)
I'll rename the function.

> Cgroups should allow YARN containers to be limited to allocated cores
> ---------------------------------------------------------------------
>                 Key: YARN-2440
>                 URL: https://issues.apache.org/jira/browse/YARN-2440
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, 
> apache-yarn-2440.2.patch, screenshot-current-implementation.jpg
> The current cgroups implementation does not limit YARN containers to the 
> cores allocated in yarn-site.xml.

This message was sent by Atlassian JIRA

Reply via email to