[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-2440:
Attachment: apache-yarn-2440.6.patch
Uploaded new patch to address Vinod's comments.
{quote}
{noformat}
+Percentage of CPU that can be allocated
+for containers. This setting allows users to limit the number of
+physical cores that YARN containers use. Currently functional only
+on Linux using cgroups. The default is to use 100% of CPU.
+
+yarn.nodemanager.resource.percentage-physical-cpu-limit
+100
+
{noformat}
"the number of physical cores" part isn't really right. It actually is 75%
across all cores, for e.g. We have this sort of "number of physical cores"
description in multiple places, let's fix that? For instance, in
NodeManagerHardwareUtils, yarn-default.xml etc.
{quote}
Fixed.
{quote}
Also,
NM_CONTAINERS_CPU_PERC -> NM_RESOURCE_PHYSICAL_CPU_LIMIT
Similarly rename DEFAULT_NM_CONTAINERS_CPU_PERC
{quote}
Done, I'd prefer to have percentage as part of the name. I've changed it to
NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT and
DEFAULT_NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT.
> Cgroups should allow YARN containers to be limited to allocated cores
> -
>
> Key: YARN-2440
> URL: https://issues.apache.org/jira/browse/YARN-2440
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
> Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch,
> apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch,
> apache-yarn-2440.5.patch, apache-yarn-2440.6.patch,
> screenshot-current-implementation.jpg
>
>
> The current cgroups implementation does not limit YARN containers to the
> cores allocated in yarn-site.xml.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-2440: Attachment: apache-yarn-2440.5.patch Uploaded new patch to address Vinod's concerns. bq.containers-limit-cpu-percentage -> yarn.nodemanager.resource.percentage-cpu-limit to be consistent? Similarly NM_CONTAINERS_CPU_PERC? I don't like the tag 'resource', it should have been 'resources' but it is what it is. I'm worried that calling it that will lead users to think it's a percentage of the vcores that they specify. In the patch I've changed it to yarn.nodemanager.resource.percentage-physical-cpu-limit but if you or Jason feel strongly about it, I can change it to yarn.nodemanager.resource.percentage-cpu-limit. bq.You still have refs to YarnConfiguration.NM_CONTAINERS_CPU_ABSOLUTE in the patch. Similarly the javadoc in NodeManagerHardwareUtils needs to be updated if we are not adding the absolute cpu config. It should no longer refer to "number of cores that should be used for YARN containers" Fixed. bq.TestCgroupsLCEResourcesHandler: You can use mockito if you only want to override num-processors in TestResourceCalculatorPlugin. Similarly in TestNodeManagerHardwareUtils. Switched to mockito. bq.The tests may fail on a machine with > 4 cores? Don't think so. The tests mock the getNumProcessors() function so we should be fine. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, > apache-yarn-2440.5.patch, screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-2440: Attachment: apache-yarn-2440.4.patch Uploaded a new patch that missed use case where usage was reset to 100% after being lowered. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-2440: Attachment: apache-yarn-2440.3.patch Uploaded new patch which addresses [~jlowe] comments. 1. Dropped yarn.nodemanager.containers-cpu-cores, only percentage can be specified. 2. Renamed yarn.nodemanager.containers-cpu-percentage to yarn.nodemanger.containers-limit-cpu-percentage 3. Added log message if we are unable to use Cgroups because the values for period or quota are too low. 4. Added check for minimum value of quota 5. Don't set any period or quota settings if all CPUs are being used. 6. Renamed getContainerCores to getContainersCores > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-2440: Attachment: apache-yarn-2440.2.patch Uploaded a new patch. It adds two new parameters to the YARN configuration. 1. yarn.nodemanager.containers-cpu-cores - can be set to the number of cores admins want YARN containers to use. 2. yarn.nodemanager.containers-cpu-percentage - can be set a percentage of overall CPU that admins want YARN containers to use. By default, we use all cores. I've also updated the description of yarn.nodemanager.resource.cpu-vcores to what I think is a more accurate version. > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > apache-yarn-2440.2.patch, screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores
[ https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-2440: Summary: Cgroups should allow YARN containers to be limited to allocated cores (was: Cgroups should limit YARN containers to cores allocated in yarn-site.xml) > Cgroups should allow YARN containers to be limited to allocated cores > - > > Key: YARN-2440 > URL: https://issues.apache.org/jira/browse/YARN-2440 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, > screenshot-current-implementation.jpg > > > The current cgroups implementation does not limit YARN containers to the > cores allocated in yarn-site.xml. -- This message was sent by Atlassian JIRA (v6.2#6252)
