[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores

2014-09-10 Thread Varun Vasudev (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-2440:

Attachment: apache-yarn-2440.6.patch

Uploaded new patch to address Vinod's comments.

{quote}
{noformat}
   property
  +descriptionPercentage of CPU that can be allocated
  +for containers. This setting allows users to limit the number of
  +physical cores that YARN containers use. Currently functional only
  +on Linux using cgroups. The default is to use 100% of CPU.
  +/description
  +nameyarn.nodemanager.resource.percentage-physical-cpu-limit/name
  +value100/value
  +  /property
{noformat}

the number of physical cores part isn't really right. It actually is 75% 
across all cores, for e.g. We have this sort of number of physical cores 
description in multiple places, let's fix that? For instance, in 
NodeManagerHardwareUtils, yarn-default.xml etc.

{quote}

Fixed.

{quote}
Also,
NM_CONTAINERS_CPU_PERC - NM_RESOURCE_PHYSICAL_CPU_LIMIT
Similarly rename DEFAULT_NM_CONTAINERS_CPU_PERC
{quote}

Done, I'd prefer to have percentage as part of the name. I've changed it to 
NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT and 
DEFAULT_NM_RESOURCE_PERCENTAGE_PHYSICAL_CPU_LIMIT.

 Cgroups should allow YARN containers to be limited to allocated cores
 -

 Key: YARN-2440
 URL: https://issues.apache.org/jira/browse/YARN-2440
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, 
 apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, 
 apache-yarn-2440.5.patch, apache-yarn-2440.6.patch, 
 screenshot-current-implementation.jpg


 The current cgroups implementation does not limit YARN containers to the 
 cores allocated in yarn-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores

2014-09-09 Thread Varun Vasudev (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-2440:

Attachment: apache-yarn-2440.5.patch

Uploaded new patch to address Vinod's concerns.

bq.containers-limit-cpu-percentage - 
yarn.nodemanager.resource.percentage-cpu-limit to be consistent? Similarly 
NM_CONTAINERS_CPU_PERC? I don't like the tag 'resource', it should have been 
'resources' but it is what it is.

I'm worried that calling it that will lead users to think it's a percentage of 
the vcores that they specify. In the patch I've changed it to 
yarn.nodemanager.resource.percentage-physical-cpu-limit but if you or Jason 
feel strongly about it, I can change it to 
yarn.nodemanager.resource.percentage-cpu-limit.

bq.You still have refs to YarnConfiguration.NM_CONTAINERS_CPU_ABSOLUTE in 
the patch. Similarly the javadoc in NodeManagerHardwareUtils needs to be 
updated if we are not adding the absolute cpu config. It should no longer refer 
to number of cores that should be used for YARN containers

Fixed.

bq.TestCgroupsLCEResourcesHandler: You can use mockito if you only want to 
override num-processors in TestResourceCalculatorPlugin. Similarly in 
TestNodeManagerHardwareUtils.

Switched to mockito.

bq.The tests may fail on a machine with  4 cores?
Don't think so. The tests mock the getNumProcessors() function so we should be 
fine.


 Cgroups should allow YARN containers to be limited to allocated cores
 -

 Key: YARN-2440
 URL: https://issues.apache.org/jira/browse/YARN-2440
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, 
 apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, 
 apache-yarn-2440.5.patch, screenshot-current-implementation.jpg


 The current cgroups implementation does not limit YARN containers to the 
 cores allocated in yarn-site.xml.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores

2014-08-27 Thread Varun Vasudev (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-2440:


Attachment: apache-yarn-2440.3.patch

Uploaded new patch which addresses [~jlowe] comments. 
1. Dropped yarn.nodemanager.containers-cpu-cores, only percentage can be 
specified.
2. Renamed yarn.nodemanager.containers-cpu-percentage to 
yarn.nodemanger.containers-limit-cpu-percentage
3. Added log message if we are unable to use Cgroups because the values  for 
period or quota are too low.
4. Added check for minimum value of quota
5. Don't set any period or quota settings if all CPUs are being used.
6. Renamed getContainerCores to getContainersCores

 Cgroups should allow YARN containers to be limited to allocated cores
 -

 Key: YARN-2440
 URL: https://issues.apache.org/jira/browse/YARN-2440
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, 
 apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, 
 screenshot-current-implementation.jpg


 The current cgroups implementation does not limit YARN containers to the 
 cores allocated in yarn-site.xml.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores

2014-08-27 Thread Varun Vasudev (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-2440:


Attachment: apache-yarn-2440.4.patch

Uploaded a new patch that missed use case where usage was reset to 100% after 
being lowered.

 Cgroups should allow YARN containers to be limited to allocated cores
 -

 Key: YARN-2440
 URL: https://issues.apache.org/jira/browse/YARN-2440
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, 
 apache-yarn-2440.2.patch, apache-yarn-2440.3.patch, apache-yarn-2440.4.patch, 
 screenshot-current-implementation.jpg


 The current cgroups implementation does not limit YARN containers to the 
 cores allocated in yarn-site.xml.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores

2014-08-24 Thread Varun Vasudev (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-2440:


Summary: Cgroups should allow YARN containers to be limited to allocated 
cores  (was: Cgroups should limit YARN containers to cores allocated in 
yarn-site.xml)

 Cgroups should allow YARN containers to be limited to allocated cores
 -

 Key: YARN-2440
 URL: https://issues.apache.org/jira/browse/YARN-2440
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, 
 screenshot-current-implementation.jpg


 The current cgroups implementation does not limit YARN containers to the 
 cores allocated in yarn-site.xml.



--
This message was sent by Atlassian JIRA
(v6.2#6252)


[jira] [Updated] (YARN-2440) Cgroups should allow YARN containers to be limited to allocated cores

2014-08-24 Thread Varun Vasudev (JIRA)

 [ 
https://issues.apache.org/jira/browse/YARN-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Vasudev updated YARN-2440:


Attachment: apache-yarn-2440.2.patch

Uploaded a new patch. It adds two new parameters to the YARN configuration.
1. yarn.nodemanager.containers-cpu-cores - can be set to the number of cores 
admins want YARN containers to use.
2. yarn.nodemanager.containers-cpu-percentage - can be set a percentage of 
overall CPU that admins want YARN containers to use.

By  default, we use all cores.

I've also updated the description of yarn.nodemanager.resource.cpu-vcores to 
what I think is a more accurate version.

 Cgroups should allow YARN containers to be limited to allocated cores
 -

 Key: YARN-2440
 URL: https://issues.apache.org/jira/browse/YARN-2440
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Varun Vasudev
Assignee: Varun Vasudev
 Attachments: apache-yarn-2440.0.patch, apache-yarn-2440.1.patch, 
 apache-yarn-2440.2.patch, screenshot-current-implementation.jpg


 The current cgroups implementation does not limit YARN containers to the 
 cores allocated in yarn-site.xml.



--
This message was sent by Atlassian JIRA
(v6.2#6252)