[
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Ferguson updated MAPREDUCE-4334:
---------------------------------------
Attachment: MAPREDUCE-4334-pre2.patch
This version is ready for testing. It has the following requires:
1) Apply patch in MAPREDUCE-4351. This allows you to set
yarn.nodemanager.containers-monitor.class to
o.a.h.yarn.server.nodemanager.containermanager.monitor.CgroupsContainersMonitor
2) Mount the cgroups memory controller at a path of your choosing. For example:
$ sudo mount -t cgroup -o memory none /cgroups/mem
The NodeManager will detect where you have mounted the cgroups.
3) Create a cgroups hierarchy which Hadoop can use. This is most easily done
with:
$ sudo cgcreate -a hadoop_user_name -g memory:hadoop-yarn
"hadoop-yarn" is the default hierarchy the NodeManager expects; this can be
configured with yarn.nodemanager.cgroups.path.
that's it!
I have tested that it enforces memory limits, and reacts appropriately when the
kernel kills processes, or when they complete successfully. It also notifies
the user when cgroups have been mis-configured (for example, if the Hadoop user
does not have write access to the cgroup hierarchy).
Currently, it only enforces memory limits, as per the trunk code. I am planning
to augment the patch in MAPREDUCE-4327 to provide CPU limits to the
ContainersMonitor. It is easy to extend this patch to any other cgroup
controller.
thank you,
Andrew
> Add support for CPU isolation/monitoring of containers
> ------------------------------------------------------
>
> Key: MAPREDUCE-4334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Attachments: MAPREDUCE-4334-pre1.patch, MAPREDUCE-4334-pre2.patch
>
>
> Once we get in MAPREDUCE-4327, it will be important to actually enforce
> limits on CPU consumption of containers.
> Several options spring to mind:
> # taskset (RHEL5+)
> # cgroups (RHEL6+)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
