[
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13296036#comment-13296036
]
Andrew Ferguson commented on MAPREDUCE-4334:
--------------------------------------------
Hi Arun,
I've thought some more about implementing taskset since our chat at the YARN
meet-up.
One benefit of cgroups is they're "set it and forget it" -- in the
ContainerExecutor, we simply place the new task in the appropriate cgroup, and
the kernel will take care of the rest. This would allow us to ditch the
ContainersMonitor infrastructure.
On the other hand, with taskset, we will need to do the CPU scheduling
ourselves. Say I have two cores and start with two processes, A (requested 0.5
cores) and B (requested 0.5 cores). I can start by putting them both on core 1
for efficiency, or I can put them on separate cores for higher utilization. But
if process C (requested 1 core) comes along, I will need to set A & B to the
same core. This is just a simple scenario, but more cores and processes will
likely grow a complicated CPU scheduler inside the NodeManager
(ContainersMonitorImpl is probably the right place, since it is already
monitoring container resource usage).
tl;dr -- I believe cgroups requires only local state when launching containers,
while taskset requires us to maintain global state.
thoughts?
thanks!
Andrew
> Add support for CPU isolation/monitoring of containers
> ------------------------------------------------------
>
> Key: MAPREDUCE-4334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
>
> Once we get in MAPREDUCE-4327, it will be important to actually enforce
> limits on CPU consumption of containers.
> Several options spring to mind:
> # taskset (RHEL5+)
> # cgroups (RHEL6+)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
