Hey Tony, I don't think Mesos would work nicely if you have multiple hierarchies (one per controller) pre 0.18.0. One solution is to un mount and remove all the cgroup hierarchies and let Mesos take control of it.
Alternatively, since 0.18.0, we did a major refactor around cgroups isolation and it works with multiple hierarchies. So you might want to give it (0.18.0-rc5) a try. @vinodkone On Fri, Mar 28, 2014 at 11:05 AM, Tony Landells <[email protected]> wrote: > Hi, > > I'm trying to get Mesos up as a PoC at work to see if it's something we > could use effectively. > > I'm running SL 6.4, installed libcgroup and ran cgconfig which creates the > cgroup mounts under /cgroup: > > [[email protected] ~]$ cat /proc/mounts | grep cgroup >> cgroup /cgroup/cpuset cgroup rw,relatime,cpuset 0 0 >> cgroup /cgroup/cpu cgroup rw,relatime,cpu 0 0 >> cgroup /cgroup/cpuacct cgroup rw,relatime,cpuacct 0 0 >> cgroup /cgroup/memory cgroup rw,relatime,memory 0 0 >> cgroup /cgroup/devices cgroup rw,relatime,devices 0 0 >> cgroup /cgroup/freezer cgroup rw,relatime,freezer 0 0 >> cgroup /cgroup/net_cls cgroup rw,relatime,net_cls 0 0 >> cgroup /cgroup/blkio cgroup rw,relatime,blkio 0 0 > > > I'm using the mesos_0.17.0_x86_64.rpm package downloaded from > mesosphere.io and I can't start a slave fine if I use process isolation, > but if I try cgroups I get the following in /var/log/messages: > > 2014-03-28T10:46:05.194958-07:00 tlandells-mesos02 mesos-slave[6242]: >> I0328 10:46:05.194540 6242 main.cpp:118] Build: 2014-02-27 15:07:00 by root >> 2014-03-28T10:46:05.194976-07:00 tlandells-mesos02 mesos-slave[6242]: >> I0328 10:46:05.194810 6242 main.cpp:120] Creating "cgroups" isolator >> 2014-03-28T10:46:05.195267-07:00 tlandells-mesos02 mesos-slave[6242]: >> I0328 10:46:05.195050 6242 main.cpp:132] Starting Mesos slave >> 2014-03-28T10:46:05.195657-07:00 tlandells-mesos02 mesos-slave[6242]: >> 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@712: >> Client environment:zookeeper.version=zookeeper C client 3.4.5 >> 2014-03-28T10:46:05.195667-07:00 tlandells-mesos02 mesos-slave[6242]: >> 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@716: >> Client environment:host.name=tlandells-mesos02.dev.box.net. >> 2014-03-28T10:46:05.195671-07:00 tlandells-mesos02 mesos-slave[6242]: >> 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@723: >> Client environment:os.name=Linux >> 2014-03-28T10:46:05.195674-07:00 tlandells-mesos02 mesos-slave[6242]: >> 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@724: >> Client environment:os.arch=2.6.32-358.18.1.el6.x86_64 >> 2014-03-28T10:46:05.195678-07:00 tlandells-mesos02 mesos-slave[6242]: >> I0328 10:46:05.195552 6242 slave.cpp:111] Slave started on 1)@ >> 10.3.18.184:5051 >> 2014-03-28T10:46:05.195681-07:00 tlandells-mesos02 mesos-slave[6242]: >> 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@725: >> Client environment:os.version=#1 SMP Tue Aug 27 14:23:09 CDT 2013 >> 2014-03-28T10:46:05.195773-07:00 tlandells-mesos02 mesos-slave[6242]: >> I0328 10:46:05.195708 6242 slave.cpp:211] Slave resources: cpus(*):2; >> mem(*):2808; disk(*):43161; ports(*):[31000-32000] >> 2014-03-28T10:46:05.196105-07:00 tlandells-mesos02 mesos-slave[6242]: >> 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@733: >> Client environment:user.name=tlandells >> 2014-03-28T10:46:05.196115-07:00 tlandells-mesos02 mesos-slave[6242]: >> 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@741: >> Client environment:user.home=/root >> 2014-03-28T10:46:05.196119-07:00 tlandells-mesos02 mesos-slave[6242]: >> 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@log_env@753: >> Client environment:user.dir=/home/tlandells >> 2014-03-28T10:46:05.196122-07:00 tlandells-mesos02 mesos-slave[6242]: >> 2014-03-28 10:46:05,195:6242(0x7fc5fafcb700):ZOO_INFO@zookeeper_init@786: >> Initiating client connection, host=tlandells-mesos01.dev.box.net:2181, >> tlandells-mesos02.dev.box.net:2181,tlandells-mesos03.dev.box.net:2181sessionTimeout=10000 >> watcher=0x7fc602fc8cb0 sessionId=0 >> sessionPasswd=<null> context=0x7fc5d8003b90 flags=0 >> 2014-03-28T10:46:05.197040-07:00 tlandells-mesos02 mesos-slave[6242]: >> I0328 10:46:05.196420 6269 cgroups_isolator.cpp:223] Using /cgroup as >> cgroups hierarchy root >> 2014-03-28T10:46:05.197058-07:00 tlandells-mesos02 mesos-slave[6242]: >> Failed to mount cgroups hierarchy at '/cgroup' because we could not remove >> existing directory: Directory not empty > > > The command being invoked by /usr/bin/mesos-init-wrapper is: > >> /usr/local/sbin/mesos-slave --master=zk:// >> tlandells-mesos01.dev.box.net:2181,tlandells-mesos02.dev.box.net:2181, >> tlandells-mesos03.dev.box.net:2181/mesos --log_dir=/var/log/mesos >> --isolation=cgroups > > > I haven't worked with cgroups before so I'm not clear if it's okay to have > Mesos using the default mounts or whether I should have something special > setup. I mean it's clear that Mesos wouldn't be able to empty out /cgroup > because it contains all the cgroup mounts... > > I saw a similar thread in the archives from about 6 months ago where the > issue was apparently a clash between Mesos and Docker, but I'm only trying > to run Mesos. > > The only thing that might be funky is that these are VMware instances I'm > currently using for testing. > > Does anyone have any experience running the cgroups isolator in SL (or > even RH, since it's basically just a rebadge) who can tell me what I'm > doing wrong? > > Thanks, > Tony >
