[Lxc-users] Problems starting OL6.3 lxc container
Hi all, I have setup my first OL6 container but it doesn't starts. When I launch lxc-start command, nothing appears: [root@ol6host templates]# lxc-start -n ol6vmserver -o /tmp/ol.log --logpriority=DEBUG My config is: # Container configuration for Oracle Linux 6 lxc.arch = x86_64 lxc.utsname = ol6vmserver #lxc.devttydir = lxc lxc.console = /vmdata/ol6vmserver/dev/console lxc.tty = 4 lxc.pts = 1024 lxc.rootfs = /vmdata/ol6vmserver lxc.mount = /vmdata/lxc-config/ol6vmserver.fstab lxc.network.type = veth lxc.network.flags = up lxc.network.link = prodif lxc.network.name = eth0 lxc.network.mtu = 1500 lxc.network.hwaddr = 00:50:56:21:2a:d2 lxc.network.ipv4 = 172.25.50.7/27 lxc.cgroup.devices.deny = a lxc.cgroup.devices.allow = c 1:3 rwm lxc.cgroup.devices.allow = c 1:5 rwm lxc.cgroup.devices.allow = c 1:7 rwm lxc.cgroup.devices.allow = c 5:0 rwm lxc.cgroup.devices.allow = c 1:8 rwm lxc.cgroup.devices.allow = c 1:9 rwm lxc.cgroup.devices.allow = c 136:* rwm lxc.cgroup.devices.allow = c 5:2 rwm lxc.cgroup.devices.allow = c 254:0 rwm Trying to access using lxc-console: [root@ol6host dev]# lxc-console -n ol6vmserver Type Ctrl+a q to exit the console And in log file, I don't see nothing strange: lxc-start 1350808610.402 DEBUGlxc_conf - allocated pty '/dev/pts/4' (4/5) lxc-start 1350808610.403 DEBUGlxc_conf - allocated pty '/dev/pts/5' (6/7) lxc-start 1350808610.403 DEBUGlxc_conf - allocated pty '/dev/pts/6' (8/9) lxc-start 1350808610.403 DEBUGlxc_conf - allocated pty '/dev/pts/7' (10/11) lxc-start 1350808610.403 INFO lxc_conf - tty's configured lxc-start 1350808610.403 DEBUGlxc_console - using '/vmdata/ol6vmserver/dev/console' as console lxc-start 1350808610.403 DEBUGlxc_start - sigchild handler set lxc-start 1350808610.403 INFO lxc_start - 'ol6vmserver' is initialized lxc-start 1350808610.408 DEBUGlxc_conf - instanciated veth 'vethMePv5x/veth2e4SEr', index is '26' lxc-start 1350808610.412 DEBUGlxc_cgroup - checking '/' (rootfs) lxc-start 1350808610.412 DEBUGlxc_cgroup - checking '/proc' (proc) lxc-start 1350808610.412 DEBUGlxc_cgroup - checking '/sys' (sysfs) lxc-start 1350808610.412 DEBUGlxc_cgroup - checking '/dev' (devtmpfs) lxc-start 1350808610.412 DEBUGlxc_cgroup - checking '/dev/pts' (devpts) lxc-start 1350808610.412 DEBUGlxc_cgroup - checking '/dev/shm' (tmpfs) lxc-start 1350808610.412 DEBUGlxc_cgroup - checking '/' (btrfs) lxc-start 1350808610.413 DEBUGlxc_cgroup - checking '/proc/bus/usb' (usbfs) lxc-start 1350808610.413 DEBUGlxc_cgroup - checking '/boot' (ext4) lxc-start 1350808610.413 DEBUGlxc_cgroup - checking '/proc/sys/fs/binfmt_misc' (binfmt_misc) lxc-start 1350808610.413 DEBUGlxc_cgroup - checking '/cgroup/cpuset' (cgroup) lxc-start 1350808610.413 INFO lxc_cgroup - found cgroup mounted at '/cgroup/cpuset' lxc-start 1350808610.413 DEBUGlxc_cgroup - cgroup /cgroup/cpuset has flags 0x2 lxc-start 1350808610.419 INFO lxc_cgroup - created cgroup '/cgroup/cpuset/ol6vmserver' lxc-start 1350808610.419 DEBUGlxc_cgroup - checking '/cgroup/cpu' (cgroup) lxc-start 1350808610.419 INFO lxc_cgroup - found cgroup mounted at '/cgroup/cpu' lxc-start 1350808610.419 DEBUGlxc_cgroup - cgroup /cgroup/cpu has flags 0x2 lxc-start 1350808610.423 INFO lxc_cgroup - created cgroup '/cgroup/cpu/ol6vmserver' lxc-start 1350808610.423 DEBUGlxc_cgroup - checking '/cgroup/cpuacct' (cgroup) lxc-start 1350808610.423 INFO lxc_cgroup - found cgroup mounted at '/cgroup/cpuacct' lxc-start 1350808610.423 DEBUGlxc_cgroup - cgroup /cgroup/cpuacct has flags 0x2 lxc-start 1350808610.426 INFO lxc_cgroup - created cgroup '/cgroup/cpuacct/ol6vmserver' lxc-start 1350808610.427 DEBUGlxc_cgroup - checking '/cgroup/memory' (cgroup) lxc-start 1350808610.427 INFO lxc_cgroup - found cgroup mounted at '/cgroup/memory' lxc-start 1350808610.427 DEBUGlxc_cgroup - cgroup /cgroup/memory has flags 0x2 lxc-start 1350808610.432 INFO lxc_cgroup - created cgroup '/cgroup/memory/ol6vmserver' lxc-start 1350808610.432 DEBUGlxc_cgroup - checking '/cgroup/devices' (cgroup) lxc-start 1350808610.432 INFO lxc_cgroup - found cgroup mounted at '/cgroup/devices' lxc-start 1350808610.432 DEBUGlxc_cgroup - cgroup /cgroup/devices has flags 0x2 lxc-start 1350808610.436 INFO lxc_cgroup - created cgroup '/cgroup/devices/ol6vmserver' lxc-start 1350808610.436 DEBUGlxc_cgroup - checking '/cgroup/freezer' (cgroup) lxc-start 1350808610.436 INFO lxc_cgroup - found cgroup mounted at '/cgroup/freezer' lxc-start 1350808610.436 DEBUGlxc_cgroup - cgroup /cgroup/freezer has flags 0x2 lxc-start 1350808610.440 INFO lxc_cgroup - created cgroup
Re: [Lxc-users] Problems starting OL6.3 lxc container
On Sun, Oct 21, 2012 at 3:42 PM, C. L. Martinez carlopm...@gmail.com wrote: Hi all, I have setup my first OL6 container but it doesn't starts. How? lxc-start 1350808610.466 WARN lxc_conf - rootfs specified but no console found at '/usr/lib64/lxc/rootfs/dev/console' Does /usr/lib64/lxc/rootfs/dev/console exists? I highly suggest you try my centos template first. It definitely creates that file. If it works for you, modify it for OL. Somebody knows where can it be the problem?? Bad container setup? Missing necessary files? -- Fajar -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Problems starting OL6.3 lxc container
On Sun, Oct 21, 2012 at 3:46 PM, Fajar A. Nugraha l...@fajar.net wrote: lxc-start 1350808610.466 WARN lxc_conf - rootfs specified but no console found at '/usr/lib64/lxc/rootfs/dev/console' Does /usr/lib64/lxc/rootfs/dev/console exists? Sorry. It should be does /dev/console exists under your container rootfs? Also, you might not need this line: lxc.console = /vmdata/ol6vmserver/dev/console In fact, I'd say remove it, and see if it solves your problem. -- Fajar -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Problems starting OL6.3 lxc container
On Sun, Oct 21, 2012 at 8:51 AM, Fajar A. Nugraha l...@fajar.net wrote: On Sun, Oct 21, 2012 at 3:46 PM, Fajar A. Nugraha l...@fajar.net wrote: lxc-start 1350808610.466 WARN lxc_conf - rootfs specified but no console found at '/usr/lib64/lxc/rootfs/dev/console' Does /usr/lib64/lxc/rootfs/dev/console exists? Sorry. It should be does /dev/console exists under your container rootfs? Also, you might not need this line: lxc.console = /vmdata/ol6vmserver/dev/console In fact, I'd say remove it, and see if it solves your problem. -- No, problem continues ... I have used this template to create my lxc container: https://github.com/lxc/lxc/blob/staging/templates/lxc-oracle.in -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Problems starting OL6.3 lxc container
On Sun, Oct 21, 2012 at 4:14 PM, C. L. Martinez carlopm...@gmail.com wrote: On Sun, Oct 21, 2012 at 8:51 AM, Fajar A. Nugraha l...@fajar.net wrote: On Sun, Oct 21, 2012 at 3:46 PM, Fajar A. Nugraha l...@fajar.net wrote: lxc-start 1350808610.466 WARN lxc_conf - rootfs specified but no console found at '/usr/lib64/lxc/rootfs/dev/console' Does /usr/lib64/lxc/rootfs/dev/console exists? Sorry. It should be does /dev/console exists under your container rootfs? Also, you might not need this line: lxc.console = /vmdata/ol6vmserver/dev/console In fact, I'd say remove it, and see if it solves your problem. -- No, problem continues ... I have used this template to create my lxc container: In that I says use the unmodified config file first. For example, it says lxc.devttydir = lxc (which you commented out). If you HAVE used the default config file created by the template, but it still doesn't work, you should probably contact the template creator directly (it's on top of the template file) and ask them how to use the template. -- Fajar -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Problems starting OL6.3 lxc container
On Sun, Oct 21, 2012 at 9:20 AM, Fajar A. Nugraha l...@fajar.net wrote: -- No, problem continues ... I have used this template to create my lxc container: In that I says use the unmodified config file first. For example, it says lxc.devttydir = lxc (which you commented out). If you HAVE used the default config file created by the template, but it still doesn't work, you should probably contact the template creator directly (it's on top of the template file) and ask them how to use the template. -- Fajar Yes, I have commented out because when I launch lxc-start, returns me this error: lxc-start 1350810587.498 ERRORlxc_confile - unknow key lxc.devttydir lxc-start 1350810587.498 ERRORlxc_start_ui - failed to read configuration file Thanks. -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Problems starting OL6.3 lxc container
On Sun, Oct 21, 2012 at 4:23 PM, C. L. Martinez carlopm...@gmail.com wrote: On Sun, Oct 21, 2012 at 9:20 AM, Fajar A. Nugraha l...@fajar.net wrote: -- No, problem continues ... I have used this template to create my lxc container: In that I says use the unmodified config file first. For example, it says lxc.devttydir = lxc (which you commented out). If you HAVE used the default config file created by the template, but it still doesn't work, you should probably contact the template creator directly (it's on top of the template file) and ask them how to use the template. -- Fajar Yes, I have commented out because when I launch lxc-start, returns me this error: lxc-start 1350810587.498 ERRORlxc_confile - unknow key lxc.devttydir lxc-start 1350810587.498 ERRORlxc_start_ui - failed to read configuration file Looks like an old version problem. Did you know that the staging git repo on github is newer than released lxc version? I wouldn't be surprised if you need to recompile lxc -- using sources from that repo --- to get the template to work. Personally I just use Ubuntu as the host :) It already supports devttydir configuration item. -- Fajar -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Problems starting OL6.3 lxc container
On Sun, Oct 21, 2012 at 9:27 AM, Fajar A. Nugraha l...@fajar.net wrote: On Sun, Oct 21, 2012 at 4:23 PM, C. L. Martinez carlopm...@gmail.com wrote: On Sun, Oct 21, 2012 at 9:20 AM, Fajar A. Nugraha l...@fajar.net wrote: -- No, problem continues ... I have used this template to create my lxc container: In that I says use the unmodified config file first. For example, it says lxc.devttydir = lxc (which you commented out). If you HAVE used the default config file created by the template, but it still doesn't work, you should probably contact the template creator directly (it's on top of the template file) and ask them how to use the template. -- Thanks Fajar, I will try to use centos6 instead of OL6 ... Are these your instructions?? http://wiki.1tux.org/wiki/Centos6/Installation/Minimal_installation_using_yum http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6 -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] Problems starting OL6.3 lxc container
On Sun, Oct 21, 2012 at 4:41 PM, C. L. Martinez carlopm...@gmail.com wrote: If you HAVE used the default config file created by the template, but it still doesn't work, you should probably contact the template creator directly (it's on top of the template file) and ask them how to use the template. -- Thanks Fajar, I will try to use centos6 instead of OL6 ... Are these your instructions?? http://wiki.1tux.org/wiki/Centos6/Installation/Minimal_installation_using_yum http://wiki.1tux.org/wiki/Lxc/Installation/Guest/Centos/6 Yes, those are the manual way of creating them. You can also try the centos template from the link I sent earlier, rename it as lxc-centos, chmod 755, and put it on your templates directory (usually /usr/lib/lxc/templates). Tested on Ubuntu host, should work for other hosts as well. -- Fajar -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] [lxc-attach error] Failed to open /proc/4468/ns/pid. Failed to enter namespace
Laptop (Linux 3.2.0-25-generic x86_64) $ ls /proc/self/ns ipc net uts Desktop (Linux 3.5.0-17-generic x86_64) $ ls /proc/self/ns ipc net uts Do you have links to the kernel sources + patches that I need in for lxc-attach to work? Also, where is the official lxc repo located? Github, sourceforge? Cheers, Frank On Tue, Oct 16, 2012 at 5:48 PM, Serge Hallyn serge.hal...@canonical.com wrote: Quoting Frank Scholten (fr...@frankscholten.nl): frank@franktop:~$ uname -a Linux franktop 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux I thought setns was added in 3.0? Not for pid ns. do 'ls /proc/self/ns' to see the list of namespaces to which you can setns. I also have the setns man page. How do I enable it? Do I have to compile a new kernel? Cheers, Frank On Mon, Oct 15, 2012 at 6:25 PM, Serge Hallyn serge.hal...@canonical.com wrote: Quoting Frank Scholten (fr...@frankscholten.nl): Hi all, I am trying to run commands inside the container. Running lxc 0.7.5.1, commit 60a742e0afd from sourceforge. I created and started an Ubuntu container and when I run $ sudo lxc-attach -e -n test whoami I get 'Failed to open /proc/4468/ns/pid. Failed to enter namespace' Any idea what could be wrong? Your kernel does not support setns for pid. /proc/$$/ns/pid does not exist. -serge -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On 19/10/12 16:51, Serge Hallyn wrote: Add: lxc.network.type = empty If you don't have any lxc.network.type sections, then the container shares network with the host, and so the container talks to the host's systemd. (same with upstart) Thanks for the reply, I will try that tomorrow. I am sorry I wasn't around to check for replies before now. One question though... I actually want a separate network in the container (hence using veth) so it has its own address distinct from the host. Are you saying that I can't do this any more? I've also read the later replies and they seem to be saying that this simply does not work (systemd inside a container). Given its proliferation into other distros (I'm on Arch and that's the reason I am looking at this now), where does systemd come in the priorities of LXC? I really hope we can get this working, as LXC has so far worked very well for me. Thanks, John -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
Quoting Michael H. Warfield (m...@wittsend.com):
Serge,
I'm going to top post here simply because this is going to go off in a
different direction and bringing in an old thread but it is related...
Back on February 14 you responded to a message about Fedora 16 in a
container, which is something I've been trying to do and I had run into
that posters problems. You responded with this:
Subject: Re: [Lxc-users] fedora 16 under lxc
On Tue, 2012-02-14 at 09:23 -0600, Serge Hallyn wrote:
Quoting Ramez Hanna (rha...@informatiq.org):
now all my efforts have not succeedd to get getty on tty1 to start
unmasking udev did something different
it created all the /dev devices
and made getty start but it started on the hosts's tty not on the
container's
could someone shed some light here?
Blind guess:
lxc-start creates some ptys and bind mounts them onto the guest's
/dev/{console,tty{1,2,3,4}}. It sounds like fedora's init is mounting
over the /dev set up by lxc causing a new /dev/tty to be created as
chardev 4:{1-4}. Devices namespaces would help this. We're hoping to
discuss design for those at next UDS, but those will come after user
namespaces. In the mean time, you'll need to make sure that the guest
does not mount over /dev, and does not remount /dev/pts.
-serge
That got me thinking and started into looking deeper into systemd, which
Fedora 16 and above uses and why it may be related here. I've made
Fedora 16 work in the past by installing upstart and disabling systemd
but that really becomes impractical in Fedora 17 because they're
including so few of the compatibility scripts. Yes, you are right, the
Fedora's init (systemd) is mounting something on /dev like this:
devtmpfs on /dev type devtmpfs
(rw,nosuid,seclabel,size=1784160k,nr_inodes=446040,mode=755)
This is very bad for the reasons you pointed out in Feb. Looking at the
source code for systemd, this is hard coded into the binary and is not
configurable.
systemd-37/src/mount-setup.c:
--
/* The first three entries we might need before SELinux is up. The
* other ones we can delay until SELinux is loaded. */
#define N_EARLY_MOUNT 3
static const MountPoint mount_table[] = {
{ proc, /proc, proc, NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ sysfs,/sys, sysfs,NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ devtmpfs, /dev, devtmpfs, mode=755,
MS_NOSUID,true },
{ tmpfs,/dev/shm, tmpfs,mode=1777,
MS_NOSUID|MS_NODEV, true },
{ devpts, /dev/pts, devpts, mode=620,gid=
STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
{ tmpfs,/run, tmpfs,mode=755,
MS_NOSUID|MS_NODEV, true },
{ tmpfs,/sys/fs/cgroup, tmpfs,mode=755,
MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
{ cgroup, /sys/fs/cgroup/systemd, cgroup,
none,name=systemd, MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
};
--
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
-serge
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
Lxc-users mailing list
Lxc-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
Quoting John (l...@jelmail.com): On 19/10/12 16:51, Serge Hallyn wrote: Add: lxc.network.type = empty If you don't have any lxc.network.type sections, then the container shares network with the host, and so the container talks to the host's systemd. (same with upstart) Thanks for the reply, I will try that tomorrow. I am sorry I wasn't around to check for replies before now. One question though... I actually want a separate network in the container (hence using veth) so it has its own address distinct from the host. Are you saying that I can't do this any more? Not at all. But if you're saying you have a 'lxc.network.type = veth' in your container config, then what I said doesn't apply anyway. It sounds like the remount of /dev which Micheal mentioned is in fact your real problem! I've also read the later replies and they seem to be saying that this simply does not work (systemd inside a container). Given its proliferation into other distros (I'm on Arch and that's the reason I am looking at this now), where does systemd come in the priorities of LXC? Where does LXC come in the priorities of systemd? :) (my point being that it might be far easier to patch systemd to make the filesystems to mount configurable, versus implementing a devices namespace in the kernel so that lxc can work around it) But, lxc is open source, as is the kernel (and systemd) - when you send patches, your priorities influence its priorities. I really hope we can get this working, as LXC has so far worked very well for me. -serge -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
[Lxc-users] Unable to run systemd in an LXC / cgroup container.
Hello, This is being directed to the systemd-devel community but I'm cc'ing the lxc-users community and the Fedora community on this for their input as well. I know it's not always good to cross post between multiple lists but this is of interest to all three communities who may have valuable input. I'm new to this particular list, just having joined after tracking a problem down to some systemd internals... Several people over the last year or two on the lxc-users list have been discussions trying to run certain distros (notably Fedora 16 and above, recent Arch Linux and possibly others) in LXC containers, virualizing entire servers this way. This is very similar to Virtuoso / OpenVZ only it's using the native Linux cgroups for the containers (primary reason I dumped OpenVZ was to avoid their custom patched kernels). These recent distros have switched to systemd for the main init process and this has proven to be disastrous for those of us using LXC and trying to install or update our containers. To put it bluntly, it doesn't work and causes all sorts of problems on the host. To summarize the problem... The LXC startup binary sets up various things for /dev and /dev/pts for the container to run properly and this works perfectly fine for SystemV start-up scripts and/or Upstart. Unfortunately, systemd has mounts of devtmpfs on /dev and devpts on /dev/pts which then break things horribly. This is because the kernel currently lacks namespaces for devices and won't for some time to come (in design). When devtmpfs gets mounted over top of /dev in the container, it then hijacks the hosts console tty and several other devices which had been set up through bind mounts by LXC and should have been LEFT ALONE. Yes! I recognize that this problem with devtmpfs and lack of namespaces is a potential security problem anyways that could (and does) cause serious container-to-host problems. We're just not going to get that fixed right away in the linux cgroups and namespaces. How do we work around this problem in systemd where it has hard coded mounts in the binary that we can't override or configure? Or is it there and I'm just missing it trying to examine the sources? That's how I found where the problem lay. Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On Sun, 2012-10-21 at 14:49 -0500, Serge Hallyn wrote:
Quoting Michael H. Warfield (m...@wittsend.com):
Serge,
I'm going to top post here simply because this is going to go off in a
different direction and bringing in an old thread but it is related...
Back on February 14 you responded to a message about Fedora 16 in a
container, which is something I've been trying to do and I had run into
that posters problems. You responded with this:
Subject: Re: [Lxc-users] fedora 16 under lxc
On Tue, 2012-02-14 at 09:23 -0600, Serge Hallyn wrote:
Quoting Ramez Hanna (rha...@informatiq.org):
now all my efforts have not succeedd to get getty on tty1 to start
unmasking udev did something different
it created all the /dev devices
and made getty start but it started on the hosts's tty not on the
container's
could someone shed some light here?
Blind guess:
lxc-start creates some ptys and bind mounts them onto the guest's
/dev/{console,tty{1,2,3,4}}. It sounds like fedora's init is mounting
over the /dev set up by lxc causing a new /dev/tty to be created as
chardev 4:{1-4}. Devices namespaces would help this. We're hoping to
discuss design for those at next UDS, but those will come after user
namespaces. In the mean time, you'll need to make sure that the guest
does not mount over /dev, and does not remount /dev/pts.
-serge
That got me thinking and started into looking deeper into systemd, which
Fedora 16 and above uses and why it may be related here. I've made
Fedora 16 work in the past by installing upstart and disabling systemd
but that really becomes impractical in Fedora 17 because they're
including so few of the compatibility scripts. Yes, you are right, the
Fedora's init (systemd) is mounting something on /dev like this:
devtmpfs on /dev type devtmpfs
(rw,nosuid,seclabel,size=1784160k,nr_inodes=446040,mode=755)
This is very bad for the reasons you pointed out in Feb. Looking at the
source code for systemd, this is hard coded into the binary and is not
configurable.
systemd-37/src/mount-setup.c:
--
/* The first three entries we might need before SELinux is up. The
* other ones we can delay until SELinux is loaded. */
#define N_EARLY_MOUNT 3
static const MountPoint mount_table[] = {
{ proc, /proc, proc, NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ sysfs,/sys, sysfs,NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ devtmpfs, /dev, devtmpfs, mode=755,
MS_NOSUID,true },
{ tmpfs,/dev/shm, tmpfs,mode=1777,
MS_NOSUID|MS_NODEV, true },
{ devpts, /dev/pts, devpts, mode=620,gid=
STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
{ tmpfs,/run, tmpfs,mode=755,
MS_NOSUID|MS_NODEV, true },
{ tmpfs,/sys/fs/cgroup, tmpfs,mode=755,
MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
{ cgroup, /sys/fs/cgroup/systemd, cgroup,
none,name=systemd, MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
};
--
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
I'm not sure how that would work or if that would work in the case where
you didn't have selinux in the host kernel or you were crossing apparmor
in the container and selinux in the host or vice-versa.
In any case, I'm hitting the systemd-devel list looking to raise their
awareness of the problem and including this list and the fedora list.
If someone wants to mention it on the Arch Linux list, please do, I
don't participate over there.
-serge
Thanks
Regards,
Mike
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
Lxc-users mailing list
Lxc-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
--
Michael H. Warfield
Re: [Lxc-users] [systemd-devel] Unable to run systemd in an LXC / cgroup container.
On Mon, 2012-10-22 at 02:53 +0200, Kay Sievers wrote: On Sun, Oct 21, 2012 at 11:25 PM, Michael H. Warfield m...@wittsend.com wrote: This is being directed to the systemd-devel community but I'm cc'ing the lxc-users community and the Fedora community on this for their input as well. I know it's not always good to cross post between multiple lists but this is of interest to all three communities who may have valuable input. I'm new to this particular list, just having joined after tracking a problem down to some systemd internals... Several people over the last year or two on the lxc-users list have been discussions trying to run certain distros (notably Fedora 16 and above, recent Arch Linux and possibly others) in LXC containers, virualizing entire servers this way. This is very similar to Virtuoso / OpenVZ only it's using the native Linux cgroups for the containers (primary reason I dumped OpenVZ was to avoid their custom patched kernels). These recent distros have switched to systemd for the main init process and this has proven to be disastrous for those of us using LXC and trying to install or update our containers. To put it bluntly, it doesn't work and causes all sorts of problems on the host. To summarize the problem... The LXC startup binary sets up various things for /dev and /dev/pts for the container to run properly and this works perfectly fine for SystemV start-up scripts and/or Upstart. Unfortunately, systemd has mounts of devtmpfs on /dev and devpts on /dev/pts which then break things horribly. This is because the kernel currently lacks namespaces for devices and won't for some time to come (in design). When devtmpfs gets mounted over top of /dev in the container, it then hijacks the hosts console tty and several other devices which had been set up through bind mounts by LXC and should have been LEFT ALONE. Yes! I recognize that this problem with devtmpfs and lack of namespaces is a potential security problem anyways that could (and does) cause serious container-to-host problems. We're just not going to get that fixed right away in the linux cgroups and namespaces. How do we work around this problem in systemd where it has hard coded mounts in the binary that we can't override or configure? Or is it there and I'm just missing it trying to examine the sources? That's how I found where the problem lay. As a first step, this probably explains most of it: http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface A very long ways, yeah. That looks like it could be just what we've been looking for. Just gotta figure out how to set that environment variable but that's up to a couple of others to comment on in the lxc-users list. Then we'll see where we go from there. Many thanks! Kay Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On Sun, 2012-10-21 at 14:49 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Serge, ... Short of building a custom systemd, I don't know how to fix that problem and I suspect this OP is going to run into this same thing (container taking over host's console) and might explain some of what he's seeing. Several of these look like they could cause problems (like /dev/pts in there). I've really reached an impasse at getting systemd (at least Fedora 16 and 17) to work in a container without screwing up the host. Prohibiting mounts entirely in the container might work but I suspect (having read some systemd error messages) systemd is going to have some serious heartburn there. Thoughts? IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the container should work, i.e. systemd was not going to fail as a result. Hopefully, you've seen the message from Kay Sievers cc'ed to this list from my post to the systemd-devel list. Looks like they have a mechanism in place to do this... http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface First step appears to be to set a container=LXC (or some other short string) before invoking init in the container. Is there a mechanism to do this? Might look over the rest of their recommendation and see if there's anything else we need to do. Looks like there might be some additional mounts (some read-only) in there that need to be handled in lxc-start as well. -serge -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
