Re: [Lxc-users] systemd inside LXC
On 10/23/2012 12:05 AM, Michael H. Warfield wrote: On Mon, 2012-10-22 at 16:21 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): On Mon, 2012-10-22 at 15:14 -0500, Serge Hallyn wrote: Trimming some overhead we've seen enough of... How about just a devtmpfs? We actually now do this by default (as of very recently) in ubuntu by adding devtmpfsdev devtmpfs defaults 0 0 NO! That's the problem! That leads to the container connecting to the hosts console and other devices and committing random acts of terrorism. No, it shouldn't, because lxc sets up the console after doing the mounts. Damn, dude! That worked! That kludge rang da bell. Of course, I also discovered the boneheaded typo I had in attempting the tmpfs mount in the process. :-P I now have a container running systemd up and running with Fedora 17 in it. I'm not sure I'm totally happy with it. Because of doing the devtmpfs thing, the guest can immediately see things like removable drives coming and going and might, presumably, be able to mount them. Not thrilled with that from a security standpoint. Would also mean the guests could access things like my permanent forensic CDs that are in the CD drives. I guess that can be restricted in the config but still makes me a bit uncomfortable that the guest has complete visibility into the hosts dev system. That's actually similar to what Ubuntu has had for the past few releases as we're running udevd in the container. Basically all the block devices of the host and any hotplugged device will appear in /dev but our default configuration is to only allow mknoding them, not read or write to them. So the end result is basically the same as if they weren't there to start with, except that for those that are actually allowed, they then behave like they'd on the host by showing up when added and disappearing when removed without any manual interaction. It's not ideal, but it's safe. For the ideal implementation, we'll need to wait for the device namespace. Another gotcha, albeit a much more minor one... When systemd drops into this mode, you no longer have vty consoles available so lxc-console won't work. That's actually on their page. I remember seeing this: -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users -- Stéphane Graber Ubuntu developer http://www.ubuntu.com signature.asc Description: OpenPGP digital signature -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On Mon, 2012-10-22 at 15:14 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Serge, On Mon, 2012-10-22 at 09:12 -0500, Serge Hallyn wrote: Quoting Serge Hallyn (serge.hal...@canonical.com): Quoting Michael H. Warfield (m...@wittsend.com): On Sun, 2012-10-21 at 14:49 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Serge, ... Short of building a custom systemd, I don't know how to fix that problem and I suspect this OP is going to run into this same thing (container taking over host's console) and might explain some of what he's seeing. Several of these look like they could cause problems (like /dev/pts in there). I've really reached an impasse at getting systemd (at least Fedora 16 and 17) to work in a container without screwing up the host. Prohibiting mounts entirely in the container might work but I suspect (having read some systemd error messages) systemd is going to have some serious heartburn there. Thoughts? IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the container should work, i.e. systemd was not going to fail as a result. Hopefully, you've seen the message from Kay Sievers cc'ed to this list from my post to the systemd-devel list. Looks like they have a mechanism in place to do this... http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface Saw the email, haven't yet read the page, thanks. So based on that page, what we do (set 'container=lxc') should already be sufficient. Thanks to the dude asking a libvirt-lxc question on the list, I was let to a page that let to a page that led to some discussion you were having back in March with Ramez Hanna on this very subject, Re: [Lxc-users] f16 update... http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg03263.html thanks, I knew we'd been over some of this, but couldn't find my logs of it. This would look to be the kludge to make a workaround for this problem, I'm just not sure how to make it happen. Given you already found the answer that the device for /dev has to be different than the device for the parent, what should we do. I tried this in the config... lxc.mount.entry=tmpfs /var/lib/lxc/private/Alcove/dev tmpfs defaults 0 0 How about just a devtmpfs? We actually now do this by default (as of very recently) in ubuntu by adding devtmpfsdev devtmpfs defaults 0 0 NO! That's the problem! That leads to the container connecting to the hosts console and other devices and committing random acts of terrorism. That's exactly the case we are trying to avoid and which is causing the problems to begin with. That's what systemd is doing if it doesn't find /dev mounted to begin with. to /var/lib/lxc/$container/fstab Or, are you on an older kernel which doesn't have devtmpfs? Maybe I got that entry wrong but it didn't do anything (and would have resulted in other failures later as I found out). A similar mount entry for a shared filesystem worked just fine. Trying an empty /dev fails because it's missing EVERYTHING so starting out with a tmpfs without populating it is not the answer either. The correct answer would be to mount a tmpfs file system and then populate it before firing up systemd in the container. I don't see how to do that. A bind mount isn't going to work either, for the reasons you stated in that thread, it ends up on the same device. Having another mount on another real device would be a workaround but a really bad kludge that would complicate things immensely. -serge -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On Mon, 2012-10-22 at 16:21 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): On Mon, 2012-10-22 at 15:14 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Serge, On Mon, 2012-10-22 at 09:12 -0500, Serge Hallyn wrote: Quoting Serge Hallyn (serge.hal...@canonical.com): Quoting Michael H. Warfield (m...@wittsend.com): On Sun, 2012-10-21 at 14:49 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Serge, ... Short of building a custom systemd, I don't know how to fix that problem and I suspect this OP is going to run into this same thing (container taking over host's console) and might explain some of what he's seeing. Several of these look like they could cause problems (like /dev/pts in there). I've really reached an impasse at getting systemd (at least Fedora 16 and 17) to work in a container without screwing up the host. Prohibiting mounts entirely in the container might work but I suspect (having read some systemd error messages) systemd is going to have some serious heartburn there. Thoughts? IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the container should work, i.e. systemd was not going to fail as a result. Hopefully, you've seen the message from Kay Sievers cc'ed to this list from my post to the systemd-devel list. Looks like they have a mechanism in place to do this... http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface Saw the email, haven't yet read the page, thanks. So based on that page, what we do (set 'container=lxc') should already be sufficient. Thanks to the dude asking a libvirt-lxc question on the list, I was let to a page that let to a page that led to some discussion you were having back in March with Ramez Hanna on this very subject, Re: [Lxc-users] f16 update... http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg03263.html thanks, I knew we'd been over some of this, but couldn't find my logs of it. This would look to be the kludge to make a workaround for this problem, I'm just not sure how to make it happen. Given you already found the answer that the device for /dev has to be different than the device for the parent, what should we do. I tried this in the config... lxc.mount.entry=tmpfs /var/lib/lxc/private/Alcove/dev tmpfs defaults 0 0 How about just a devtmpfs? We actually now do this by default (as of very recently) in ubuntu by adding devtmpfsdev devtmpfs defaults 0 0 NO! That's the problem! That leads to the container connecting to the hosts console and other devices and committing random acts of terrorism. No, it shouldn't, because lxc sets up the console after doing the mounts. Maybe it shouldn't but that appears to be what is happening and even you remarked that maybe the problem was something doing a remount of /dev after entering the container... I see your point though. If you did that mount after LXC set up the console, then systemd wouldn't set it up and would drop into its more restricted mode. That MIGHT help but you still have the entire dev space of the host exposed to the guest which is what you were talking about before wrt namespaces on devices. It might help. Would it be the answer? Given that we've restricted access to those nodes in the config, maybe yes. I'm just not so sure. Will give it a shot though. Strange, though, my earlier effort at tmpfs on dev had no effect. Will give it a shot. -serge Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
Quoting Michael H. Warfield (m...@wittsend.com): On Mon, 2012-10-22 at 16:21 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): On Mon, 2012-10-22 at 15:14 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Serge, On Mon, 2012-10-22 at 09:12 -0500, Serge Hallyn wrote: Quoting Serge Hallyn (serge.hal...@canonical.com): Quoting Michael H. Warfield (m...@wittsend.com): On Sun, 2012-10-21 at 14:49 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Serge, ... Short of building a custom systemd, I don't know how to fix that problem and I suspect this OP is going to run into this same thing (container taking over host's console) and might explain some of what he's seeing. Several of these look like they could cause problems (like /dev/pts in there). I've really reached an impasse at getting systemd (at least Fedora 16 and 17) to work in a container without screwing up the host. Prohibiting mounts entirely in the container might work but I suspect (having read some systemd error messages) systemd is going to have some serious heartburn there. Thoughts? IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the container should work, i.e. systemd was not going to fail as a result. Hopefully, you've seen the message from Kay Sievers cc'ed to this list from my post to the systemd-devel list. Looks like they have a mechanism in place to do this... http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface Saw the email, haven't yet read the page, thanks. So based on that page, what we do (set 'container=lxc') should already be sufficient. Thanks to the dude asking a libvirt-lxc question on the list, I was let to a page that let to a page that led to some discussion you were having back in March with Ramez Hanna on this very subject, Re: [Lxc-users] f16 update... http://www.mail-archive.com/lxc-users@lists.sourceforge.net/msg03263.html thanks, I knew we'd been over some of this, but couldn't find my logs of it. This would look to be the kludge to make a workaround for this problem, I'm just not sure how to make it happen. Given you already found the answer that the device for /dev has to be different than the device for the parent, what should we do. I tried this in the config... lxc.mount.entry=tmpfs /var/lib/lxc/private/Alcove/dev tmpfs defaults 0 0 How about just a devtmpfs? We actually now do this by default (as of very recently) in ubuntu by adding devtmpfsdev devtmpfs defaults 0 0 NO! That's the problem! That leads to the container connecting to the hosts console and other devices and committing random acts of terrorism. No, it shouldn't, because lxc sets up the console after doing the mounts. Maybe it shouldn't but that appears to be what is happening and even you remarked that maybe the problem was something doing a remount of /dev after entering the container... I see your point though. If you did that mount after LXC set up the console, then systemd wouldn't set it up and would drop into its more restricted mode. That MIGHT help but you still have the entire dev space of the host exposed to the guest which is what you were talking about before wrt namespaces on devices. It might help. Would it be the answer? Given that we've restricted access to those nodes in the config, maybe yes. I'm just not so sure. Will give it a shot though. Strange, though, my earlier effort at tmpfs on dev had no effect. Will give it a shot. Thanks again for looking into this - I know just how frustrating it can be! :) -serge -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On Mon, 2012-10-22 at 16:21 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): On Mon, 2012-10-22 at 15:14 -0500, Serge Hallyn wrote: Trimming some overhead we've seen enough of... How about just a devtmpfs? We actually now do this by default (as of very recently) in ubuntu by adding devtmpfsdev devtmpfs defaults 0 0 NO! That's the problem! That leads to the container connecting to the hosts console and other devices and committing random acts of terrorism. No, it shouldn't, because lxc sets up the console after doing the mounts. Damn, dude! That worked! That kludge rang da bell. Of course, I also discovered the boneheaded typo I had in attempting the tmpfs mount in the process. :-P I now have a container running systemd up and running with Fedora 17 in it. I'm not sure I'm totally happy with it. Because of doing the devtmpfs thing, the guest can immediately see things like removable drives coming and going and might, presumably, be able to mount them. Not thrilled with that from a security standpoint. Would also mean the guests could access things like my permanent forensic CDs that are in the CD drives. I guess that can be restricted in the config but still makes me a bit uncomfortable that the guest has complete visibility into the hosts dev system. Another gotcha, albeit a much more minor one... When systemd drops into this mode, you no longer have vty consoles available so lxc-console won't work. That's actually on their page. I remember seeing this: -- If systemd detects it is run in a container it will spawn a single shell on /dev/console, and not care about VTs or multiple gettys on VTs -- Suboptimal but a small price to pay I suppose. -serge Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On Mon, 2012-10-22 at 18:05 -0400, Michael H. Warfield wrote: On Mon, 2012-10-22 at 16:21 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): On Mon, 2012-10-22 at 15:14 -0500, Serge Hallyn wrote: Trimming some overhead we've seen enough of... How about just a devtmpfs? We actually now do this by default (as of very recently) in ubuntu by adding devtmpfsdev devtmpfs defaults 0 0 NO! That's the problem! That leads to the container connecting to the hosts console and other devices and committing random acts of terrorism. No, it shouldn't, because lxc sets up the console after doing the mounts. Damn, dude! That worked! That kludge rang da bell. Of course, I also discovered the boneheaded typo I had in attempting the tmpfs mount in the process. :-P I now have a container running systemd up and running with Fedora 17 in it. I'm not sure I'm totally happy with it. Because of doing the devtmpfs thing, the guest can immediately see things like removable drives coming and going and might, presumably, be able to mount them. Not thrilled with that from a security standpoint. Would also mean the guests could access things like my permanent forensic CDs that are in the CD drives. I guess that can be restricted in the config but still makes me a bit uncomfortable that the guest has complete visibility into the hosts dev system. Another downside. Container does not shut down cleanly... init 0 inside the container... In lxc-start - Unmounting file systems. Could not remount as read-only /: Device or resource busy Not all file systems unmounted, 1 left. Detaching loop devices. Could not delete loopback /dev/loop7: Operation not permitted Could not delete loopback /dev/loop6: Operation not permitted Could not delete loopback /dev/loop5: Operation not permitted Could not delete loopback /dev/loop4: Operation not permitted Could not delete loopback /dev/loop3: Operation not permitted Could not delete loopback /dev/loop2: Operation not permitted Could not delete loopback /dev/loop1: Operation not permitted Could not delete loopback /dev/loop0: Operation not permitted Not all loop devices detached, 8 left. Cannot finalize remaining file systems and devices, giving up. Exiting container. lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/systemd/Alcove' Not good. The tasks file is empty but... Can't get rid of it. Operation not permitted. Sigh... Getting closer though. Much closer. Another gotcha, albeit a much more minor one... When systemd drops into this mode, you no longer have vty consoles available so lxc-console won't work. That's actually on their page. I remember seeing this: -- If systemd detects it is run in a container it will spawn a single shell on /dev/console, and not care about VTs or multiple gettys on VTs -- Suboptimal but a small price to pay I suppose. -serge Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On Mon, 2012-10-22 at 18:37 -0400, Michael H. Warfield wrote:
On Mon, 2012-10-22 at 18:05 -0400, Michael H. Warfield wrote:
On Mon, 2012-10-22 at 16:21 -0500, Serge Hallyn wrote:
Quoting Michael H. Warfield (m...@wittsend.com):
On Mon, 2012-10-22 at 15:14 -0500, Serge Hallyn wrote:
Trimming some overhead we've seen enough of...
How about just a devtmpfs? We actually now do this by default (as of
very
recently) in ubuntu by adding
devtmpfsdev devtmpfs defaults 0 0
NO! That's the problem! That leads to the container connecting to the
hosts console and other devices and committing random acts of terrorism.
No, it shouldn't, because lxc sets up the console after doing the mounts.
Damn, dude! That worked! That kludge rang da bell. Of course, I also
discovered the boneheaded typo I had in attempting the tmpfs mount in
the process. :-P I now have a container running systemd up and running
with Fedora 17 in it.
I'm not sure I'm totally happy with it.
Because of doing the devtmpfs thing, the guest can immediately see
things like removable drives coming and going and might, presumably, be
able to mount them. Not thrilled with that from a security standpoint.
Would also mean the guests could access things like my permanent
forensic CDs that are in the CD drives. I guess that can be restricted
in the config but still makes me a bit uncomfortable that the guest has
complete visibility into the hosts dev system.
Another downside. Container does not shut down cleanly...
And another...
Container seems to hang if lxc-start is run in disconnected mode
(lxc-start -d -o {log}). Starts up fine with a console that's connected
to pty's but not to a log it seems...
init 0 inside the container...
In lxc-start -
Unmounting file systems.
Could not remount as read-only /: Device or resource busy
Not all file systems unmounted, 1 left.
Detaching loop devices.
Could not delete loopback /dev/loop7: Operation not permitted
Could not delete loopback /dev/loop6: Operation not permitted
Could not delete loopback /dev/loop5: Operation not permitted
Could not delete loopback /dev/loop4: Operation not permitted
Could not delete loopback /dev/loop3: Operation not permitted
Could not delete loopback /dev/loop2: Operation not permitted
Could not delete loopback /dev/loop1: Operation not permitted
Could not delete loopback /dev/loop0: Operation not permitted
Not all loop devices detached, 8 left.
Cannot finalize remaining file systems and devices, giving up.
Exiting container.
lxc-start: Device or resource busy - failed to remove cgroup
'/sys/fs/cgroup/systemd/Alcove'
Not good. The tasks file is empty but... Can't get rid of it.
Operation not permitted.
Sigh...
Getting closer though. Much closer.
Another gotcha, albeit a much more minor one... When systemd drops into
this mode, you no longer have vty consoles available so lxc-console
won't work. That's actually on their page.
I remember seeing this:
--
If systemd detects it is run in a container it will spawn a single shell
on /dev/console, and not care about VTs or multiple gettys on VTs
--
Suboptimal but a small price to pay I suppose.
-serge
Regards,
Mike
--
Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com
/\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/
NIC whois: MHW9 | An optimist believes we live in the best of all
PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it!
signature.asc
Description: This is a digitally signed message part
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct___
Lxc-users mailing list
Lxc-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On 19/10/12 16:51, Serge Hallyn wrote: Add: lxc.network.type = empty If you don't have any lxc.network.type sections, then the container shares network with the host, and so the container talks to the host's systemd. (same with upstart) Thanks for the reply, I will try that tomorrow. I am sorry I wasn't around to check for replies before now. One question though... I actually want a separate network in the container (hence using veth) so it has its own address distinct from the host. Are you saying that I can't do this any more? I've also read the later replies and they seem to be saying that this simply does not work (systemd inside a container). Given its proliferation into other distros (I'm on Arch and that's the reason I am looking at this now), where does systemd come in the priorities of LXC? I really hope we can get this working, as LXC has so far worked very well for me. Thanks, John -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
Quoting Michael H. Warfield (m...@wittsend.com):
Serge,
I'm going to top post here simply because this is going to go off in a
different direction and bringing in an old thread but it is related...
Back on February 14 you responded to a message about Fedora 16 in a
container, which is something I've been trying to do and I had run into
that posters problems. You responded with this:
Subject: Re: [Lxc-users] fedora 16 under lxc
On Tue, 2012-02-14 at 09:23 -0600, Serge Hallyn wrote:
Quoting Ramez Hanna (rha...@informatiq.org):
now all my efforts have not succeedd to get getty on tty1 to start
unmasking udev did something different
it created all the /dev devices
and made getty start but it started on the hosts's tty not on the
container's
could someone shed some light here?
Blind guess:
lxc-start creates some ptys and bind mounts them onto the guest's
/dev/{console,tty{1,2,3,4}}. It sounds like fedora's init is mounting
over the /dev set up by lxc causing a new /dev/tty to be created as
chardev 4:{1-4}. Devices namespaces would help this. We're hoping to
discuss design for those at next UDS, but those will come after user
namespaces. In the mean time, you'll need to make sure that the guest
does not mount over /dev, and does not remount /dev/pts.
-serge
That got me thinking and started into looking deeper into systemd, which
Fedora 16 and above uses and why it may be related here. I've made
Fedora 16 work in the past by installing upstart and disabling systemd
but that really becomes impractical in Fedora 17 because they're
including so few of the compatibility scripts. Yes, you are right, the
Fedora's init (systemd) is mounting something on /dev like this:
devtmpfs on /dev type devtmpfs
(rw,nosuid,seclabel,size=1784160k,nr_inodes=446040,mode=755)
This is very bad for the reasons you pointed out in Feb. Looking at the
source code for systemd, this is hard coded into the binary and is not
configurable.
systemd-37/src/mount-setup.c:
--
/* The first three entries we might need before SELinux is up. The
* other ones we can delay until SELinux is loaded. */
#define N_EARLY_MOUNT 3
static const MountPoint mount_table[] = {
{ proc, /proc, proc, NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ sysfs,/sys, sysfs,NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ devtmpfs, /dev, devtmpfs, mode=755,
MS_NOSUID,true },
{ tmpfs,/dev/shm, tmpfs,mode=1777,
MS_NOSUID|MS_NODEV, true },
{ devpts, /dev/pts, devpts, mode=620,gid=
STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
{ tmpfs,/run, tmpfs,mode=755,
MS_NOSUID|MS_NODEV, true },
{ tmpfs,/sys/fs/cgroup, tmpfs,mode=755,
MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
{ cgroup, /sys/fs/cgroup/systemd, cgroup,
none,name=systemd, MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
};
--
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
-serge
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
Lxc-users mailing list
Lxc-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
Quoting John (l...@jelmail.com): On 19/10/12 16:51, Serge Hallyn wrote: Add: lxc.network.type = empty If you don't have any lxc.network.type sections, then the container shares network with the host, and so the container talks to the host's systemd. (same with upstart) Thanks for the reply, I will try that tomorrow. I am sorry I wasn't around to check for replies before now. One question though... I actually want a separate network in the container (hence using veth) so it has its own address distinct from the host. Are you saying that I can't do this any more? Not at all. But if you're saying you have a 'lxc.network.type = veth' in your container config, then what I said doesn't apply anyway. It sounds like the remount of /dev which Micheal mentioned is in fact your real problem! I've also read the later replies and they seem to be saying that this simply does not work (systemd inside a container). Given its proliferation into other distros (I'm on Arch and that's the reason I am looking at this now), where does systemd come in the priorities of LXC? Where does LXC come in the priorities of systemd? :) (my point being that it might be far easier to patch systemd to make the filesystems to mount configurable, versus implementing a devices namespace in the kernel so that lxc can work around it) But, lxc is open source, as is the kernel (and systemd) - when you send patches, your priorities influence its priorities. I really hope we can get this working, as LXC has so far worked very well for me. -serge -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
On Sun, 2012-10-21 at 14:49 -0500, Serge Hallyn wrote:
Quoting Michael H. Warfield (m...@wittsend.com):
Serge,
I'm going to top post here simply because this is going to go off in a
different direction and bringing in an old thread but it is related...
Back on February 14 you responded to a message about Fedora 16 in a
container, which is something I've been trying to do and I had run into
that posters problems. You responded with this:
Subject: Re: [Lxc-users] fedora 16 under lxc
On Tue, 2012-02-14 at 09:23 -0600, Serge Hallyn wrote:
Quoting Ramez Hanna (rha...@informatiq.org):
now all my efforts have not succeedd to get getty on tty1 to start
unmasking udev did something different
it created all the /dev devices
and made getty start but it started on the hosts's tty not on the
container's
could someone shed some light here?
Blind guess:
lxc-start creates some ptys and bind mounts them onto the guest's
/dev/{console,tty{1,2,3,4}}. It sounds like fedora's init is mounting
over the /dev set up by lxc causing a new /dev/tty to be created as
chardev 4:{1-4}. Devices namespaces would help this. We're hoping to
discuss design for those at next UDS, but those will come after user
namespaces. In the mean time, you'll need to make sure that the guest
does not mount over /dev, and does not remount /dev/pts.
-serge
That got me thinking and started into looking deeper into systemd, which
Fedora 16 and above uses and why it may be related here. I've made
Fedora 16 work in the past by installing upstart and disabling systemd
but that really becomes impractical in Fedora 17 because they're
including so few of the compatibility scripts. Yes, you are right, the
Fedora's init (systemd) is mounting something on /dev like this:
devtmpfs on /dev type devtmpfs
(rw,nosuid,seclabel,size=1784160k,nr_inodes=446040,mode=755)
This is very bad for the reasons you pointed out in Feb. Looking at the
source code for systemd, this is hard coded into the binary and is not
configurable.
systemd-37/src/mount-setup.c:
--
/* The first three entries we might need before SELinux is up. The
* other ones we can delay until SELinux is loaded. */
#define N_EARLY_MOUNT 3
static const MountPoint mount_table[] = {
{ proc, /proc, proc, NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ sysfs,/sys, sysfs,NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ devtmpfs, /dev, devtmpfs, mode=755,
MS_NOSUID,true },
{ tmpfs,/dev/shm, tmpfs,mode=1777,
MS_NOSUID|MS_NODEV, true },
{ devpts, /dev/pts, devpts, mode=620,gid=
STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
{ tmpfs,/run, tmpfs,mode=755,
MS_NOSUID|MS_NODEV, true },
{ tmpfs,/sys/fs/cgroup, tmpfs,mode=755,
MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
{ cgroup, /sys/fs/cgroup/systemd, cgroup,
none,name=systemd, MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
};
--
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the
container should work, i.e. systemd was not going to fail as a result.
I'm not sure how that would work or if that would work in the case where
you didn't have selinux in the host kernel or you were crossing apparmor
in the container and selinux in the host or vice-versa.
In any case, I'm hitting the systemd-devel list looking to raise their
awareness of the problem and including this list and the fedora list.
If someone wants to mention it on the Arch Linux list, please do, I
don't participate over there.
-serge
Thanks
Regards,
Mike
--
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
___
Lxc-users mailing list
Lxc-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/lxc-users
--
Michael H. Warfield
Re: [Lxc-users] systemd inside LXC
On Sun, 2012-10-21 at 14:49 -0500, Serge Hallyn wrote: Quoting Michael H. Warfield (m...@wittsend.com): Serge, ... Short of building a custom systemd, I don't know how to fix that problem and I suspect this OP is going to run into this same thing (container taking over host's console) and might explain some of what he's seeing. Several of these look like they could cause problems (like /dev/pts in there). I've really reached an impasse at getting systemd (at least Fedora 16 and 17) to work in a container without screwing up the host. Prohibiting mounts entirely in the container might work but I suspect (having read some systemd error messages) systemd is going to have some serious heartburn there. Thoughts? IIRC, simply having apparmor(/selinux) refuse the mount of /dev by the container should work, i.e. systemd was not going to fail as a result. Hopefully, you've seen the message from Kay Sievers cc'ed to this list from my post to the systemd-devel list. Looks like they have a mechanism in place to do this... http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface First step appears to be to set a container=LXC (or some other short string) before invoking init in the container. Is there a mechanism to do this? Might look over the rest of their recommendation and see if there's anything else we need to do. Looks like there might be some additional mounts (some read-only) in there that need to be handled in lxc-start as well. -serge -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users Regards, Mike -- Michael H. Warfield (AI4NB) | (770) 985-6132 | m...@wittsend.com /\/\|=mhw=|\/\/ | (678) 463-0932 | http://www.wittsend.com/mhw/ NIC whois: MHW9 | An optimist believes we live in the best of all PGP Key: 0x674627FF| possible worlds. A pessimist is sure of it! signature.asc Description: This is a digitally signed message part -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
Serge,
I'm going to top post here simply because this is going to go off in a
different direction and bringing in an old thread but it is related...
Back on February 14 you responded to a message about Fedora 16 in a
container, which is something I've been trying to do and I had run into
that posters problems. You responded with this:
Subject: Re: [Lxc-users] fedora 16 under lxc
On Tue, 2012-02-14 at 09:23 -0600, Serge Hallyn wrote:
Quoting Ramez Hanna (rha...@informatiq.org):
now all my efforts have not succeedd to get getty on tty1 to start
unmasking udev did something different
it created all the /dev devices
and made getty start but it started on the hosts's tty not on the
container's
could someone shed some light here?
Blind guess:
lxc-start creates some ptys and bind mounts them onto the guest's
/dev/{console,tty{1,2,3,4}}. It sounds like fedora's init is mounting
over the /dev set up by lxc causing a new /dev/tty to be created as
chardev 4:{1-4}. Devices namespaces would help this. We're hoping to
discuss design for those at next UDS, but those will come after user
namespaces. In the mean time, you'll need to make sure that the guest
does not mount over /dev, and does not remount /dev/pts.
-serge
That got me thinking and started into looking deeper into systemd, which
Fedora 16 and above uses and why it may be related here. I've made
Fedora 16 work in the past by installing upstart and disabling systemd
but that really becomes impractical in Fedora 17 because they're
including so few of the compatibility scripts. Yes, you are right, the
Fedora's init (systemd) is mounting something on /dev like this:
devtmpfs on /dev type devtmpfs
(rw,nosuid,seclabel,size=1784160k,nr_inodes=446040,mode=755)
This is very bad for the reasons you pointed out in Feb. Looking at the
source code for systemd, this is hard coded into the binary and is not
configurable.
systemd-37/src/mount-setup.c:
--
/* The first three entries we might need before SELinux is up. The
* other ones we can delay until SELinux is loaded. */
#define N_EARLY_MOUNT 3
static const MountPoint mount_table[] = {
{ proc, /proc, proc, NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ sysfs,/sys, sysfs,NULL,
MS_NOSUID|MS_NOEXEC|MS_NODEV, true },
{ devtmpfs, /dev, devtmpfs, mode=755,
MS_NOSUID,true },
{ tmpfs,/dev/shm, tmpfs,mode=1777,
MS_NOSUID|MS_NODEV, true },
{ devpts, /dev/pts, devpts, mode=620,gid=
STRINGIFY(TTY_GID), MS_NOSUID|MS_NOEXEC, false },
{ tmpfs,/run, tmpfs,mode=755,
MS_NOSUID|MS_NODEV, true },
{ tmpfs,/sys/fs/cgroup, tmpfs,mode=755,
MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
{ cgroup, /sys/fs/cgroup/systemd, cgroup,
none,name=systemd, MS_NOSUID|MS_NOEXEC|MS_NODEV, false },
};
--
Short of building a custom systemd, I don't know how to fix that problem
and I suspect this OP is going to run into this same thing (container
taking over host's console) and might explain some of what he's seeing.
Several of these look like they could cause problems (like /dev/pts in
there). I've really reached an impasse at getting systemd (at least
Fedora 16 and 17) to work in a container without screwing up the host.
Prohibiting mounts entirely in the container might work but I suspect
(having read some systemd error messages) systemd is going to have some
serious heartburn there.
Thoughts?
Regards,
Mike
On Fri, 2012-10-19 at 10:51 -0500, Serge Hallyn wrote:
Quoting John (l...@jelmail.com):
Hello, I'm in the middle of a migration from init to systemd. I've
completed the transition of my host environment and my 6 existing
containers continue to work as expected (they all use sysvinit
internally). I've started work on a systemd container and am getting
some odd effects.
First off, if I use systemd-nspawn to start the container, it starts
fine. I can log in and halt it and all goes as expected. If, however I
use lxc-start, it clobbers my desktop, which is running in another
container.
So I have 2 problems: (a) the container does not boot and (b) it manages
to effect changes in another container.
I've been searching the 'net for most of this morning looking for
information on using systemd inside a container.
I'm using Arch Linux (3.6.2-1-ARCH) with LXC 0.8.0-rc2. Arch now uses
systemd by default.
To try to test this, I created a basic container and this exhibits the
same problems:
$ mkarchroot test base
Starting with systemd-nspawn works fine:
$ systemd-nspawn -D test/ /sbin/init
Starting with LXC does not:
$ lxc-create -n test -f test.conf
$ lxc-start -n test
The file test.conf contains these two lines:
[Lxc-users] systemd inside LXC
Hello, I'm in the middle of a migration from init to systemd. I've completed the transition of my host environment and my 6 existing containers continue to work as expected (they all use sysvinit internally). I've started work on a systemd container and am getting some odd effects. First off, if I use systemd-nspawn to start the container, it starts fine. I can log in and halt it and all goes as expected. If, however I use lxc-start, it clobbers my desktop, which is running in another container. So I have 2 problems: (a) the container does not boot and (b) it manages to effect changes in another container. I've been searching the 'net for most of this morning looking for information on using systemd inside a container. I'm using Arch Linux (3.6.2-1-ARCH) with LXC 0.8.0-rc2. Arch now uses systemd by default. To try to test this, I created a basic container and this exhibits the same problems: $ mkarchroot test base Starting with systemd-nspawn works fine: $ systemd-nspawn -D test/ /sbin/init Starting with LXC does not: $ lxc-create -n test -f test.conf $ lxc-start -n test The file test.conf contains these two lines: lxc.utsname = test2 lxc.rootfs = /srv/lxc/test When I start the container in LXC, all that happens is that my X session dies (this is running in another container). The X session re-starts but the keyboard does not work. I have to connect using another machine to kill the test container and re-start my desktop container. I can't see anything starting inside the test container. I'd be grateful for any help and/or pointers in the right direction so I can complete this transition to systemd. Many thanks, John -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
Re: [Lxc-users] systemd inside LXC
Quoting John (l...@jelmail.com): Hello, I'm in the middle of a migration from init to systemd. I've completed the transition of my host environment and my 6 existing containers continue to work as expected (they all use sysvinit internally). I've started work on a systemd container and am getting some odd effects. First off, if I use systemd-nspawn to start the container, it starts fine. I can log in and halt it and all goes as expected. If, however I use lxc-start, it clobbers my desktop, which is running in another container. So I have 2 problems: (a) the container does not boot and (b) it manages to effect changes in another container. I've been searching the 'net for most of this morning looking for information on using systemd inside a container. I'm using Arch Linux (3.6.2-1-ARCH) with LXC 0.8.0-rc2. Arch now uses systemd by default. To try to test this, I created a basic container and this exhibits the same problems: $ mkarchroot test base Starting with systemd-nspawn works fine: $ systemd-nspawn -D test/ /sbin/init Starting with LXC does not: $ lxc-create -n test -f test.conf $ lxc-start -n test The file test.conf contains these two lines: lxc.utsname = test2 lxc.rootfs = /srv/lxc/test Add: lxc.network.type = empty If you don't have any lxc.network.type sections, then the container shares network with the host, and so the container talks to the host's systemd. (same with upstart) When I start the container in LXC, all that happens is that my X session dies (this is running in another container). The X session re-starts but the keyboard does not work. I have to connect using another machine to kill the test container and re-start my desktop container. I can't see anything starting inside the test container. I'd be grateful for any help and/or pointers in the right direction so I can complete this transition to systemd. Many thanks, John -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users -- Everyone hates slow websites. So do we. Make your web apps faster with AppDynamics Download AppDynamics Lite for free today: http://p.sf.net/sfu/appdyn_sfd2d_oct ___ Lxc-users mailing list Lxc-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/lxc-users
