[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
After years of running LXC and LXD, this is a somewhat recurring topic which has no good answers, all distros do it differently and the definition of ready differs even user to user. So from LXC's point of view, it's best to stay away from this and instead have users actually check for what they care, be it a web service being ready, cloud-init indicating it's done running or whatever else and just use LXC's tools and API to run those checks. ** Changed in: lxc (Ubuntu) Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
I just got hinted at "lxc-attach". With that, clients like adt-virt-lxc can implement their own polling/waiting loop to wait until the runlevel switches away from unknown/S, which is a lot better (and more generic) than the current hackery that it's doing. I captured that in bug 1273725. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
Speaking as the upstream for LXC and package maintainer, the only way I consider to be acceptable for this is the one Serge and I described by adding a new socket owned by LXC which allows for a simple text based communication between the container and its host. This protocol would allow "state ready" kind of command which would allow switching to a list of allowed states that'd in turn be visible in lxc-wait. Monitoring the container's runlevel isn't practical as inotify isn't reliable, lxc-attach doesn't work on all kernels and containers may store that information on a filesystem we can't see from the host (e.g. /run on tmpfs) so I'll NACK any patch trying to implement something like this, both as an upstream change and as a distro-specific change. I've been fighting pretty hard this cycle to make sure LXC in Ubuntu is perfectly in sync with upstream, we don't carry any patches, our upstart jobs and apparmor profiles all come from upstream now, I'm not about to change this for this kind of hack, especially not when I'll have to maintain this for 5 years. We also have a policy in Ubuntu that LXC should not do any customization of the rootfs that'd be wrong to do on a host. So adding upstart jobs from the template is out of the question, any extra upstart jobs has to come through the distro (there again, we've been fighting to achieve this for years, I'm not about to compromise on this now). Note that in any case, this feature very likely won't make it before LXC 1.1 as I'm about to tag the first release candidate of LXC 1.0 after which point I won't be accepting feature changes anymore. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
> Each distro+release guest would need to be updated using its own init system to inform us when the boot is complete LXC could provide a default implementation (like the above "wait until runlevel switches to 2 and then notify the caller"), and then templates could optionally provide a more refined version of that. I see nothing wrong with that, given that the notion of "when is the container safe to work with" seems quite universal? I see nothing wrong with putting that knowledge into the templates; that's pretty much exactly where it belongs to IMHO. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
On Tue, Jan 28, 2014 at 02:06:30PM -, Serge Hallyn wrote: > Each distro+release guest would need to be updated using its own init > system to inform us when the boot is complete, resulting in > container-specific changes to the rootfs which is something we want > to avoid. I understand your reasoning, but requiring tools that consume LXC to make these container-specific changes to the rootfs is worse; especially when there's one general concept that will need duplicated implementations across all consumers. You're just forcing the problem downstream, and the problem is exacerbated there. If you insist on not doing it this way, how about "contrib" style scripts, provided in the lxc package, which understands different init systems and arranges the wait mechanism? This would be a "idempotently-modify-this-container-to-add-boot-wait" command, with another "boot-wait-on-the-container-i-modified" command. Then at least the code will be in one place, and all consumers will use the same mechanism. A consumer would have to run the container modification script in between clone and container start (I assume this is possible). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
In this millenium the concept of "finished booting" becomes more and more blurry anyway. We still have something like that with sysvinit if you do an rc2.d/S99_something, but with upstart/systemd/udev rules this already isn't accurate. To use the least common denominator we could have a late rc2.d sysvinit script do that, to notify callers that the container has booted sufficiently to have a working file system etc. This already won't guarantee us that the network is up (NetworkManager!) but any particular condition (wait for network, wait for ssh, etc.) that the lxc-wait caller expects can then be done using commands that you send to the container. So I think the initial "wait until its safe to send files to the container and run commands in it" provides a nice baseline which shouldn't be Debian specific if we use the runlevel/sysv approach from above. (Please keep in mind that this isn't just for autopkgtests, but that you can easily have e. g. a Fedora container on Ubuntu). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
Quoting Robie Basak (1266...@bugs.launchpad.net): > I want it generic because I want adt-virt-lxc to work with any container > it is given, without having to tell the user to set up a special > container with special stuff in this special way. > > Other tools will want this too. I have a simple generic "trusty" > container that I use to "start-ephemeral" (or "clone", I suppose) > whatever testing I need. Multiple tools can end up using this (adt-virt- > lxc and perhaps eventually a uvt-lxc). They need to agree on a common > way to achieve this common task; I think the suitable place for this to > go is lxc. Each distro+release guest would need to be updated using its own init system to inform us when the boot is complete, resulting in container-specific changes to the rootfs which is something we want to avoid. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
I want it generic because I want adt-virt-lxc to work with any container it is given, without having to tell the user to set up a special container with special stuff in this special way. Other tools will want this too. I have a simple generic "trusty" container that I use to "start-ephemeral" (or "clone", I suppose) whatever testing I need. Multiple tools can end up using this (adt-virt- lxc and perhaps eventually a uvt-lxc). They need to agree on a common way to achieve this common task; I think the suitable place for this to go is lxc. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
FWIW, I just tried the following: sudo mknod /tmp/x1 sudo lxc-create -t ubuntu-cloud -n c1 sudo cat >> /var/lib/lxc/c1/config << EOF lxc.mount.entry = /tmp/x1 x1 none bind,create=file 0 0 EOF sudo cat > /var/lib/lxc/c1/rootfs/etc/init/x.conf << EOF description "tell lxc monitor we are ready" author "Serge Hallyn" start on runlevel [2345] task script echo "READY" > /x1 || true end script EOF Terminal 1: sudo tail -f /tmp/x1 Terminal 2: sudo lxc-start -n c1 "READY" showed up in my /tmp/x1 tail output as expected. So this should be a workable for what you want, to the point that I'm not sure whether something more generic is desirable. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
I'd prefer to wait until the system is fully booted (traditional equivalent: rc2.d scripts are all complete), rather than just that startup scripts are done (traditional equivalent: rcS.d scripts are all complete). It doesn't seem to be that waiting for everything to be started would be much of an issue, if we could do it. And we might pre-empt some unusual bugs by more general assumptions being made by LXC consumers. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
I didn't look too much into this, but would it be possible to make lxc- wait check the container's runlevel, and have a new "booted" state (unless you want to change the meaning of RUNNING) become true once the runlevel stops being unknown or N or S? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
The listening end would be lxc-start, the other end would be whatever is appropriate. In Ubuntu this could be an upstart job, on Debian a sysvinit script, ... There wouldn't be a need for a persistent connection, the socket would just be there to send messages from the container to the host, so anything could use netcat to do something like: echo "state ready" | nc -U /dev/lxc -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
One possibility, an upstart job could write to /dev/lxc in the container. lxc-start would listen to the other end on the host, and update the container state to USERSPACE_RUNNING or somesuch. Then lxc-ls would show USERSPACE_RUNNING instead of RUNNING. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
Thanks. What would listen or own the other end of the socket? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1266808] Re: No mechanism to wait until a started container is ready and has finished booting
This was discussed a few times upstream. The suggested implementation is to have lxc-start create and bind /dev/lxc in the container as a standard unix socket. A simple text based protocol would then be built on top of this including support for switching to a new "ready" state. The idea is that an init script in the container could then do: echo "state ready" | nc -U /dev/lxc However this never got implemented and isn't on the roadmap for now, so if you need this feature, patches implementing the suggestion above are definitely welcome. ** Changed in: lxc (Ubuntu) Status: New => Triaged -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1266808 Title: No mechanism to wait until a started container is ready and has finished booting To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1266808/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
