Ok, so investigation shows that: - LXD bind-mounts all that stuff, it doesn't have a choice as it's not privileged enough to mount things itself - mountall fails to run if its "optional" filesystems fail to mount (because that makes a lot of sense...) - systemd sets up the host filesystems, on a clean boot they all seem fine - "something" apparently remounts debugfs ro sometimes, this breaks containers - "something" apparently makes the /proc/sys/fs/binfmt_misc autofs go nuts (loop of symlinks) which also breaks containers
We could try to teach mountall to do the right thing with optional mount and ignore their failures, however we'd need to SRU that to trusty and precise and then nag other distros in doing the same (centos, oracle, rhel, ...) before we can get rid of our workaround. As a clean Xenial system does work properly, I think it would be best to figure out what's messing with debugfs and binfmt_misc post-boot and fix whatever it is to stop doing that. Would be useful if the bug reporters could document exactly what they did on their system between the time it worked fine and the time it stopped working so we can figure out what's messing with those mounts. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1551854 Title: LXD bootstrap issues on xenial Status in linux package in Ubuntu: Incomplete Bug description: I'm using lxd with zfs block storage on xenial, and having issues with trusty containers. I've witnessed this problem when trying to bootstrap as well as after rebooting the host and a container failed to start. In the latter case, the container that failed to start was the juju controller: c@mawhrin-skel:~/omnibus-layers$ lxc list [3/3] +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ | juju-5f4bd172-ad22-4726-8d84-47185ab31b54-machine-0 | STOPPED | | | PERSISTENT | 0 | +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ | juju-5f4bd172-ad22-4726-8d84-47185ab31b54-machine-1 | RUNNING | 10.0.3.28 (eth0) | | PERSISTENT | 0 | +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ | juju-5f4bd172-ad22-4726-8d84-47185ab31b54-machine-2 | RUNNING | 10.0.3.85 (eth0) | | PERSISTENT | 0 | +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ | juju-5f4bd172-ad22-4726-8d84-47185ab31b54-machine-3 | RUNNING | 10.0.3.176 (eth0) | | PERSISTENT | 0 | +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ | juju-5f4bd172-ad22-4726-8d84-47185ab31b54-machine-4 | RUNNING | 10.0.3.66 (eth0) | | PERSISTENT | 0 | +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ | juju-5f4bd172-ad22-4726-8d84-47185ab31b54-machine-5 | RUNNING | 10.0.3.31 (eth0) | | PERSISTENT | 0 | +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ | juju-5f4bd172-ad22-4726-8d84-47185ab31b54-machine-6 | RUNNING | 10.0.3.196 (eth0) | | PERSISTENT | 0 | +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ | juju-5f4bd172-ad22-4726-8d84-47185ab31b54-machine-7 | RUNNING | 10.0.3.186 (eth0) | | PERSISTENT | 0 | | | | 10.0.4.1 (lxcbr0) | | | | +-----------------------------------------------------+---------+--------------------------------+------+------------+-----------+ I manually started it, but found that no upstart services were started. Remembering this thread, https://lists.ubuntu.com/archives/juju/2016-February/006698.html, I checked /var/log/mountall.log in the machine-0 container, and sure enough: root@juju-5f4bd172-ad22-4726-8d84-47185ab31b54-machine-0:~# cat /var/log/upstart/mountall.log mount: permission denied mountall: mount /sys/kernel/debug [187] terminated with status 32 mountall: Filesystem could not be mounted: /sys/kernel/debug The problem persists if I stop and start the container. If I remount /sys/kernel/debug on the host, then stop and start the container, upstart succeeds and the juju controller starts up. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: linux-image-4.4.0-8-generic 4.4.0-8.23 ProcVersionSignature: Ubuntu 4.4.0-8.23-generic 4.4.2 Uname: Linux 4.4.0-8-generic x86_64 NonfreeKernelModules: zfs zunicode zcommon znvpair zavl ApportVersion: 2.20-0ubuntu3 Architecture: amd64 AudioDevicesInUse: USER PID ACCESS COMMAND /dev/snd/controlC0: c 24562 F.... pulseaudio CurrentDesktop: MATE Date: Tue Mar 1 10:52:45 2016 EcryptfsInUse: Yes HibernationDevice: RESUME=UUID=007cedda-f922-4e4c-89b1-57b31f18292e InstallationDate: Installed on 2016-02-28 (2 days ago) InstallationMedia: Ubuntu-MATE 16.04 LTS "Xenial Xerus" - Alpha amd64 (20160224) MachineType: LENOVO 2306CTO ProcFB: 0 inteldrmfb ProcKernelCmdLine: BOOT_IMAGE=/vmlinuz-4.4.0-8-generic.efi.signed root=/dev/mapper/ubuntu--mate--vg-root ro quiet splash vt.handoff=7 RelatedPackageVersions: linux-restricted-modules-4.4.0-8-generic N/A linux-backports-modules-4.4.0-8-generic N/A linux-firmware 1.156 RfKill: 0: phy0: Wireless LAN Soft blocked: no Hard blocked: no SourcePackage: linux UpgradeStatus: No upgrade log present (probably fresh install) dmi.bios.date: 04/22/2014 dmi.bios.vendor: LENOVO dmi.bios.version: G2ETA1WW (2.61 ) dmi.board.asset.tag: Not Available dmi.board.name: 2306CTO dmi.board.vendor: LENOVO dmi.board.version: Not Defined dmi.chassis.asset.tag: No Asset Information dmi.chassis.type: 10 dmi.chassis.vendor: LENOVO dmi.chassis.version: Not Available dmi.modalias: dmi:bvnLENOVO:bvrG2ETA1WW(2.61):bd04/22/2014:svnLENOVO:pn2306CTO:pvrThinkPadX230:rvnLENOVO:rn2306CTO:rvrNotDefined:cvnLENOVO:ct10:cvrNotAvailable: dmi.product.name: 2306CTO dmi.product.version: ThinkPad X230 dmi.sys.vendor: LENOVO To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1551854/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp