[Bug 1504781] Re: lxc: test suites failing on trusty
> or a failure to reach the cloud image server. The tests work on wily and vivid, so in principle they can talk to the cloud image server or linuxcontainers.org. It might of course be that later LXC versions got some proxy fixes or something such. However, the tests in trusty also worked until October 6th and started failing from October 7th on (http://autopkgtest.ubuntu.com/packages/l/lxc/trusty/amd64/). There was no change in autopkgtest or autopkgtest-cloud, or a re-roll of the infrastructure then. The main difference in http://autopkgtest.ubuntu.com/data/packages/trusty/amd64/l/lxc/20151007_063858@.log is indeed kernel -65 to -66. Anyway, I'll investigate this more closely and follow up here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1504781 Title: lxc: test suites failing on trusty To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1504781/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1504781] Re: lxc: test suites failing on trusty
Running lxc test against trusty-release in the CI production environment still works fine (against kernel -65). I do get the hang with running against -proposed, under otherwise the exact same circumstances. The dist-upgrade to -proposed does the following: The following NEW packages will be installed: linux-headers-3.13.0-66 linux-headers-3.13.0-66-generic linux-image-3.13.0-66-generic The following packages will be upgraded: apport grub-common grub-pc grub-pc-bin grub2-common libpam-systemd libpython3.4-minimal libpython3.4-stdlib libsystemd-daemon0 libsystemd-login0 libudev1 linux-headers-generic linux-headers-virtual linux-image-virtual linux-virtual ntpdate python3-apport python3-problem-report python3.4 python3.4-minimal systemd-services udev I obviously hangs in lxc-test-ubuntu. I wonder if that's the first test which actually uses a bootstrapped full ubuntu image, not just a simple busybox one? When it hangs, the following test related processes are running: lxc-dns+ 3298 0.0 0.0 28204 956 ?S09:07 0:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative root 12758 0.0 0.0 4440 656 ?S09:11 0:00 /bin/sh /usr/bin/lxc-test-ubuntu root 31374 0.0 0.0 34724 1348 ?Ss 09:13 0:00 /usr/lib/x86_64-linux-gnu/lxc/lxc-monitord /var/lib/lxc 5 root 31426 0.0 0.0 34712 1504 ?S09:13 0:00 lxc-wait -n 4a5f2adb-d593-4837-8698-f5455e95729e -s RUNNING $ sudo lxc-ls -f NAME STATEIPV4 IPV6 AUTOSTART 4a5f2adb-d593-4837-8698-f5455e95729e STOPPED - - NO so it seems the container never starts up? /var/lib/lxc/4a5f2adb-d593-4837-8698-f5455e95729e/rootfs/ looks like a normal rootfs, and /var/lib/lxc/4a5f2adb-d593-4837-8698-f5455e95729e/config exists too. However, dmesg contains [ 352.395653] type=1400 audit(1444554813.144:26): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="/usr/bin /lxc-start" name="/dev/ptmx" pid=31390 comm="lxc-start" srcname="/dev/pts/ptmx" flags="rw, bind" and when I try to start it, I indeed get root@adt:~# lxc-start -n 4a5f2adb-d593-4837-8698-f5455e95729e -F lxc-start: conf.c: setup_pts: 1772 Permission denied - mount failed '/dev/pts/ptmx'->'/dev/ptmx' lxc-start: conf.c: lxc_setup: 4230 failed to setup the new pts instance lxc-start: start.c: do_start: 688 failed to setup the container lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2 lxc-start: start.c: __lxc_start: 1080 failed to spawn '4a5f2adb-d593-4837-8698-f5455e95729e' lxc-start: lxc_start.c: main: 342 The container failed to start. lxc-start: lxc_start.c: main: 346 Additional information can be obtained by setting the --logfile and --logpriority options. and the same apparmor error repeated. So this surely does look like some lxc/kernel/apparmor related change to me, and I'd hold back that kernel until this gets examined more closely. ** Summary changed: - lxc: test suites failing on trusty + lxc-test-ubuntu hangs forever in trusty-proposed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1504781 Title: lxc-test-ubuntu hangs forever in trusty-proposed To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1504781/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1504781] Re: lxc: test suites failing on trusty
Running lxc test against trusty-release in the CI production environment still works fine (against kernel -65). I do get the hang with running against -proposed, under otherwise the exact same circumstances. The dist-upgrade to -proposed does the following: The following NEW packages will be installed: linux-headers-3.13.0-66 linux-headers-3.13.0-66-generic linux-image-3.13.0-66-generic The following packages will be upgraded: apport grub-common grub-pc grub-pc-bin grub2-common libpam-systemd libpython3.4-minimal libpython3.4-stdlib libsystemd-daemon0 libsystemd-login0 libudev1 linux-headers-generic linux-headers-virtual linux-image-virtual linux-virtual ntpdate python3-apport python3-problem-report python3.4 python3.4-minimal systemd-services udev I obviously hangs in lxc-test-ubuntu. I wonder if that's the first test which actually uses a bootstrapped full ubuntu image, not just a simple busybox one? When it hangs, the following test related processes are running: lxc-dns+ 3298 0.0 0.0 28204 956 ?S09:07 0:00 dnsmasq -u lxc-dnsmasq --strict-order --bind-interfaces --pid-file=/run/lxc/dnsmasq.pid --conf-file= --listen-address 10.0.3.1 --dhcp-range 10.0.3.2,10.0.3.254 --dhcp-lease-max=253 --dhcp-no-override --except-interface=lo --interface=lxcbr0 --dhcp-leasefile=/var/lib/misc/dnsmasq.lxcbr0.leases --dhcp-authoritative root 12758 0.0 0.0 4440 656 ?S09:11 0:00 /bin/sh /usr/bin/lxc-test-ubuntu root 31374 0.0 0.0 34724 1348 ?Ss 09:13 0:00 /usr/lib/x86_64-linux-gnu/lxc/lxc-monitord /var/lib/lxc 5 root 31426 0.0 0.0 34712 1504 ?S09:13 0:00 lxc-wait -n 4a5f2adb-d593-4837-8698-f5455e95729e -s RUNNING $ sudo lxc-ls -f NAME STATEIPV4 IPV6 AUTOSTART 4a5f2adb-d593-4837-8698-f5455e95729e STOPPED - - NO so it seems the container never starts up? /var/lib/lxc/4a5f2adb-d593-4837-8698-f5455e95729e/rootfs/ looks like a normal rootfs, and /var/lib/lxc/4a5f2adb-d593-4837-8698-f5455e95729e/config exists too. However, dmesg contains [ 352.395653] type=1400 audit(1444554813.144:26): apparmor="DENIED" operation="mount" info="failed type match" error=-13 profile="/usr/bin /lxc-start" name="/dev/ptmx" pid=31390 comm="lxc-start" srcname="/dev/pts/ptmx" flags="rw, bind" and when I try to start it, I indeed get root@adt:~# lxc-start -n 4a5f2adb-d593-4837-8698-f5455e95729e -F lxc-start: conf.c: setup_pts: 1772 Permission denied - mount failed '/dev/pts/ptmx'->'/dev/ptmx' lxc-start: conf.c: lxc_setup: 4230 failed to setup the new pts instance lxc-start: start.c: do_start: 688 failed to setup the container lxc-start: sync.c: __sync_wait: 51 invalid sequence number 1. expected 2 lxc-start: start.c: __lxc_start: 1080 failed to spawn '4a5f2adb-d593-4837-8698-f5455e95729e' lxc-start: lxc_start.c: main: 342 The container failed to start. lxc-start: lxc_start.c: main: 346 Additional information can be obtained by setting the --logfile and --logpriority options. and the same apparmor error repeated. So this surely does look like some lxc/kernel/apparmor related change to me, and I'd hold back that kernel until this gets examined more closely. ** Summary changed: - lxc: test suites failing on trusty + lxc-test-ubuntu hangs forever in trusty-proposed -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1504781 Title: lxc-test-ubuntu hangs forever in trusty-proposed To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1504781/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1504781] Re: lxc: test suites failing on trusty
> or a failure to reach the cloud image server. The tests work on wily and vivid, so in principle they can talk to the cloud image server or linuxcontainers.org. It might of course be that later LXC versions got some proxy fixes or something such. However, the tests in trusty also worked until October 6th and started failing from October 7th on (http://autopkgtest.ubuntu.com/packages/l/lxc/trusty/amd64/). There was no change in autopkgtest or autopkgtest-cloud, or a re-roll of the infrastructure then. The main difference in http://autopkgtest.ubuntu.com/data/packages/trusty/amd64/l/lxc/20151007_063858@.log is indeed kernel -65 to -66. Anyway, I'll investigate this more closely and follow up here. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1504781 Title: lxc: test suites failing on trusty To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1504781/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1504781] Re: lxc: test suites failing on trusty
Keeping notes: I did a local QEMU run against trusty release and trusty- proposed: adt-run lxc -s --- qemu /srv/vm/adt-trusty-amd64-cloud.img adt-run --apt-pocket=proposed -U lxc -s --- qemu /srv/vm/adt-trusty-amd64-cloud.img They both fail for the same reason: five tests fail due to "ERROR: Unable to fetch GPG key from keyserver." -- presumably because the test has some special magic with "Running in the Canonical CI environment" which doesn't apply to my laptop where no proxy is in use. The test doesn't hang there, but that doesn't say that much as the test apparently behaves rather different in local qemu vs. Canonical cloud with proxy. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1504781 Title: lxc: test suites failing on trusty To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1504781/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1504781] Re: lxc: test suites failing on trusty
Keeping notes: I did a local QEMU run against trusty release and trusty- proposed: adt-run lxc -s --- qemu /srv/vm/adt-trusty-amd64-cloud.img adt-run --apt-pocket=proposed -U lxc -s --- qemu /srv/vm/adt-trusty-amd64-cloud.img They both fail for the same reason: five tests fail due to "ERROR: Unable to fetch GPG key from keyserver." -- presumably because the test has some special magic with "Running in the Canonical CI environment" which doesn't apply to my laptop where no proxy is in use. The test doesn't hang there, but that doesn't say that much as the test apparently behaves rather different in local qemu vs. Canonical cloud with proxy. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1504781 Title: lxc: test suites failing on trusty To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1504781/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
[Bug 1504781] Re: lxc: test suites failing on trusty
They all get stuck in lxc-test-ubuntu which would indicate either a hang in debootstrap (newly introduced debconf question) or a failure to reach the cloud image server. In either case, you've not actually regressed LXC, the other tests would have failed if that was the case. So I'd toss this one over to pitti for investigation and release the updated kernels regardless. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1504781 Title: lxc: test suites failing on trusty To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1504781/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1504781] Re: lxc: test suites failing on trusty
They all get stuck in lxc-test-ubuntu which would indicate either a hang in debootstrap (newly introduced debconf question) or a failure to reach the cloud image server. In either case, you've not actually regressed LXC, the other tests would have failed if that was the case. So I'd toss this one over to pitti for investigation and release the updated kernels regardless. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to lxc in Ubuntu. https://bugs.launchpad.net/bugs/1504781 Title: lxc: test suites failing on trusty To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1504781/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs