[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
On Hirsute this now also became (from a yet unknown change in a different component) flaky. Essentially having timing issues between "start" and "lxc-enter-namespace". Since we like the test, but not actually care too much about libvirt-lxc (we favor LXD by orders of magnitude for that use case) I think what I'll implement is: 1. make the test skippable 2. skip if the restart of libvirtd failed (we still want to test the consequence of guests being alive if it worked, but if the restart itself failed we should skip) 3. on check_domain since we seem to have races between start and being ready add a retry mechanism I'll test that in hirsute where the current arm64 issues happen and if working well there that can be what we backport along the next SRUs. ** Summary changed: - autopkgtest almost always fails in Bionic on amd64 + autopkgtest fails Bionic@amd64 (service) and Hirsute@arm64 (racy) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest fails Bionic@amd64 (service) and Hirsute@arm64 (racy) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
FYI: Still no other libvirt change in the queue for bionic that would justify an SRU upload. Current test ratio on the flaky test ~60% passes on smoke-lxc and all other subtests pass 100%. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
I've encountered this failure, too: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac /autopkgtest- bionic/bionic/amd64/libv/libvirt/20201109_214647_148b5@/log.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
Another three runs without issues :-/ So we have a heisenbug :-/ I must say that I clearly showed this isn't due to a change in libvirt NOR does it reveal any debug data - so I'm puzzled at what it might be. I'm ok to mark the test flaky if the situation remains that bad in Bionic (after all it is "only" smoke-lxc which isn't really supported anyway). But an SRU upload just for that feels wrong, I'd think we will do so whenever we do an Bionic SRU anyway. An Block-proposed SRU upload won't be enough as we are not looking at build, but autopkgtest time issues. Those would only be picking up -proposed if triggered that way and if we do that we can as well retry (as we do already). Until then please speak up here if the situation continues (tests keep failing) or get worse (rate increases further). -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
We know it won't reproduce in local VM, but haunts tests on infra. I've added some debug (no functional change) to see what dependent job might fail on that restart, but even running on LP-infra the fail won't trigger ... :-/ https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-ci-train-ppa-service-4311/bionic/amd64/libv/libvirt/20201013_115528_1f7a3@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-ci-train-ppa-service-4311/bionic/amd64/libv/libvirt/20201013_140034_003f0@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-ci-train-ppa-service-4311/bionic/amd64/libv/libvirt/20201014_060703_651fd@/log.gz https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-bionic-ci-train-ppa-service-4311/bionic/amd64/libv/libvirt/20201014_083426_e5163@/log.gz ... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
I added some debugging for the state of the service(s) before and after that call. Hopefully I'm able to trigger the issue on the autopkgtest infrastructure that way to get some insight what is going on. PPA: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/4311 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
The code didn't change much to Focal, but there the test did not degrade. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
Ok, we indeed have ~45% rate of fails in the last 20 runs :-/ All we got from the log is that "dependency job" issue mentioned in the report. Indeed it does pass for me locally as well. But before had 0% issues, so it seems something has changed. I checked the history and there was no upload of libvirt anywhere near the switch to the bad state. Various kernel upload and systemd/237-3ubuntu10.42 is close, but I see no clear connection between those and the fails yet. ** Changed in: libvirt (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
smoke-lxc (F 15% f 0% S 0% B 0% => P 85%/) ..FF.FFF.FF..F...F.. ** Bug watch added: Debian Bug tracker #848317 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848317 ** Bug watch added: Debian Bug tracker #867379 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867379 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
The test switches to set -x for the actual test steps, so we only have very few commands that run before. In fact it only is 1. export LIBVIRT_DEFAULT_URI='lxc:///' 2. systemctl restart libvirtd The latter is what fails with our: "dependency job for libvirtd.service failed" That restart is for https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=848317 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=867379 But we don't see the effect of these bugs, instead we need to track down what dependency issue happens on autopkgtest-infra when running this test. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
Furthermore worth to note that the issue only affects x64. I see it on amd64 + i386 but not on arm/s390x/ppc64el. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1899180] Re: autopkgtest almost always fails in Bionic on amd64
Hmm, we had issues in the past with this one related to various kernel images and other little details. I'll take a look. ** Tags added: server-next -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1899180 Title: autopkgtest almost always fails in Bionic on amd64 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1899180/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
