[Bug 1732028] Re: transient boot fail with overlayroot [open-iscsi autopkg tests]
I could confirm that I can run the guest that way and it uses the intended root=/dev/disk/by-path/ip-10.0.12.2:3260-iscsi-tgt-boot-test- b7X8g2-lun-1-part1. The only and unfortunate difference to the issue when run on LP infra stays, that it works all of the time :-/ Note: I ran most of them in background after verifying once how they behave. But wanted to make sure they complete reproducibly. a - manual login, no user data case 3/3 b - user data collect data to disk and shut down 3/3 c - user data collect just shut down 3/3 d - user data collect data to disk and shut down, no tty (nohup CMD > log 2>&1) 3/3 e - non interactive like autopkgtest would run it 3/3 f - non interactive like autopkgtest would run it forcing KVM mode 3/3 Next I was parsing all the logs that the fails on LP accumulated recently. I found this errors whichi is interesting: iscsistart: initiator reported error (15 - session exists) I realized this is present on ALL logs that we gathered. But after thinking I had a lead on this I realized that the good cases had those messages as well. Also found the mentioned "ordering cycle on media-root\x2dro.mount/start" In good as well as bad logs. So neither of these "is it" Essentially the boot around the iscsi root has these steps with some noise in between - looking for differences in good/bad cases. They start the same even with sharing a few errors that seem to be red herrings: [...] (early boot) all logs (id changes) - Logging into tgt-boot-test-o3PlsL 10.1.1.2:3260,1 all logs - mounted filesystem with ordered data on 7/17 logs (also good) - Found ordering cycle ... only all bad cases - Dependency failed for Local File Systems only all bad cases - Timed out waiting for device (devices change) only all bad cases - Started Emergency Shell That matches this bug here. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1732028 Title: timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests on LP Infra] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1732028/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1732028] Re: transient boot fail with overlayroot [open-iscsi autopkg tests]
There also was a slight remaining uncertainty if this might be the detection of KVM being broken and running in nested KVm on Launchpad. But I was able to confirm that it runs as intended in TCG mode: should_try_kvm = no. virt=kvm (nested kvm is finicky). set _USE_KVM=1 to force. [1]: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac /autopkgtest-cosmic-ci-train-ppa-service-3325/cosmic/amd64/o/open- iscsi/20180719_104631_271d7@/log.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1732028 Title: timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests on LP Infra] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1732028/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1732028] Re: transient boot fail with overlayroot [open-iscsi autopkg tests]
FYI: In Cosmic this currently blocks (and will block more soon due to python bound on it): qemu, debconf, python3, targetcli-fb, netifaces It also is no more transient, but happens always on LP infrastructure recently (~20 reruns now). Knowing it likely is "too slow" even being unsure why LP in particular is affected we can try to recreate knowing that. I checked CPU consumption in: - KVM mode (super low) - TCG mode (~1.2 CPUs) Adapting title as well ... ** Summary changed: - transient boot fail with overlayroot [open-iscsi autopkg tests] + timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests] ** Summary changed: - timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests] + timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests on LP Infra] -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1732028 Title: timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests on LP Infra] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1732028/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1732028] Re: transient boot fail with overlayroot [open-iscsi autopkg tests]
CE: Moving over some of the discussion that started on a related MP [1]. Smoser: I put together some doc on how to run the https://hackmd.io/E0ydu7Y7QEe-kroPb6-OOA That should help you to possibly recreate outside of the adt test harness. [1]: https://code.launchpad.net/~paelzer/ubuntu/+source/open-iscsi/+git /open-iscsi/+merge/349799 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1732028 Title: timeout in iscsi boot fail with overlayroot [open-iscsi autopkg tests on LP Infra] To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/open-iscsi/+bug/1732028/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
