[Bug 1885186] Re: autopkgtests sometimes timeout
The Eoan Ermine has reached end of life, so this bug will not be fixed for that release ** Changed in: edk2 (Ubuntu Eoan) Status: New => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1885186] Re: autopkgtests sometimes timeout
This build log shows tests running for ~40s: https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac /autopkgtest-groovy/groovy/armhf/e/edk2/20200807_201106_d36d2@/log.gz This provides the evidence I was looking for in Comment #6 that the tests can take longer than 30s and still pass. I therefore think it worth including this fix in a future SRU. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1885186] Re: autopkgtests sometimes timeout
Have we tried adding edk2 source package, to "big_packages" configuration of autopkgtest-cloud to spin up larger VM for edk2 tests to run? I.e. a multi-core one. As the default VM is single core I think. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1885186] Re: autopkgtests sometimes timeout
The initial autopkgtest results after bumping the timeout to 60s passed, but each test ran within a time length that may have passed even with the previous 30s timeout (longest case I see was 32s). Before (low- impact) SRU'ing this back, ideally we'd be able to point to a test that clearly blew passed the 30s timeout, but still eventually completed. On the other hand, waiting for such proof to appear naturally in groovy before SRU'ing back could result in more unnecessary failures. While I could be swayed either way, I think I'll wait a bit for a demonstrable case in groovy. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1885186] Re: autopkgtests sometimes timeout
Thanks for spawning such a quality analysis out of my trivial IRC ping! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1885186] Re: autopkgtests sometimes timeout
This bug was fixed in the package edk2 - 2020.05-2ubuntu1 --- edk2 (2020.05-2ubuntu1) groovy; urgency=medium * Increase autopkgtest timeout from 30s to 60s. LP: #1885186. -- dann frazier Thu, 25 Jun 2020 16:17:15 -0600 ** Changed in: edk2 (Ubuntu Groovy) Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1885186] Re: autopkgtests sometimes timeout
** Description changed: - Recently edk2 autopkgtests fail intermittently - but the frequency has - increased quite a bit recently. The autopkgtests run QEMU to boot to an - EFI prompt, and then power off. The run for each of the supported archs + edk2 autopkgtests fail intermittently - but the frequency has increased + quite a bit recently. The autopkgtests run QEMU to boot to an EFI + prompt, and then power off. The run for each of the supported archs (ARM, AARCH64 & X64). The failure symptom is that 1 or more of these tests times out after not seeing a prompt for 30s. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1885186] Re: autopkgtests sometimes timeout
The failures don't appear to follow an Ubuntu release, a host architecture, or the guest architecture being tested. So did something change externally? Looking at the data: Prior to June 2020, we only saw a 2.7% failure rate (3/112). Starting in June, that's grown to 25% (8/32). This is across all releases, so it's unlikely due to a change in edk2 itself (eoan/focal releases haven't changed recently). Perhaps something changed with our test systems. Just looking at armhf tests, time to run a test seems to have gone up, on average, since then - +6s/+3s/+6s for AARCH64/ARM/X86 guests, respectively. What's also interesting is that, again just looking at armhf, 100% of the time (7/7) we had a timeout, there was another test very close to timing out (>= 30s). When no timeouts occurred, only 13% (6/46) had any test close to timing out. What that says to me is that this is unlikely to be an actual hang. Rather, sometimes we get a test host w/ less available resources[*], and the odds of that have gone up recently for some reason. I'll therefore plan to bump the timeout to 60s to see if that resolves the issue. [*] Hard to say what resource(s) could be limited here - one theory is that QEMU is waiting on available entropy, but it could just be CPU. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1885186] Re: autopkgtests sometimes timeout
It's not clear to me what the pattern is here, so I wrote a script to generate a csv file of the failures, including how long each test took and which ones timed out. ** Attachment added: "edk2-autopkgtest-times.py" https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+attachment/5387224/+files/edk2-autopkgtest-times.py -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1885186] Re: autopkgtests sometimes timeout
And here's the resulting csv from that script. ** Attachment added: "times.csv" https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+attachment/5387225/+files/times.csv -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1885186 Title: autopkgtests sometimes timeout To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/edk2/+bug/1885186/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
