Looking at this again now I'm not sure we have enough information to determine where actually things went wrong. It's not clear to me if we were in "is_failure" and how many times the test was actually retried. Unfortunately, the historical autopkgtest-cloud-worker log files are gone so we are at an impasse here.
-- You received this bug notification because you are a member of Canonical's Ubuntu QA, which is subscribed to Auto Package Testing. https://bugs.launchpad.net/bugs/1978741 Title: FAIL_PKG_STRINGS not considered in temp_fails Status in Auto Package Testing: New Bug description: 106 # Some packages can provoke specific breakage. For most packages, this would be 107 # a sign of infrastructure trouble, but for these we should play it safe and 108 # consider these to be regressions. If they *are* infrastructure problems, 109 # we'll have to retry them. 110 FAIL_PKG_STRINGS = {'systemd*': ['timed out waiting for testbed to reboot', 111 'Timed out on waiting for ssh connection', 112 'Temporary failure resolving', 113 'VirtSubproc.Timeout', 114 'ERROR: testbed failure: testbed auxverb failed with exit code 255'], FAIL_PKG_STRINGS isn't used when constructing temp_fails 799 elif is_failure: 800 contents = log_contents(out_dir) 801 temp_fails = [s for s in (set(TEMPORARY_TEST_FAIL_STRINGS) 802 - set(getglob(OK_PKG_STRINGS, pkgname, []))) 803 if s in contents] 804 if temp_fails: 805 logging.warning('Saw %s in log, which is a sign of a temporary failure.', 806 ' and '.join(temp_fails)) 807 logging.warning('%sLog follows:', retrying) 808 logging.error(contents) 809 if retry < 2: 810 submit_metric(architecture, code, pkgname, current_region, True, release) 811 cleanup_and_sleep(out_dir) 812 else: 813 break I noticed this because we ended up in a situation where systemd was in fact failing to resolve ftpmaster.internal and the tests were continuously retried. E: Failed to fetch http://ftpmaster.internal/ubuntu/pool/main/e/elfutils/libdw-dev_0.187-1_ppc64el.deb Temporary failure resolving 'ftpmaster.internal' E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? autopkgtest [15:09:55]: ERROR: testbed failure: apt repeatedly failed to download packages autopkgtest [15:09:56]: test tests-in-lxd: -----------------------] autopkgtest [15:09:57]: test tests-in-lxd: - - - - - - - - - - results - - - - - - - - - - tests-in-lxd FAIL non-zero exit status To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/1978741/+subscriptions -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : [email protected] Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp
