Your message dated Mon, 02 Jul 2018 11:19:01 +0000 with message-id <[email protected]> and subject line Bug#851558: fixed in autopkgtest 5.4 has caused the Debian Bug report #851558, regarding autopkgtest: define Restrictions for tests that aren't suitable for gating CI to be marked as done.
This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact [email protected] immediately.) -- 851558: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=851558 Debian Bug Tracking System Contact [email protected] with problems
--- Begin Message ---Package: autopkgtest Version: 4.3 Severity: wishlist As mentioned in recent discussion on debian-devel, I don't think all autopkgtests are sufficiently reliable, stable or important to be used to gate CI or block testing migration. (In Debian terms: every test failure is a bug, but not every bug is release-critical.) However, I don't want to have to disable unimportant or known-unreliable tests altogether, and lose the ability to see how their results change over time: I would prefer to keep running them for the logs, and just ignore the side-effects. Specifically, if a maintainer is unable to fix a particular unreliable or broken test, or the non-RC bug that it exposes, I think it's still correct for that test to continue to run on CI infrastructure, with its result logged but the usual side-effects of that result ignored. This can give the maintainer valuable feedback to send to upstream (for example I can now tell ostree's author that ostree/test-local-pull.sh.test fails around half the time, but the other pull tests that sporadically failed have stopped failing since a thread-safety issue in libsoup was fixed). With the ability to ignore selected restrictions (#850494), this could be addressed by a restriction, perhaps something like one of these: Restrictions: experimental Restrictions: intermittently-fails Restrictions: may-fail Restrictions: unimportant possibly accompanied by: Features: known-bug=http://bugs.debian.org/1234 Tests with that restriction could perhaps be run normally, but skipped when determining whether a batch of changes would break testing. This is similar in concept to the TODO directive in TAP <https://testanything.org/tap-specification.html>, which marks a failure as not invalidating the overall success of the test. S
--- End Message ---
--- Begin Message ---Source: autopkgtest Source-Version: 5.4 We believe that the bug you reported is fixed in the latest version of autopkgtest, which is due to be installed in the Debian FTP archive. A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to [email protected], and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. Paul Gevers <[email protected]> (supplier of updated autopkgtest package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing [email protected]) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Format: 1.8 Date: Mon, 02 Jul 2018 11:50:21 +0200 Source: autopkgtest Binary: autopkgtest Architecture: source Version: 5.4 Distribution: unstable Urgency: medium Maintainer: Debian CI team <[email protected]> Changed-By: Paul Gevers <[email protected]> Description: autopkgtest - automatic as-installed testing for Debian packages Closes: 832751 851558 867081 897170 901643 Changes: autopkgtest (5.4) unstable; urgency=medium . [ Simon McVittie ] * README.package-tests.rst: Document AUTOPKGTEST_NORMAL_USER * d/tests/lxd: Don't assume all test-runners set AUTOPKGTEST_NORMAL_USER * qemu: Only set up base image device if requested * qemu: Document --baseimage * qemu: Update test for --baseimage no longer being the default * qemu, lxc, lxd: Look for a user account in the 1000-59999 range (Closes: #897170) * qemu: Add a shortcut for running tests on an EFI-booted image * doc: Describe how to parse Features, Restrictions, Classes * Add support for flaky tests (Closes: #851558) * Add support for tests declaring themselves to have been skipped * autopkgtest(1): Document FLAKY as a possible summary status . [ Balint Reczey ] * Fix bashism in retrying apt update . [ Paul Gevers ] * Enable testing to continue after badpkg (Closes: #832751) * adt_testbed.py fix missed piece of regular expression in commit a7e1dad * d/tests/autopkgtest Update SchrootRunner * runner/autopkgtest: Drop Ubuntu 12.04 fallback * manpage: make ordering consistent with --help (Closes: #901643) . [ Julian Andres Klode ] * ssh-setup/nova: Add support for keystone v3 auth (LP: #1767433) . [ Rafael Laboissiere ] * Set Maintainer email address to [email protected] . [ Niko Tyni ] * Add a couple of testcases for versioned provides support * Ensure synthesized test dependencies are not satisfied by versioned Provides (Closes: #867081) * Remove the old '(>= 0)' hack for ensuring '@' pulls in real packages Checksums-Sha1: 1720155af7c18ae32212b2e6a0bde1a73b79e98b 1582 autopkgtest_5.4.dsc f9b0c5ab38b22484d0176eec1f115bb0b369116b 173636 autopkgtest_5.4.tar.xz Checksums-Sha256: 5ccc59ec2985b5557763764309867a6068f7139de3e9d06f1e43e240894b4aa4 1582 autopkgtest_5.4.dsc c8c5c14298105d1798ce9a20a4f7d2b837279939a94da5fdc761111517575030 173636 autopkgtest_5.4.tar.xz Files: b12bae17fc3461b1d46d93bf4863c768 1582 devel optional autopkgtest_5.4.dsc 7abe82af7806ef1c477f0fb1be65b7c7 173636 devel optional autopkgtest_5.4.tar.xz -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEWLZtSHNr6TsFLeZynFyZ6wW9dQoFAls59cMACgkQnFyZ6wW9 dQo6kQf/XkKDoSKtvBNmtwkmV62jefiKqvdZlJVVxCyfnrQ5V/3ObsZAkKNCIL8e MIlVOXrbGWSFj4qsnVH1l6+1pZkKjYboL+65xqazt2K8bWwTe1zi6/TDFnU/dcg5 HXS6s0/5U2eO5UvrDObxaKZFwvbex/DrMzTGR3xluvsYheVIA89+qD5MfJ/rUSP8 fMBb8GcbGN2hTIB1PufXhg0wnj99j3tE30Ziycpe87+14Ol9aTpu8b+p3+XU1Uwa LJ4F0+iDnfkG0433GXEGVV4dhq0IpaQySp77DPRC9GQ/OeXod406mU/CtUZkQxnK ZhMfnGZ81Ez8qmrpFDEJoLzTn+pWUw== =joYo -----END PGP SIGNATURE-----
--- End Message ---
