control: tag -1 confirmed Hello Simon,
Simon McVittie [2017-01-16 9:06 +0000]: > Specifically, if a maintainer is unable to fix a particular unreliable > or broken test, or the non-RC bug that it exposes, I think it's still > correct for that test to continue to run on CI infrastructure, with > its result logged but the usual side-effects of that result ignored. FTR, in Ubuntu (same as in the discussion going on for adopting this into Debian) we use britney overrides for this, like in http://bazaar.launchpad.net/~ubuntu-release/britney/hints-ubuntu/view/head:/pitti i. e. the tests still run, but britney won't block the package on its failure. The advantage is that it's much easier to control by the release team (it doesn't require package uploads), and that you can keep more state: the hints usually apply to a (maximum) version so that you can say "1.2-3 is known-broken because of unrelated changes in $foo, but I expect the next upload to fix this again", so that you can't cheat from the side of the developer. As long as Debian doesn't actually use these tests for gating, there is no problem in making them appear as failed. Particular in the tracker you actually do want to point out failing tests. Or what other side effects do you mean? > With the ability to ignore selected restrictions (#850494), this could > be addressed by a restriction, perhaps something like one of these: > > Restrictions: experimental > Restrictions: intermittently-fails > Restrictions: may-fail > Restrictions: unimportant I'm fine with adding this, as from the maintainer's perspective she could just as well do an upload with commenting out that test, and both would appear in a diff either way. This is for a different use case than chasing down regressing tests (what I mentioned above), and thus seems useful for introducing new tests which still need some work to fully work on the distro infrastructure. Maybe "expected-failure" which is the term used by other test runners? (The bikeshedding season is on! ☺ ) > possibly accompanied by: > > Features: known-bug=http://bugs.debian.org/1234 This won't modify the behaviour of the test, so a comment above Restrictions: would just do as well -- but as unknown features are ignored, this is fine of course. > Tests with that restriction could perhaps be run normally, but > skipped when determining whether a batch of changes would break testing. I think autopkgtest would show their result as "EXPECTED-FAIL" or so and exit with 0 (at least for that particular test case)? Thanks for the suggestion! Martin -- Martin Pitt | http://www.piware.de Ubuntu Developer (www.ubuntu.com) | Debian Developer (www.debian.org)
signature.asc
Description: PGP signature
