On Mon, 16 Jan 2017 at 12:34:50 +0100, Martin Pitt wrote: > Maybe "expected-failure" which is the term used by other test runners? (The > bikeshedding season is on! ☺ )
The bikeshedding season appears to be 1st January to 31st December, inclusive. (No bikeshedding allowed during the leap second? :-) I avoided suggesting "expected-failure" because it is actually not fully consistent. In some test frameworks (Automake XFAIL_TESTS), success where a failure was expected makes the overall test *fail*; in others (TAP tests that report "not ok [n] # TODO"), it leaves the overall test succeeding, possibly with a warning (because in principle now you can switch it to be a normal test, if you trust it to be reliably passing, which of course you don't necessarily for a while). Because in the real world tests are not always 100% reliable, my priority is to define something that I can correctly put on a test that intermittently fails. If you want to have "must fail" *as well*, that would be fine (but I'm probably not going to use that one). It would also be nice if we had an exit status that marked a test as skipped, perhaps 77 like in Automake - at the moment, a test that probes its environment and determines that it can't run here has to exit 0 and show up as passed. Would you be interested in a patch for this? > > Features: known-bug=http://bugs.debian.org/1234 > > This won't modify the behaviour of the test, so a comment above Restrictions: > would just do as well -- but as unknown features are ignored, this is fine of > course. My thought was that frontends (ci.debian.net) could maybe extract this information and use it to present autopkgtest's raw results a bit more helpfully. > I think autopkgtest would show their result as "EXPECTED-FAIL" or so and exit > with 0 (at least for that particular test case)? Something like that, yes; either that, or report EXPECTED-FAIL or similar in text, but pretend the test case was skipped when deciding what exit-status autopkgtest should have? (So in practice it would exit 2) S
