Hi Rebecca, all, [I added Niels from the Release Team to get his opinion last remarks.]
Thanks for your interest in these bugs. On 27-07-18 22:49, Rebecca N. Palmer wrote: > I take this discussion to mean the existence of autopkgtests is > considered a good thing even when they can't be run on ci.debian.net > (should we explicitly say that somewhere?). Good point, any suggestion? Maybe improve the existing lintian tag further? see e.g.: https://salsa.debian.org/lintian/lintian/commit/84a6066ba477f1d881dcee17d3ad3a2f1e473cba > I hence added some to > beignet (an OpenCL driver, so hardware specific). Great. > Simon McVittie wrote: >> At some point I might add a way to mark some tests as trivial, with >> the intention that migration is never sped up by trivial tests passing, >> only slowed down by them failing - but that's a future feature. > > With the neutral state proposed here, a test can effectively mark itself > as trivial by returning 77(skip) if it passes. (beignet's test1 is sort > of this: its skip state means "the package couldn't find a suitable GPU, > but did manage to say so without crashing", which is not a total no-op > (it would have caught #745363 and #779213) but also not the test it does > when it does find one.) > > However, that's also true of flaky (return 77 on fail), and I agree it > would be more user-friendly to make this option explicit. Which option, to use flaky versus skippable? Sorry, could you try to phrase that sentence again, because I don't get what you mean here. > Do you propose to add this to autodep8-generated "check that we can > import the top-level module" tests? (And should this discussion move to > a new wishlist bug?) Please raise a new bug about that. A similar idea has come to my mind about allow-stderr, so I think this is worth discussing. > The proposed changes would move them to debci's new >> neutral state, > > There is at least one circumstance where debci sees a package that > doesn't have (or claim to have) tests: when a package starts having > tests, debci checks both the unstable (with the new tests) and testing > (without tests) versions, e.g. > https://ci.debian.net/packages/b/beignet/testing/amd64/ It's not debci that does that, it's britney. And there is already a merge request (about to be merged by the looks of it) to remove that code. https://salsa.debian.org/release-team/britney2/merge_requests/5/ >> which britney would effectively treat the same as "always >> failed" afaics. > > If I read the above code correctly (I haven't tried actually running > it), no. While 'always failed' and 'neutral' give the same action > (default migration time), they aren't the same state, and neutral -> > fail counts as a regression. That is something we can debate of course. The change is a one liner change I think. I think the downside of not calling this a regression is that a package can in principle slide from passing to neutral, and later slide further from neutral to failing without noticing. Remember, not only for uploads of the package itself (e.g. by changing what it tests) but in principle also with uploads from dependencies. > Hence, the above would change the outcome if a package that previously > had no tests adds some and they fail, from "always failed" neutral to > "regression" delay. I'm not sure whether that's a good or bad thing. It is a bug (in britney) to trigger these references (see above for the MR). Once that is fixed, packages that add a test are truly initialized as failing, so if their first version of the autopkgtest fails, they aren't regressing. So in the end, after thinking about this while typing this e-mail (I may change my mind), I think we should keep neutral-to-fail a regression. Paul
signature.asc
Description: OpenPGP digital signature
