> >> As elixir-lang is going to be AUTORM if nobody takes care of it, I > >> already added a bit of code in debian/rules to make it retry unit tests > >> if they fail. It will retry 3 times, meaning that if it used to fail 10% > >> of the time, making it retry 3 times makes the unit tests fail 0.01% of > >> the times, which is IMO acceptable. The recent failures on some arch > >> shows that my debian/rules hack works well! :) > > > > Please don't do that sort of thing. It makes things gratuitously painful > > for people like me doing QA. > > > > Either a unit test is to be trusted or it's not (by "trusted" I mean > > here that a failure in the test means the package is unsuitable for > > release). > > > > If the test is trusted but it fails, accept that it fails and do not > > let the package to propagate to testing. > > > > If it's not trusted, there is no point in making it to fail with any > > probability. Not 10%, and not 0.01%. Just disable the test. > > > > Thanks. > > I don't agree with this view. Yes, this makes your work harder. But > running tests improve the quality of the package. Removing them is in no > way an improvement.
My complain here is specifically about artificially reducing the probability of a random test to fail. Remember that tests only improve the quality of the package when you actually do something when they fail. If you artificially reduce the probability and do nothing about failing tests, there is no increase of quality at all. [ Please Cc me on any replies, I'm not monitoring this bug ] Thanks.
