On Sun, 12 Apr 2020 11:21:29 +0200 Michał Górny <mgo...@gentoo.org> wrote:
> On Sun, 2020-04-12 at 10:43 +0200, Agostino Sarubbo wrote: > > Hello all, > > > > If you work on the stabilization workflow you may have noticed that: > > > > - There are people that rant if you don't run src_test against their > > packages; > > - There are people that rant if you open a test failure bug against their > > packages and you block the stabilization. > > > > So, unless there will be a clear policy about that, I'd suggest to > > introduce a > > way to establish if a package is expected to pass src_test or not. > > > > A simple way to achieve this goal would be: > > introduce a new bugzilla custom flag (like "Runtime testing required" we > > already have) maybe called "src_test pass" or similar, that, by default is > > yes > > and the maintainer should set it to no when needed. > > > > In case of 'yes', the arch team member must compile with FEATURE="test" and > > he > > is allowed to block the stabilization in case of test-failure. > > > > In case there will be a test-failure there are two choices: > > 1) The maintainer fixes the test failure; > > 2) The maintainer does not want to take care, so he can simply remove the > > blocker and set "src_test pass" to no. > > > > Both of the above are fine for me. > > > > Obviously, if there are already test-failure bugs open, the flag "src_test > > pass" should be set to 'no' when the stabilization bugs is filled. > > > > This is not a problem that can be solved by a binary flag. > > If package's test suite is entirely broken and unmaintained, the package > should use RESTRICT=test and not silently ask arch teams to ignore it. > > If package's test suite is only slightly broken, then I'd prefer saying > 'please report but ignore *these* test failures' because I can't fix > them right now but they don't seem major. Skipping the test suite > entirely is not a solution because it doesn't disambiguate between 'few > tests fail' and 'every single test explodes'. > I agree with this, the metric for marking an ebuild as "tests don't work, please ignore them is RESTRICT=test". Usually for packages that have a *few* tests that fail and don't seem major, I would suggest trying to patch out the tests, or ask the test harness to skip the known bad tests. This way the tinderbox will (hopefully) still succeed.
