retitle 1059168 flaky autopkgtest user debian...@lists.debian.org usertag 1059168 flaky thanks BCC: control@b.d.o
Hi, On 21-12-2023 10:04, Jérémy Lal wrote:
Thank you for https://bugs.debian.org/1059168 <https://bugs.debian.org/1059168>I noticed "Marked as found in versions nodejs/18.19.0+dfsg-6 and reopened."while the test is flaky, retrying it will make it pass, so the bug is not as important as it seems. And indeed, https://tracker.debian.org/pkg/zlib <https://tracker.debian.org/pkg/zlib> is no longer blocked by nodejs.
It happens that I typically file bugs about flaky tests with severity serious with the following reasoning:
""" Because the unstable-to-testing migration software now blocks on regressions in testing, flaky tests, i.e. tests that flip between passing and failing without changes to the list of installed packages, are causing people unrelated to your package to spend time on these tests. """Having said that, I hardly ever fight over the severity if the maintainer(s) drop severity. It depends a bit on how flaky the test is, passing 1/3 or worse is really bad, passing more than 9/10 can be defended as acceptable but ideally should be fixed. My personal boundary for filing flaky bugs seems to lay around 1/5 to failures.
Paul
OpenPGP_signature.asc
Description: OpenPGP digital signature
