severity 1117471 serious thanks On Mon, Oct 06, 2025 at 09:25:19AM +0200, Simon Josefsson wrote: > [...] > > For flaky tests, I usually report these upstreams and mask it out of the > Debian build. I initially thought keeping flaky tests around was > acceptable, but I've realized it causes serious Debian testing migration > pains so I believe we must not have any flaky tests in Debian.
They are indeed a pain in the ass, to the point that a few years ago release managers announced that flaky tests should be considered RC. So, if you see a flaky test somewhere, please don't be shy and report it as "important" at least :-) For this package, I'm getting a failure rate of 40% on machines with 1 CPU and also 40% on machines with 2 CPUs (after trying 25 times in each case), neither of which I would consider acceptable for a stable release, so I'm raising the severity accordingly. I've put a small collection of failed build logs here: https://people.debian.org/~sanvila/build-logs/202609/ I see that last year Otto tried to improve the status of some tests. I'm not sure if doing that is worth the effort for this particular package, because the upstream repository is archived: https://github.com/google/goexpect So: what are your current thoughts about this? Note that the failing tests are not always the same, but the following two happen quite often: TestSendSignal TestSpawnWithArgs My preferred course of action here would be to disable the above two tests. I volunteer to do that, because those tests are noise in my own building environment. However, if somebody still wants to spend time fixing them, they are welcome of course. > Here is an idea: > > override_dh_auto_test: > dh_auto_test $(DH_BUILD_OPTS) -- -skip=ExampleGExpect_SendSignal > -dh_auto_test $(DH_BUILD_OPTS) > > This allows the failure message to still become available in build logs, > for future references in bug reports. I see the idea of "preserving information" behind that, but it has a price: it makes debian/rules a little bit more longer and therefore a little bit more difficult to understand and maintain. (So, personally, I don't particularly like such construct and would prefer the simple approach of skipping flaky tests where necessary) > If it only has happened on a particular arch, here is an idea: > > ifneq (,$(filter $(DEB_TARGET_ARCH), ppc64el)) > override_dh_auto_test: > dh_auto_test $(DH_BUILD_OPTS) -- -skip=ExampleGExpect_SendSignal > -dh_auto_test $(DH_BUILD_OPTS) > endif Note that this can be tricky. It may be the case that the test has only failed *so far* on a particular architecture, but that may happen for a lot of different reasons. If the failure is not *inherent* to the architecture, and it happens just because the test is wrongly designed, skipping the test for everybody might be a better option. > For complex workarounds I have used this: > > ifneq (,$(filter $(DEB_TARGET_ARCH), i386 ppc64el riscv64 s390x)) > override_dh_auto_test: > ifneq (,$(filter $(DEB_TARGET_ARCH), i386)) > dh_auto_test $(DH_BUILD_OPT) -- --skip=TestAllocate > endif > ifneq (,$(filter $(DEB_TARGET_ARCH), ppc64el riscv64 s390x)) > dh_auto_test $(DH_BUILD_OPT) -- --skip='TestTreePersistent|TestBuffer' > endif > -dh_auto_test $(DH_BUILD_OPT) -- --timeout=20m > endif As before, the more complex this becomes, the more complex it will be to read it, understand it, or maintaing it. In my opinion, we are already good enough citizens in the FOSS world when we forward the bug upstream. After that, the issue is in the hands of the authors, and we have the "right", so to speak, not to make things more complex than required for ourselves. [ sorry for the long email... ] Thanks.
