Bug#1055639: surf: flaky autopkgtest: Too few characters detected (0)
Hi, On 09-12-2023 13:08, Reiner Herrmann wrote: I thought as autopkgtest is installed on systems running the tests, I could use it. But it's not installed in the testbed, so it gets installed only for this reason. And because your test depends on it, there is now also a Testsuite-Trigger relation. Paul OpenPGP_signature.asc Description: OpenPGP digital signature
Bug#1055639: surf: flaky autopkgtest: Too few characters detected (0)
Hi Paul, On Thu, Nov 09, 2023 at 02:16:47PM +0100, Paul Gevers wrote: > I looked at the results of the autopkgtest of your package. I noticed that > it regularly fails (in this case because it's blocking migration of > src:autopkgtest), mostly on armhf and a bit on ppc64el and s390x. > > Because the unstable-to-testing migration software now blocks on > regressions in testing, flaky tests, i.e. tests that flip between > passing and failing without changes to the list of installed packages, > are causing people unrelated to your package to spend time on these > tests. The fails don't look regular to me. Most of the times they are passing. It's difficult to figure out why it sometimes doesn't draw anything on the weaker architectures (the screenshot that is used for OCR is blank). I'm going to mark the test as flaky now. > PS: why does it even use text from a different and very unrelated package? > If there's not enough text in it's own source, couldn't it use something > that's installed on all Debian systems, such that it doesn't need to be > installed additionally and trigger migration runs? The autopkgtest documentation is available in HTML and plaintext (rst) format. That's needed to compare the OCR'd HTML-text against. It was the first example I could find, and I thought as autopkgtest is installed on systems running the tests, I could use it. If you are aware of other packages that are installed by default that ship documentation in HTML and some plaintext, then I could switch to it in the test. Kind regards, Reiner
Bug#1055639: surf: flaky autopkgtest: Too few characters detected (0)
Source: surf Version: 2.1+git20221016-5 Severity: serious User: debian...@lists.debian.org Usertags: flaky Dear maintainer(s), I looked at the results of the autopkgtest of your package. I noticed that it regularly fails (in this case because it's blocking migration of src:autopkgtest), mostly on armhf and a bit on ppc64el and s390x. Because the unstable-to-testing migration software now blocks on regressions in testing, flaky tests, i.e. tests that flip between passing and failing without changes to the list of installed packages, are causing people unrelated to your package to spend time on these tests. Don't hesitate to reach out if you need help and some more information from our infrastructure. Paul PS: why does it even use text from a different and very unrelated package? If there's not enough text in it's own source, couldn't it use something that's installed on all Debian systems, such that it doesn't need to be installed additionally and trigger migration runs? https://ci.debian.net/data/autopkgtest/testing/s390x/s/surf/39724716/log.gz 655s autopkgtest [12:16:40]: test command3: timeout -v 5m xvfb-run debian/tests/test_text.sh 655s autopkgtest [12:16:40]: test command3: [--- 657s 657s (surf:7172): dbind-WARNING **: 12:16:42.983: AT-SPI: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files 658s Could not determine the accessibility bus address 662s Too few characters detected (0) 662s autopkgtest [12:16:47]: test command3: ---] OpenPGP_signature.asc Description: OpenPGP digital signature
