Bug#986256: [Debian-med-packaging] Bug#986256: simka: flaky amd64 autopkgtest: regularly times out after 2:47 h
Hi Graham, Graham Inggs, on 2021-04-06 20:15:30 +0200: > This is still occurring with simka 1.5.3-3, see: > https://ci.debian.net/packages/s/simka/testing/amd64/ Thanks for having noticed it. It looks like I got caught by the Python script not taking the -nb-cores argument, contrary to the shell script. While I couldn't reproduce the hang on emulator, it seems I can just reproduce the problem by enforcing the cores count to 48 within the Python script. The test is precisely hanging at this location: [...] Command for visualizing results with metadata annotations: python ../scripts/visualization/run-visualization.py -in ./simka_results/ -out ./simka_results/ -pca -heatmap -tree -metadata-in ../example/dataset_metadata.csv -metadata-variable VARIABLE_1 TESTING k=31 t=0 simka -in simka_input.txt -out ./__results__/results_k31_t0 -out-tmp ./temp_output -simple-dist -complex-dist -kmer-size 31 -abundance-min 0 -verbose 0 -nb-cores 48 OK TESTING k=21 t=0 simka -in simka_input.txt -out ./__results__/results_k21_t0 -out-tmp ./temp_output -simple-dist -complex-dist -kmer-size 21 -abundance-min 0 -verbose 0 -nb-cores 48 OK TESTING k=31 t=2 simka -in simka_input.txt -out ./__results__/results_k31_t2 -out-tmp ./temp_output -simple-dist -complex-dist -kmer-size 31 -abundance-min 2 -verbose 0 -nb-cores 48 _ The output on CI infrastructure is a bit misleading, because the standard output is not flushed on each output line by default. I enforced flushed output manually in the Python script to have a cleared view. But I /believe/ I could reproduce precisely the deadlock this time. Hopefully a further upload with controlled cores count at this level should do the trick. Have a nice day, :) -- Étienne Mollier Fingerprint: 8f91 b227 c7d6 f2b1 948c 8236 793c f67e 8f0d 11da Sent from /dev/pts/2, please excuse my verbosity. signature.asc Description: PGP signature
Bug#986256: simka: flaky amd64 autopkgtest: regularly times out after 2:47 h
Control: reopen -1 This is still occurring with simka 1.5.3-3, see: https://ci.debian.net/packages/s/simka/testing/amd64/
Bug#986256: simka: flaky amd64 autopkgtest: regularly times out after 2:47 h
Hi Nilesh, On 02-04-2021 13:40, Nilesh Patra wrote: > Hi, > > On Thu, 1 Apr 2021 20:53:00 +0200 Paul Gevers wrote: > >> Your package has an autopkgtest, great. However, I looked into >> the history of your autopkgtest [1] and I noticed version 1.5.3-2 fails >> regularly on amd64, while sporadically a rerun passes. I copied some of >> the output at the bottom of this report. It hits the autopkgtest time >> out after 2hours and 47 minutes. Successful runs pass in less than a minute. >> >> Because the unstable-to-testing migration software now blocks on >> regressions in testing, flaky tests, i.e. tests that flip between >> passing and failing without changes to the list of installed packages, >> are causing people unrelated to your package to spend time on these >> tests. > > That makes sense - do you think marking this test as flaky can be > solution? No, because when a test times out, that restriction doesn't work. You'll need to keep the test below 2:47, and as it normally takes less than a minute, it may point at something seriously hanging. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#986256: simka: flaky amd64 autopkgtest: regularly times out after 2:47 h
Hi, On Thu, 1 Apr 2021 20:53:00 +0200 Paul Gevers wrote: > Your package has an autopkgtest, great. However, I looked into > the history of your autopkgtest [1] and I noticed version 1.5.3-2 fails > regularly on amd64, while sporadically a rerun passes. I copied some of > the output at the bottom of this report. It hits the autopkgtest time > out after 2hours and 47 minutes. Successful runs pass in less than a minute. > > Because the unstable-to-testing migration software now blocks on > regressions in testing, flaky tests, i.e. tests that flip between > passing and failing without changes to the list of installed packages, > are causing people unrelated to your package to spend time on these > tests. That makes sense - do you think marking this test as flaky can be solution? Nilesh signature.asc Description: PGP signature
Bug#986256: simka: flaky amd64 autopkgtest: regularly times out after 2:47 h
Source: simka Version: 1.5.3-2 Severity: serious Tags: sid bullseye X-Debbugs-CC: debian...@lists.debian.org User: debian...@lists.debian.org Usertags: flaky timeout Dear maintainer(s), Your package has an autopkgtest, great. However, I looked into the history of your autopkgtest [1] and I noticed version 1.5.3-2 fails regularly on amd64, while sporadically a rerun passes. I copied some of the output at the bottom of this report. It hits the autopkgtest time out after 2hours and 47 minutes. Successful runs pass in less than a minute. Because the unstable-to-testing migration software now blocks on regressions in testing, flaky tests, i.e. tests that flip between passing and failing without changes to the list of installed packages, are causing people unrelated to your package to spend time on these tests. Paul [1] https://ci.debian.net/packages/s/simka/testing/amd64/ https://ci.debian.net/data/autopkgtest/testing/amd64/s/simka/11355132/log.gz Computing stats... Stats Reads Total:8470M0G Min:980M0G Max:4020M0G Average:1690M0G Kmers Distinct Kmers (before merging):146700M0G Distinct Kmers (after merging):81230M0G Shared distinct Kmers:43220M0G Kmers:501300M0G Mean k-mer coverage: 2.66341 Output dir: ./simka_results/ *** Test: PASSED Command used: /usr/bin/simka -in ../example/simka_input.txt -out ./simka_results/ -out-tmp ./simka_temp_output Command for visualizing results: python ../scripts/visualization/run-visualization.py -in ./simka_results/ -out ./simka_results/ -pca -heatmap -tree Command for visualizing results with metadata annotations: python ../scripts/visualization/run-visualization.py -in ./simka_results/ -out ./simka_results/ -pca -heatmap -tree -metadata-in ../example/dataset_metadata.csv -metadata-variable VARIABLE_1 autopkgtest [07:42:20]: ERROR: timed out on command "su -s /bin/bash debci -c set -e; export USER=`id -nu`; . /etc/profile >/dev/null 2>&1 || true; . ~/.profile >/dev/null 2>&1 || true; buildtree="/tmp/autopkgtest-lxc.zk5p_5wg/downtmp/build.NZn/src"; mkdir -p -m 1777 -- "/tmp/autopkgtest-lxc.zk5p_5wg/downtmp/run-unit-test-artifacts"; export AUTOPKGTEST_ARTIFACTS="/tmp/autopkgtest-lxc.zk5p_5wg/downtmp/run-unit-test-artifacts"; export ADT_ARTIFACTS="$AUTOPKGTEST_ARTIFACTS"; mkdir -p -m 755 "/tmp/autopkgtest-lxc.zk5p_5wg/downtmp/autopkgtest_tmp"; export AUTOPKGTEST_TMP="/tmp/autopkgtest-lxc.zk5p_5wg/downtmp/autopkgtest_tmp"; export ADTTMP="$AUTOPKGTEST_TMP"; export DEBIAN_FRONTEND=noninteractive; export LANG=C.UTF-8; export DEB_BUILD_OPTIONS=parallel=48; unset LANGUAGE LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT LC_IDENTIFICATION LC_ALL;rm -f /tmp/autopkgtest_script_pid; set -C; echo $$ > /tmp/autopkgtest_script_pid; set +C; trap "rm -f /tmp/autopkgtest_script_pid" EXIT INT QUIT PIPE; cd "$buildtree"; chmod +x /tmp/autopkgtest-lxc.zk5p_5wg/downtmp/build.NZn/src/debian/tests/run-unit-test; touch /tmp/autopkgtest-lxc.zk5p_5wg/downtmp/run-unit-test-stdout /tmp/autopkgtest-lxc.zk5p_5wg/downtmp/run-unit-test-stderr; /tmp/autopkgtest-lxc.zk5p_5wg/downtmp/build.NZn/src/debian/tests/run-unit-test 2> >(tee -a /tmp/autopkgtest-lxc.zk5p_5wg/downtmp/run-unit-test-stderr >&2) > >(tee -a /tmp/autopkgtest-lxc.zk5p_5wg/downtmp/run-unit-test-stdout);" (kind: test) autopkgtest [07:42:20]: test run-unit-test: ---] OpenPGP_signature Description: OpenPGP digital signature
