Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi, On 19-03-2022 14:48, Nilesh Patra wrote: For avoidance of doubt, the canonical definition lives in the autopkgtest package: https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst I usually refer /usr/share/doc/autopkgtest/README.package-tests.rst.gz in the autopkgtest package, I believe that's the same thingy. Agree, except it doesn't refer so nicely in on-line discussions *and* the on-line document may be more up-to-date than your local copy. But, you're right of course. And I would have recommended negative architectures: "-arm64" and try to test on all architectures not explicitly mentioned for the case where new architectures are added. That is what I always do, I add a negation as "!", which is also what I did in this case: https://salsa.debian.org/med-team/toil/-/blob/master/debian/tests/control#L8 Oops. I should use the right syntax if I teach something... Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
On Sat, Mar 19, 2022 at 02:32:12PM +0100, Paul Gevers wrote: > > [1] https://people.debian.org/~eriberto/README.package-tests.html > > For avoidance of doubt, the canonical definition lives in the autopkgtest > package: > https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst I usually refer /usr/share/doc/autopkgtest/README.package-tests.rst.gz in the autopkgtest package, I believe that's the same thingy. > And I would have recommended negative architectures: "-arm64" and try to > test on all architectures not explicitly mentioned for the case where new > architectures are added. That is what I always do, I add a negation as "!", which is also what I did in this case: https://salsa.debian.org/med-team/toil/-/blob/master/debian/tests/control#L8 Regards, Nilesh signature.asc Description: PGP signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi, On 19-03-2022 09:26, Sascha Steinbiss wrote: Would it be possible to add a hint to ignore arm64 autopkgtest suite? BTW I think this is possible already in the autopkgtest definition [1] by adding an Architecture: section and leaving out arm64 in the list of archs you list in there — if that is what you mean. Cheers Sascha [1] https://people.debian.org/~eriberto/README.package-tests.html For avoidance of doubt, the canonical definition lives in the autopkgtest package: https://salsa.debian.org/ci-team/autopkgtest/-/blob/master/doc/README.package-tests.rst And I would have recommended negative architectures: "-arm64" and try to test on all architectures not explicitly mentioned for the case where new architectures are added. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
On 19 March 2022 1:56:17 pm IST, Sascha Steinbiss wrote: >Hi Nilesh, > >[…] >> Would it be possible to add a hint to ignore arm64 autopkgtest suite? > >BTW I think this is possible already in the autopkgtest definition [1] by >adding an Architecture: section and leaving out arm64 in the list of archs you >list in there — if that is what you mean. That is what I eventually did. The package even migrated to testing now[2], along with cwltool. So all good, I guess. Regards, Nilesh >[1] https://people.debian.org/~eriberto/README.package-tests.html [2]: https://tracker.debian.org/news/1311322/toil-560-5-migrated-to-testing/
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi Nilesh, […] > Would it be possible to add a hint to ignore arm64 autopkgtest suite? BTW I think this is possible already in the autopkgtest definition [1] by adding an Architecture: section and leaving out arm64 in the list of archs you list in there — if that is what you mean. Cheers Sascha [1] https://people.debian.org/~eriberto/README.package-tests.html
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi again, Paul. On 3/10/22 6:28 PM, Nilesh Patra wrote: On Thu, Mar 10, 2022 at 01:09:41PM +0100, Paul Gevers wrote: I was finally able to make time and do this. I first ran admin@ci-worker08:/tmp/toil$ apt source toil/unstable admin@ci-worker08:/tmp/toil/toil-5.6.0$ autopkgtest --shell --shell-fail . -- lxc --sudo autopkgtest-unstable-i386 to confirm it triggers the issue. Than I put the try/except around the block you proposed (patch didn't apply). The test then ran OK. Thanks a lot, Paul! I uploaded with the patch. It passes everywhere now, except arm64[1]. Chokes with | File "/usr/lib/python3.9/http/client.py", line 302, in _read_status | raise BadStatusLine(line) | http.client.BadStatusLine: The request is too frequent. Please try again later! | [2022-03-11T03:59:28+] [MainThread] [E] [toil.worker] Exiting the worker because of a failed job on host ci-070-d6e3879e No idea why this is the case - not sure how to even debug this. It passes in the porter machine so I do not know what is up. It does not look like $arch specific to me but somewhat $worker-spec specific? Would it be possible to add a hint to ignore arm64 autopkgtest suite? Regards, Nilesh OpenPGP_signature Description: OpenPGP digital signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
On Thu, Mar 10, 2022 at 01:09:41PM +0100, Paul Gevers wrote: > I was finally able to make time and do this. I first ran > > admin@ci-worker08:/tmp/toil$ apt source toil/unstable > > admin@ci-worker08:/tmp/toil/toil-5.6.0$ autopkgtest --shell --shell-fail . > -- lxc --sudo autopkgtest-unstable-i386 > > to confirm it triggers the issue. Than I put the try/except around the block > you proposed (patch didn't apply). The test then ran OK. Thanks a lot, Paul! I uploaded with the patch. Regards, Nilesh signature.asc Description: PGP signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi, On 03-03-2022 21:35, Paul Gevers wrote: On 03-03-2022 21:01, Paul Gevers wrote: b) If you could bypass the unlink (patch pasted below to do so) and check once, that would be great That's a bit hard to do on the infra. I just started the test locally to see if it reproduces on my system. If it does, I'll try with the patch. Unfortunately it passes on my local system. I'll try again on ci.d.n infrastructure, but that will take a couple of days. I was finally able to make time and do this. I first ran admin@ci-worker08:/tmp/toil$ apt source toil/unstable admin@ci-worker08:/tmp/toil/toil-5.6.0$ autopkgtest --shell --shell-fail . -- lxc --sudo autopkgtest-unstable-i386 to confirm it triggers the issue. Than I put the try/except around the block you proposed (patch didn't apply). The test then ran OK. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi, On 03-03-2022 21:01, Paul Gevers wrote: b) If you could bypass the unlink (patch pasted below to do so) and check once, that would be great That's a bit hard to do on the infra. I just started the test locally to see if it reproduces on my system. If it does, I'll try with the patch. Unfortunately it passes on my local system. I'll try again on ci.d.n infrastructure, but that will take a couple of days. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi, On 02-03-2022 21:13, Nilesh Patra wrote: a) As per Sascha's idea[1], if you can simply run the test once, and check there is nothing(no daemon/service) removing files in /tmp/.. dir How would I do that? autopkgtest itself does that, but only of tries it just finished using. So I'm not sure what you're looking for. b) If you could bypass the unlink (patch pasted below to do so) and check once, that would be great That's a bit hard to do on the infra. I just started the test locally to see if it reproduces on my system. If it does, I'll try with the patch. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi Paul, On 3/2/22 11:51 PM, Paul Gevers wrote: I tried re-triggering it to check once again but looks like runners are not up unfortunately. You mean the salsa ones, right? Yes And so if you could trigger a test suite for this package at your end once, and help debug it, that'd be really nice. What do you propose I look for? I'm not very good at debugging random software. Me neither, but I could propose to check for two things: a) As per Sascha's idea[1], if you can simply run the test once, and check there is nothing(no daemon/service) removing files in /tmp/.. dir b) If you could bypass the unlink (patch pasted below to do so) and check once, that would be great [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1005479#10 --- a/src/toil/jobStores/fileJobStore.py +++ b/src/toil/jobStores/fileJobStore.py @@ -489,13 +489,16 @@ return except OSError as e: if e.errno == errno.EEXIST: -# Overwrite existing file, emulating shutil.copyfile(). -os.unlink(localFilePath) -# It would be very unlikely to fail again for same reason but possible -# nonetheless in which case we should just give up. -os.link(jobStoreFilePath, localFilePath) -# Now we succeeded and don't need to copy -return +try: +# Overwrite existing file, emulating shutil.copyfile(). +os.unlink(localFilePath) +# It would be very unlikely to fail again for same reason but possible +# nonetheless in which case we should just give up. +os.link(jobStoreFilePath, localFilePath) +# Now we succeeded and don't need to copy +return +except: +pass elif e.errno == errno.EXDEV: # It's a cross-device link even though it didn't appear to be. # Just keep going and hit the file copy case. OpenPGP_signature Description: OpenPGP digital signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi, On 02-03-2022 16:53, Nilesh Patra wrote: Sascha even pointed out that it goes fine on reproducible build machines, and this package runs same build time test as autopkgtest Yes, but normally not in lxc. The salsa pipeline was passing as well[1] which IIRC uses similar setup as debci. True, as far as I'm aware. I tried re-triggering it to check once again but looks like runners are not up unfortunately. You mean the salsa ones, right? ci.d.n runners should all be up. And so if you could trigger a test suite for this package at your end once, and help debug it, that'd be really nice. What do you propose I look for? I'm not very good at debugging random software. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi Paul, On 3/2/22 9:09 PM, Paul Gevers wrote: On 02-03-2022 14:58, Nilesh Patra wrote: @Paul, could you please help reproduce this in the debci environment and help us with any clues as to what might be wrong? The autopkgtest command line is at the top of each log. I am aware, and I use something $similar, but that is not working for me and Sascha locally, hence asked for help :) Sascha even pointed out that it goes fine on reproducible build machines, and this package runs same build time test as autopkgtest The salsa pipeline was passing as well[1] which IIRC uses similar setup as debci. I tried re-triggering it to check once again but looks like runners are not up unfortunately. And so if you could trigger a test suite for this package at your end once, and help debug it, that'd be really nice. [1]: https://salsa.debian.org/med-team/toil/-/jobs/2396200 Regards, Nilesh OpenPGP_signature Description: OpenPGP digital signature
Bug#1005479: toil: FTBFS: dh_auto_test: error: pybuild --test --test-pytest -i python{version} -p "3.10 3.9" returned exit code 13
Hi, On 02-03-2022 14:58, Nilesh Patra wrote: @Paul, could you please help reproduce this in the debci environment and help us with any clues as to what might be wrong? The autopkgtest command line is at the top of each log. | File "/usr/lib/python3/dist-packages/toil/jobStores/fileJobStore.py", line 498, in read_file |os.link(jobStoreFilePath, local_path) | OSError: [Errno 18] Invalid cross-device link: '/tmp/autopkgtest-lxc.rgt3f050/downtmp/build.YHo/src/jobstore-test-de9381ff-d8be-4b36-a691-be2c5fa759d2/files/no-job/file-d7145278c6b84845bd03cf5cff29e690/stream' -> '/home/debci/tmp/tmp1lgiu28c' > It looks (to me) that home and tmp are mounted on different partitions and os.link does not work with such a setting. I *think* the /tmp inside the lxc is mapped to /tmp of the host. /home is completely native to the container. (Not sure if all this matters). On *some* architectures /tmp and /var/lib/lxc of the host are on a separate partitions with respect to the rest. I'm pretty sure that for all our architectures all files inside the container are on the same partition. Paul OpenPGP_signature Description: OpenPGP digital signature
