Public bug reported: Some autopkgtests were running on autopkgtest-cloud-worker-lrg/7 when it ran out of disk space. From the test log we can see:
autopkgtest [16:44:34]: testing package golang-github-bep-overlayfs version 0.6.0-2 autopkgtest [16:44:34]: ERROR: unexpected error: OSError: [Errno 28] No space left on device During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 843, in main process_actions() File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 767, in process_actions tests_tree = build_source(kind, arg, built_binaries) File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 572, in build_source f.write('%s %s\n' % (testpkg_name, testpkg_version)) OSError: [Errno 28] No space left on device During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 855, in <module> main() File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 845, in main errorcode = print_exception(sys.exc_info(), '') File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 248, in print_exception adtlog.psummary('quitting: unexpected error, see log') File "/home/ubuntu/autopkgtest/lib/adtlog.py", line 101, in psummary summary_stream.write(m.encode('UTF-8')) OSError: [Errno 28] No space left on device And this the journal from around that time frame: Mar 22 16:44:30 lrg-root4 /home/ubuntu/autopkgtest-cloud/worker/worker[706687]: INFO: autopkgtest exited with code 20 Mar 22 16:44:30 lrg-root4 sh[706687]: OSError: [Errno 28] No space left on device Mar 22 16:44:30 lrg-root4 sh[706687]: During handling of the above exception, another exception occurred: Mar 22 16:44:30 lrg-root4 sh[706687]: Traceback (most recent call last): Mar 22 16:44:30 lrg-root4 sh[706687]: File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 1123, in <module> Mar 22 16:44:30 lrg-root4 sh[706687]: main() Mar 22 16:44:30 lrg-root4 sh[706687]: File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 1116, in main Mar 22 16:44:30 lrg-root4 sh[706687]: queue.wait() Mar 22 16:44:30 lrg-root4 sh[706687]: File "/usr/lib/python3/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait Mar 22 16:44:30 lrg-root4 sh[706687]: return self.dispatch_method(method_sig, args, content) Mar 22 16:44:30 lrg-root4 sh[706687]: File "/usr/lib/python3/dist-packages/amqplib/client_0_8/abstract_channel.py", line 117, in dispatc h_method Mar 22 16:44:30 lrg-root4 sh[706687]: return amqp_method(self, args, content) Mar 22 16:44:30 lrg-root4 sh[706687]: File "/usr/lib/python3/dist-packages/amqplib/client_0_8/channel.py", line 2060, in _basic_deliver Mar 22 16:44:30 lrg-root4 sh[706687]: func(msg) Mar 22 16:44:30 lrg-root4 sh[706687]: File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 882, in request Mar 22 16:44:30 lrg-root4 sh[706687]: f.write('%i\n' % code) Mar 22 16:44:30 lrg-root4 sh[706687]: OSError: [Errno 28] No space left on device Mar 22 16:44:30 lrg-root4 systemd[1]: autopkgtest@bos02-arm64-21.service: Main process exited, code=exited, status=1/FAILURE As I see it there are two problems here: 1) the log files are left in /tmp/autopkgtest-work.$TEMP/out/ 2) the autopkgtest process itself can still be left running e.g. ubuntu 716051 0.0 0.0 45316 20736 ? S Mar22 0:00 /usr/bin/python3 -u /home/ubuntu/autopkgtest/runner/autopkgtest --outpu t-dir /tmp/autopkgtest-work.d5zzl576/out --timeout-copy=6000 -a i386 --setup-commands /home/ubuntu/autopkgtest-cloud/worker-config-product ion/setup-canonical.sh --setup-commands /home/ubuntu/autopkgtest/setup-commands/setup-testbed --apt-pocket=proposed=src:golang-golang-x-to ols --apt-upgrade golang-github-bep-overlayfs --timeout-short=300 --timeout-copy=20000 --timeout-build=20000 --env=ADT_TEST_TRIGGERS=golan g-golang-x-tools/1:0.5.0+ds-2 -- ssh -s /home/ubuntu/autopkgtest/ssh-setup/nova -- --flavor autopkgtest --security-groups autopkgtest-lrg- root4@lgw01-13.secgroup --name adt-lunar-i386-golang-github-bep-overlayfs-20230322-162349-lrg-root4 --image adt/ubuntu-lunar-amd64-server --keyname testbed-lrg-root4 --net-id=net_prod-proposed-migration -e TERM=linux -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy= http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,ports.ubuntu.com, security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpad.net,10.24.0.0/24' --mirror=http://ftpmaster.internal/ubuntu/ ** Affects: auto-package-testing Importance: Undecided Status: New -- You received this bug notification because you are a member of Canonical's Ubuntu QA, which is subscribed to Auto Package Testing. https://bugs.launchpad.net/bugs/2012667 Title: autopkgtest-cloud running out of space is not handled well Status in Auto Package Testing: New Bug description: Some autopkgtests were running on autopkgtest-cloud-worker-lrg/7 when it ran out of disk space. From the test log we can see: autopkgtest [16:44:34]: testing package golang-github-bep-overlayfs version 0.6.0-2 autopkgtest [16:44:34]: ERROR: unexpected error: OSError: [Errno 28] No space left on device During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 843, in main process_actions() File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 767, in process_actions tests_tree = build_source(kind, arg, built_binaries) File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 572, in build_source f.write('%s %s\n' % (testpkg_name, testpkg_version)) OSError: [Errno 28] No space left on device During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 855, in <module> main() File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 845, in main errorcode = print_exception(sys.exc_info(), '') File "/home/ubuntu/autopkgtest/runner/autopkgtest", line 248, in print_exception adtlog.psummary('quitting: unexpected error, see log') File "/home/ubuntu/autopkgtest/lib/adtlog.py", line 101, in psummary summary_stream.write(m.encode('UTF-8')) OSError: [Errno 28] No space left on device And this the journal from around that time frame: Mar 22 16:44:30 lrg-root4 /home/ubuntu/autopkgtest-cloud/worker/worker[706687]: INFO: autopkgtest exited with code 20 Mar 22 16:44:30 lrg-root4 sh[706687]: OSError: [Errno 28] No space left on device Mar 22 16:44:30 lrg-root4 sh[706687]: During handling of the above exception, another exception occurred: Mar 22 16:44:30 lrg-root4 sh[706687]: Traceback (most recent call last): Mar 22 16:44:30 lrg-root4 sh[706687]: File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 1123, in <module> Mar 22 16:44:30 lrg-root4 sh[706687]: main() Mar 22 16:44:30 lrg-root4 sh[706687]: File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 1116, in main Mar 22 16:44:30 lrg-root4 sh[706687]: queue.wait() Mar 22 16:44:30 lrg-root4 sh[706687]: File "/usr/lib/python3/dist-packages/amqplib/client_0_8/abstract_channel.py", line 97, in wait Mar 22 16:44:30 lrg-root4 sh[706687]: return self.dispatch_method(method_sig, args, content) Mar 22 16:44:30 lrg-root4 sh[706687]: File "/usr/lib/python3/dist-packages/amqplib/client_0_8/abstract_channel.py", line 117, in dispatc h_method Mar 22 16:44:30 lrg-root4 sh[706687]: return amqp_method(self, args, content) Mar 22 16:44:30 lrg-root4 sh[706687]: File "/usr/lib/python3/dist-packages/amqplib/client_0_8/channel.py", line 2060, in _basic_deliver Mar 22 16:44:30 lrg-root4 sh[706687]: func(msg) Mar 22 16:44:30 lrg-root4 sh[706687]: File "/home/ubuntu/autopkgtest-cloud/worker/worker", line 882, in request Mar 22 16:44:30 lrg-root4 sh[706687]: f.write('%i\n' % code) Mar 22 16:44:30 lrg-root4 sh[706687]: OSError: [Errno 28] No space left on device Mar 22 16:44:30 lrg-root4 systemd[1]: autopkgtest@bos02-arm64-21.service: Main process exited, code=exited, status=1/FAILURE As I see it there are two problems here: 1) the log files are left in /tmp/autopkgtest-work.$TEMP/out/ 2) the autopkgtest process itself can still be left running e.g. ubuntu 716051 0.0 0.0 45316 20736 ? S Mar22 0:00 /usr/bin/python3 -u /home/ubuntu/autopkgtest/runner/autopkgtest --outpu t-dir /tmp/autopkgtest-work.d5zzl576/out --timeout-copy=6000 -a i386 --setup-commands /home/ubuntu/autopkgtest-cloud/worker-config-product ion/setup-canonical.sh --setup-commands /home/ubuntu/autopkgtest/setup-commands/setup-testbed --apt-pocket=proposed=src:golang-golang-x-to ols --apt-upgrade golang-github-bep-overlayfs --timeout-short=300 --timeout-copy=20000 --timeout-build=20000 --env=ADT_TEST_TRIGGERS=golan g-golang-x-tools/1:0.5.0+ds-2 -- ssh -s /home/ubuntu/autopkgtest/ssh-setup/nova -- --flavor autopkgtest --security-groups autopkgtest-lrg- root4@lgw01-13.secgroup --name adt-lunar-i386-golang-github-bep-overlayfs-20230322-162349-lrg-root4 --image adt/ubuntu-lunar-amd64-server --keyname testbed-lrg-root4 --net-id=net_prod-proposed-migration -e TERM=linux -e 'http_proxy=http://squid.internal:3128' -e 'https_proxy= http://squid.internal:3128' -e 'no_proxy=127.0.0.1,127.0.1.1,localhost,localdomain,novalocal,internal,archive.ubuntu.com,ports.ubuntu.com, security.ubuntu.com,ddebs.ubuntu.com,changelogs.ubuntu.com,launchpad.net,10.24.0.0/24' --mirror=http://ftpmaster.internal/ubuntu/ To manage notifications about this bug go to: https://bugs.launchpad.net/auto-package-testing/+bug/2012667/+subscriptions -- Mailing list: https://launchpad.net/~canonical-ubuntu-qa Post to : canonical-ubuntu-qa@lists.launchpad.net Unsubscribe : https://launchpad.net/~canonical-ubuntu-qa More help : https://help.launchpad.net/ListHelp