Bug#1016070: sbuild: autopkgtest fails with "No space left on device"
Quoting Paul Gevers (2022-07-26 22:24:55) > That worker (ci-worker13) has 7 TB of disk available, so space shouldn't be > the problem. I'm also not seeing [1] the disk usage anytime higher than 7% > (and it's actually that /mnt/lxc-containers that's being used and that > doesn't peak above 1.2% since we changed hosts). Yes, disk space is not a problem. The failing test is unshare-qemuwrapper which tests the unshare backend of sbuild. Since neither the salsaci nor the debci autopkgtest runners allow unsharing the user namespace, the autopkgtest creates a qemu virtual machine and the test is then run inside of qemu. Increasing the disk size of that qemu virtual machine is easy but I first want to confirm that the buildd chroot growing by half a GB is intended and not a bug that this test found. signature.asc Description: signature
Bug#1016070: sbuild: autopkgtest fails with "No space left on device"
Hi, On 26-07-2022 13:28, Luca Boccassi wrote: The autopkgtest run for sbuild keeps failing in debci, blocking other packages migrations. The failure manifests in two different error types, but both related to "No space left on device" when running mmdebstrap. This seems to have started on July the 22nd. List of recent failures: https://ci.debian.net/packages/s/sbuild/testing/amd64/ Eg: https://ci.debian.net/data/autopkgtest/unstable/amd64/s/sbuild/23951901/log.gz + mmdebstrap --mode=unshare --variant=apt unstable /tmp/chroot.tar I: chroot architecture amd64 is equal to the host's architecture I: automatically chosen format: tar I: using /tmp/mmdebstrap.1Pi2WSLk6H as tempdir I: running apt-get update... I: downloading packages with apt... I: extracting archives... I: installing essential packages... Selecting previously unselected package gcc-12-base:amd64. (Reading database ... 0 files and directories currently installed.) Preparing to unpack .../gcc-12-base_12.1.0-7_amd64.deb ... Unpacking gcc-12-base:amd64 (12.1.0-7) ... Selecting previously unselected package libc6:amd64. Preparing to unpack .../libc6_2.33-8_amd64.deb ... Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/Debconf/Config.pm line 22. Unpacking libc6:amd64 (2.33-8) ... dpkg: error processing archive /var/cache/apt/archives/libc6_2.33-8_amd64.deb (--install): cannot copy extracted data for './usr/lib/x86_64-linux-gnu/gconv/IBM904.so' to '/usr/lib/x86_64-linux-gnu/gconv/IBM904.so.dpkg-new': failed to write (No space left on device) That worker (ci-worker13) has 7 TB of disk available, so space shouldn't be the problem. I'm also not seeing [1] the disk usage anytime higher than 7% (and it's actually that /mnt/lxc-containers that's being used and that doesn't peak above 1.2% since we changed hosts). Paul [1] https://ci.debian.net/munin/ci-worker13/ci-worker13/df.html OpenPGP_signature Description: OpenPGP digital signature
Bug#1016070: sbuild: autopkgtest fails with "No space left on device"
On Tue, 2022-07-26 at 15:43 +0200, Johannes Schauer Marin Rodrigues wrote: > Hi, > > Quoting Luca Boccassi (2022-07-26 15:27:35) > > If it's not appropriate, please do update it accordingly, but IIRC it's what > > gets used in these cases. > > a bunch of sbuild issues piled up during the last weeks so I'll be doing an > upload soon and will include a fix for this as well. > > > > I put the gcc maintainer in CC. Is the buildd tarball to be more than > > > twice > > > the size compared to before now? > > > > Good find - regardless of whether it's intended that the tarball is so > > large, perhaps it is an indication that the sbuild testsuite is a bit > > fragile w.r.t. the running environment? Would it be possible to adjust > > it to be more resilient? Is there a different disk-based scratch area > > available for test artifacts? > > the autopkgtest checks whether the sbuild unshare backend works. The > environment on salsaci and the one on ci.debian.net does not support Linux > user > namespaces. To still be able to test it, the autopkgtest creates a virtual > qemu > environment and runs the test inside a qemu virtual machine. The size of the > machine image is the limiting factor here. > > The virtual machine image size was not a problem since the introduction of > this > test in January 2021, so I wouldn't call it "fragile". Increasing the disk > size > is simple but while you call such an adjustment to make it more "resilient" I > first want to confirm that the more than twice increase in size is intentional > or not. Otherwise, changing the disk size might have the opposite effect of > "resilient" and hide problems resulting from an accidental upload which > unnecessarily increased the installation size by half a Gigabyte. > > Thanks! > > cheers, josch Right, I've misunderstood the issue, from a cursory and uninformed look at the logs it looked like it was setting up a chroot in /tmp and it was running out of space on the host. Thanks for looking into this! -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Bug#1016070: sbuild: autopkgtest fails with "No space left on device"
Hi, Quoting Luca Boccassi (2022-07-26 15:27:35) > If it's not appropriate, please do update it accordingly, but IIRC it's what > gets used in these cases. a bunch of sbuild issues piled up during the last weeks so I'll be doing an upload soon and will include a fix for this as well. > > I put the gcc maintainer in CC. Is the buildd tarball to be more than twice > > the size compared to before now? > > Good find - regardless of whether it's intended that the tarball is so > large, perhaps it is an indication that the sbuild testsuite is a bit > fragile w.r.t. the running environment? Would it be possible to adjust > it to be more resilient? Is there a different disk-based scratch area > available for test artifacts? the autopkgtest checks whether the sbuild unshare backend works. The environment on salsaci and the one on ci.debian.net does not support Linux user namespaces. To still be able to test it, the autopkgtest creates a virtual qemu environment and runs the test inside a qemu virtual machine. The size of the machine image is the limiting factor here. The virtual machine image size was not a problem since the introduction of this test in January 2021, so I wouldn't call it "fragile". Increasing the disk size is simple but while you call such an adjustment to make it more "resilient" I first want to confirm that the more than twice increase in size is intentional or not. Otherwise, changing the disk size might have the opposite effect of "resilient" and hide problems resulting from an accidental upload which unnecessarily increased the installation size by half a Gigabyte. Thanks! cheers, josch signature.asc Description: signature
Bug#1016070: sbuild: autopkgtest fails with "No space left on device"
On Tue, 2022-07-26 at 15:15 +0200, Johannes Schauer Marin Rodrigues wrote: > Hi, > > thank you for your bug report! > > Quoting Luca Boccassi (2022-07-26 13:28:50) > > Severity: serious > > why is an autopkgtest failure "serious"? Because it blocks other packages from migrating to testing, e.g.: https://tracker.debian.org/pkg/apt If it's not appropriate, please do update it accordingly, but IIRC it's what gets used in these cases. > > The autopkgtest run for sbuild keeps failing in debci, blocking other > > packages migrations. The failure manifests in two different error > > types, but both related to "No space left on device" when running > > mmdebstrap. > > The reason for this is the upload of gcc-defaults 1.198 to unstable. I > bisected > Debian unstable. On 2022-07-22T08:51:38Z a buildd chroot tarball was 396 MB > small. Starting with snapshot timestamp 2022-07-22T15:09:35Z (the first > timestamp with the new gcc-defaults version) a buildd chroot tarball is now > 906 > MB large. > > I put the gcc maintainer in CC. Is the buildd tarball to be more than twice > the > size compared to before now? > > Thanks! Good find - regardless of whether it's intended that the tarball is so large, perhaps it is an indication that the sbuild testsuite is a bit fragile w.r.t. the running environment? Would it be possible to adjust it to be more resilient? Is there a different disk-based scratch area available for test artifacts? -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Bug#1016070: sbuild: autopkgtest fails with "No space left on device"
Hi, thank you for your bug report! Quoting Luca Boccassi (2022-07-26 13:28:50) > Severity: serious why is an autopkgtest failure "serious"? > The autopkgtest run for sbuild keeps failing in debci, blocking other > packages migrations. The failure manifests in two different error > types, but both related to "No space left on device" when running > mmdebstrap. The reason for this is the upload of gcc-defaults 1.198 to unstable. I bisected Debian unstable. On 2022-07-22T08:51:38Z a buildd chroot tarball was 396 MB small. Starting with snapshot timestamp 2022-07-22T15:09:35Z (the first timestamp with the new gcc-defaults version) a buildd chroot tarball is now 906 MB large. I put the gcc maintainer in CC. Is the buildd tarball to be more than twice the size compared to before now? Thanks! cheers, josch signature.asc Description: signature
Bug#1016070: sbuild: autopkgtest fails with "No space left on device"
Source: sbuild Version: 0.83.1 Severity: serious X-Debbugs-Cc: jo...@debian.org, debian...@lists.debian.org Dear Maintainer(s), The autopkgtest run for sbuild keeps failing in debci, blocking other packages migrations. The failure manifests in two different error types, but both related to "No space left on device" when running mmdebstrap. This seems to have started on July the 22nd. List of recent failures: https://ci.debian.net/packages/s/sbuild/testing/amd64/ Eg: https://ci.debian.net/data/autopkgtest/unstable/amd64/s/sbuild/23951901/log.gz + mmdebstrap --mode=unshare --variant=apt unstable /tmp/chroot.tar I: chroot architecture amd64 is equal to the host's architecture I: automatically chosen format: tar I: using /tmp/mmdebstrap.1Pi2WSLk6H as tempdir I: running apt-get update... I: downloading packages with apt... I: extracting archives... I: installing essential packages... Selecting previously unselected package gcc-12-base:amd64. (Reading database ... 0 files and directories currently installed.) Preparing to unpack .../gcc-12-base_12.1.0-7_amd64.deb ... Unpacking gcc-12-base:amd64 (12.1.0-7) ... Selecting previously unselected package libc6:amd64. Preparing to unpack .../libc6_2.33-8_amd64.deb ... Use of uninitialized value in concatenation (.) or string at /usr/share/perl5/Debconf/Config.pm line 22. Unpacking libc6:amd64 (2.33-8) ... dpkg: error processing archive /var/cache/apt/archives/libc6_2.33-8_amd64.deb (--install): cannot copy extracted data for './usr/lib/x86_64-linux-gnu/gconv/IBM904.so' to '/usr/lib/x86_64-linux-gnu/gconv/IBM904.so.dpkg-new': failed to write (No space left on device) Or: https://ci.debian.net/data/autopkgtest/unstable/amd64/s/sbuild/24064827/log.gz + mmdebstrap --mode=unshare --variant=apt unstable /tmp/chroot.tar I: chroot architecture amd64 is equal to the host's architecture I: automatically chosen format: tar I: using /tmp/mmdebstrap.CcPte840kk as tempdir I: running apt-get update... I: downloading packages with apt... I: extracting archives... tar: ./sbin/ldconfig: Wrote only 6144 of 10240 bytes tar: ./usr/bin/catchsegv: Cannot write: No space left on device -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
