Bug#1061634: mmdebstrap: possible mmdebstrap-autopkgtest-build-qemu improvements
Quoting Francesco Poli (2024-01-29 20:54:25) > On Mon, 29 Jan 2024 10:21:29 +0100 Johannes Schauer Marin Rodrigues wrote: > > [...] > > With that said, I think mmdebstrap-autopkgtest-build-qemu should keep > > using raw images. > > OK, after reading these opinions, I can agree that it makes sense to go > on with raw images. Thank you so much for taking the time to consult the > experts! There is actually one good argument to go with qcow2: That is the default format that autopkgtest-build-qemu uses. And since mmdebstrap-autopkgtest-build-qemu tries to be like autopkgtest-build-qemu it might make sense to choose the same output format... But then I also thought: if the output format is raw, the user can always easily run qemu-img afterwards if they want qcow2. The user can then even decide for themselves if and what kind of compression or other features they want. I think it is suboptimal that autopkgtest-build-qemu doesn not make this optional and just enforces the step that converts to qcow2, eating up runtime and cpu cycles even if the user doesn't require qcow2. Since this question will come up again in the future, I added this list to the code: # The image is raw and not in qcow2 format because: # - faster run-time as the "qemu-image convert" step is not needed # - image can be used independent of qemu tooling # - modifying the image just with "mount" instead of requiring qemu-nbd # - sparse images make the file just as small as with qcow2 # - trim support is more difficult on qcow2 # - snapshots and overlays work just as well with raw images # - users who prefer qcow2 get to choose to run it themselves with their own #custom options like compression Thanks! cheers, josch signature.asc Description: signature
Bug#1061634: mmdebstrap: possible mmdebstrap-autopkgtest-build-qemu improvements
On Mon, 29 Jan 2024 10:21:29 +0100 Johannes Schauer Marin Rodrigues wrote: [...] > With that said, I think mmdebstrap-autopkgtest-build-qemu should keep > using raw images. OK, after reading these opinions, I can agree that it makes sense to go on with raw images. Thank you so much for taking the time to consult the experts! Bye. -- http://www.inventati.org/frx/ There's not a second to spare! To the laboratory! . Francesco Poli . GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE pgpafCugnhaHa.pgp Description: PGP signature
Bug#1061634: mmdebstrap: possible mmdebstrap-autopkgtest-build-qemu improvements
On Mon, 29 Jan 2024 08:34:15 +0100 Johannes Schauer Marin Rodrigues wrote: [...] > I now have this locally. Is this attribution (name/email) correct? > > From 8410dc6636817c4e02cf9eba090a70051c494c48 Mon Sep 17 00:00:00 2001 > From: Francesco Poli > Date: Mon, 29 Jan 2024 08:28:53 +0100 > Subject: [PATCH] mmdebstrap-autopkgtest-build-qemu: fix octal mode computation > > --- > mmdebstrap-autopkgtest-build-qemu | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/mmdebstrap-autopkgtest-build-qemu > b/mmdebstrap-autopkgtest-build-qemu > index 5684cbe..1ed14db 100755 > --- a/mmdebstrap-autopkgtest-build-qemu > +++ b/mmdebstrap-autopkgtest-build-qemu > @@ -357,7 +357,7 @@ echo "mmdebstrap $*" > mmdebstrap "$@" || die "mmdebstrap failed" > > unshare -U -r --map-groups=auto chown 0:0 "$IMAGE" > -chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE" > +chmod "$(printf %o "$(( 0666 & ~0$(umask) ))")" "$IMAGE" > > echo "root=LABEL=autopkgtestvm rw console=ttyS0" > "$WORKDIR/cmdline" > > -- > 2.39.2 Yes, the attribution looks correct. Thanks for accepting the suggestion so promptly and for converting it into an actual patch! -- http://www.inventati.org/frx/ There's not a second to spare! To the laboratory! . Francesco Poli . GnuPG key fpr == CA01 1147 9CD2 EFDF FB82 3925 3E1C 27E1 1F69 BFFE pgphGXH0eDhg7.pgp Description: PGP signature
Bug#1061634: mmdebstrap: possible mmdebstrap-autopkgtest-build-qemu improvements
Hi, On 2024-01-29 00:07, Francesco Poli wrote: I was under the impression that the qcow2 format was the recommended one for QEMU/KVM virtual machine images. At least, qemu-img(1) describes it as "the most versatile format"... Anyway, if you think that the raw format is a better choice for autopkgtest and sbuild VM images, I take your word for it. as I said in my other mail, I'm not the expert on the topic, so I asked some experts. :) In the other mail I already paraphrased what f_g said. I now also got feedback from mjt (Michael Tokarev) our QEMU maintainer in Debian: 09:10 < mjt> for me, snapshots in qcow2 isn't of much use (I usually do it the other way, by stacking another qcow2 on top of the main image) 09:12 < jochensp> mjt: afair stacking only works if the base image is already qcow2, or? 09:12 < mjt> speaking of qcow2 vs raw, both has their own good and bad sides, mostly minor, and the preference is more about taste than actual technical differences. People tend to forget about sparseness of a raw image, becoming surprizing the drivee usage increases dramatically after a copy 09:13 < mjt> jochensp: no, stacking works on top of any image format 09:14 < mjt> ..on the other hand, trim support is more difficult on qcow2 09:14 < mjt> myself, I choose raw for regular work and qcow2 when I have to transfer the image somewhere 09:15 < mjt> it's quite easy to mount a qcow2 image on the host too, but this needs extra layer (I usually use qemu-nbd for this) 09:17 < mjt> for direct manipulation, when you have libext2fs and the tools, raw is the only way to go With that said, I think mmdebstrap-autopkgtest-build-qemu should keep using raw images. Thanks! cheers, josch
Bug#1061634: mmdebstrap: possible mmdebstrap-autopkgtest-build-qemu improvements
Hi, Quoting Francesco Poli (2024-01-29 00:07:08) > What's *your* umask? Is it Debian default (022), by chance? yes. > This seems to work with Debian default umask (022): > > $ printf '%o\n' "$(( 0666 - 00022 ))" > 644 > > but fails whenever a umask includes an octal digit equal to 7, due to > the carryover: > > $ printf '%o\n' "$(( 0666 - 7 ))" > 657 > > Shouldn't this use the bitwise AND combined with the bitwise NOT? > > $ umask > 0007 > $ printf '%o\n' "$(( 0666 & ~0$(umask) ))" > 660 > $ umask 022 > $ umask > 0022 > $ printf '%o\n' "$(( 0666 & ~0$(umask) ))" > 644 > > I would think that mmdebstrap-autopkgtest-build-qemu should run: > > chmod "$(printf %o "$(( 0666 & ~0$(umask) ))")" "$IMAGE" > > Does it make sense to you? I now have this locally. Is this attribution (name/email) correct? >From 8410dc6636817c4e02cf9eba090a70051c494c48 Mon Sep 17 00:00:00 2001 From: Francesco Poli Date: Mon, 29 Jan 2024 08:28:53 +0100 Subject: [PATCH] mmdebstrap-autopkgtest-build-qemu: fix octal mode computation --- mmdebstrap-autopkgtest-build-qemu | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mmdebstrap-autopkgtest-build-qemu b/mmdebstrap-autopkgtest-build-qemu index 5684cbe..1ed14db 100755 --- a/mmdebstrap-autopkgtest-build-qemu +++ b/mmdebstrap-autopkgtest-build-qemu @@ -357,7 +357,7 @@ echo "mmdebstrap $*" mmdebstrap "$@" || die "mmdebstrap failed" unshare -U -r --map-groups=auto chown 0:0 "$IMAGE" -chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE" +chmod "$(printf %o "$(( 0666 & ~0$(umask) ))")" "$IMAGE" echo "root=LABEL=autopkgtestvm rw console=ttyS0" > "$WORKDIR/cmdline" -- 2.39.2 > > Can you list some reasons why qcow2 should be preferred over > > raw images? > > I was under the impression that the qcow2 format was the recommended > one for QEMU/KVM virtual machine images. At least, qemu-img(1) > describes it as "the most versatile format"... > > Anyway, if you think that the raw format is a better choice for > autopkgtest and sbuild VM images, I take your word for it. I'm asking because I'm not the expert. :) I asked in #debian-devel and f_g (Fabian Grünbichler) commented that the main reasoning for choosing qcow over raw images is snapshot, link clone and backing image support. I don't think these features are useful for users of autopkgtest or sbuild with these images. Or do you see a use for them in this area? Thanks! cheers, josch signature.asc Description: signature
Bug#1061634: mmdebstrap: possible mmdebstrap-autopkgtest-build-qemu improvements
On Sun, 28 Jan 2024 00:33:06 +0100 Johannes Schauer Marin Rodrigues wrote: [...] > Quoting Francesco Poli (wintermute) (2024-01-27 19:20:08) [...] > > My comments / suggestions for improvements: > > > > * I had to manually fix the permissions, since the image file was > > originally created with the following weird ones: > > > > $ ls -l --si sid-amd64.img > > -rw-r-xrwx 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img > > those are some funny permission bits. When I run > mmdebstrap-autopkgtest-build-qemu I get: > > $ ls -lha disk.img > -rw-r--r-- 1 josch josch 26G Jan 28 00:03 disk.img > > Looks fine to me. What's *your* umask? Is it Debian default (022), by chance? > > > I think the file permissions should be set (possibly at the end of the > > mmdebstrap-autopkgtest-build-qemu run) as they would "naturally" result > > from the user's umask: > > > > $ cd /dev/shm > > $ touch foo > > $ ls -l --si foo > > -rw-rw 1 $USER $USER 0 Jan 27 19:00 foo > > They are set, taking the user's umask into account. What is your umask? Mine is James Bond's umask: 007 ;-) $ umask 0007 > > > That's why I manually changed the permissions at the end: > > > > $ chmod 660 sid-amd64.img > > $ ls -l --si sid_amd64.img > > -rw-rw 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img > > > > I suggest that mmdebstrap-autopkgtest-build-qemu applies this fix > > automatically. > > Lets first understand the problem before adding a workaround. > mmdebstrap-autopkgtest-build-qemu runs this command to set the permissions: > > chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE" This seems to work with Debian default umask (022): $ printf '%o\n' "$(( 0666 - 00022 ))" 644 but fails whenever a umask includes an octal digit equal to 7, due to the carryover: $ printf '%o\n' "$(( 0666 - 7 ))" 657 Shouldn't this use the bitwise AND combined with the bitwise NOT? $ umask 0007 $ printf '%o\n' "$(( 0666 & ~0$(umask) ))" 660 $ umask 022 $ umask 0022 $ printf '%o\n' "$(( 0666 & ~0$(umask) ))" 644 I would think that mmdebstrap-autopkgtest-build-qemu should run: chmod "$(printf %o "$(( 0666 & ~0$(umask) ))")" "$IMAGE" Does it make sense to you? > > Could you add some debugging output to the script to figure out what went > wrong > and where? Feel free to suggest any further debug that could be useful, in case the above reasoning is not enough to shed some light on what happened. > > > * I was surprised to see a 25 GiB image fit into a 7.7 GiB filesystem: > > > > $ cd /dev/shm > > $ ls -l -h sid-amd64.img > > -rw-rw 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img > > $ df -h . > > Filesystem Size Used Avail Use% Mounted on > > tmpfs 7.7G 765M 7.0G 10% /dev/shm > > > > Well, the mystery is solved by looking at the allocated size: > > > > $ ls -l -h -s sid-amd64.img > > 662M -rw-rw 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img > > > > Would it be less confusing, if mmdebstrap-autopkgtest-build-qemu created > > .qcow2 images? > > I don't understand why sparse files are confusing to you. I am clearly not very used to seeing files where the allocated size greatly differs from the total size: $ ls -n --si -s sid-amd64.img 694M -rw-rw 1 1000 1000 27G Jan 27 17:48 sid-amd64.img I am more used to seeing files where the two sizes are approximately equal: $ ls -n --si -s sid-autopkgtest-amd64.qcow2 950M -rw-r--r-- 1 1000 1000 950M Dec 30 17:55 sid-autopkgtest-amd64.qcow2 Not exactly equal, agreed: $ ls -n -s sid-autopkgtest-amd64.qcow2 927604 -rw-r--r-- 1 1000 1000 949878784 Dec 30 17:55 sid-autopkgtest-amd64.qcow2 $ calc '927604*1024' 949866496 but very similar, indeed... > Why would qcow images be less confusing? Because it seems to me that their total and allocated sizes tend to be more similar... > Can you list some reasons why qcow2 should be preferred over > raw images? I was under the impression that the qcow2 format was the recommended one for QEMU/KVM virtual machine images. At least, qemu-img(1) describes it as "the most versatile format"... Anyway, if you think that the raw format is a better choice for autopkgtest and sbuild VM images, I take your word for it. > Compression comes to mind but that is at the cost of doing that > compression in the first place and it is not obvious to me whether cpu cycles > or disk space are more scarce. For example, my platform is an ARM Cortex A53. > To give you some idea how slow it is: it takes 30 seconds before a youtube > video starts playing. At the same time, the system has a 2 TB hard disk. I > much > prefer a bigger disk image which needs less CPU when I want to create and use > it. What is your use-case? I just like to see smaller file sizes, whenever possible. And I was speaking on the basis of
Bug#1061634: mmdebstrap: possible mmdebstrap-autopkgtest-build-qemu improvements
Hi,
Quoting Francesco Poli (wintermute) (2024-01-27 19:20:08)
> I've been able to create a QEMU/KVM virtual machine image for autopkgtests:
>
> $ mkdir -p ~/var/cache/sbuild/
> $ cd /dev/shm
> $ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
> --size=25G --boot=efi sid sid-amd64.img
> $ chmod 660 sid-amd64.img
> $ mv -i sid-amd64.img ~/var/cache/sbuild/
>
> It is able to run autopkgtests:
>
> $ autopkgtest --output-dir ./${SRCPKG}_autopkgtest.out \
> --summary./${SRCPKG}_autopkgtest.summary \
> --apt-upgrade -B ./${SRCPKG}_amd64.changes -- \
> qemu --boot=efi --overlay-dir /dev/shm \
> ~/var/cache/sbuild/sid-amd64.img
>
> My comments / suggestions for improvements:
>
> * I had to manually fix the permissions, since the image file was
> originally created with the following weird ones:
>
> $ ls -l --si sid-amd64.img
> -rw-r-xrwx 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img
those are some funny permission bits. When I run
mmdebstrap-autopkgtest-build-qemu I get:
$ ls -lha disk.img
-rw-r--r-- 1 josch josch 26G Jan 28 00:03 disk.img
Looks fine to me.
> I think the file permissions should be set (possibly at the end of the
> mmdebstrap-autopkgtest-build-qemu run) as they would "naturally" result
> from the user's umask:
>
> $ cd /dev/shm
> $ touch foo
> $ ls -l --si foo
> -rw-rw 1 $USER $USER 0 Jan 27 19:00 foo
They are set, taking the user's umask into account. What is your umask?
> That's why I manually changed the permissions at the end:
>
> $ chmod 660 sid-amd64.img
> $ ls -l --si sid_amd64.img
> -rw-rw 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img
>
> I suggest that mmdebstrap-autopkgtest-build-qemu applies this fix
> automatically.
Lets first understand the problem before adding a workaround.
mmdebstrap-autopkgtest-build-qemu runs this command to set the permissions:
chmod "$(printf %o "$(( 0666 - 0$(umask) ))")" "$IMAGE"
Could you add some debugging output to the script to figure out what went wrong
and where?
> * I was surprised to see a 25 GiB image fit into a 7.7 GiB filesystem:
>
> $ cd /dev/shm
> $ ls -l -h sid-amd64.img
> -rw-rw 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img
> $ df -h .
> Filesystem Size Used Avail Use% Mounted on
> tmpfs 7.7G 765M 7.0G 10% /dev/shm
>
> Well, the mystery is solved by looking at the allocated size:
>
> $ ls -l -h -s sid-amd64.img
> 662M -rw-rw 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img
>
> Would it be less confusing, if mmdebstrap-autopkgtest-build-qemu created
> .qcow2 images?
I don't understand why sparse files are confusing to you. Why would qcow images
be less confusing? Can you list some reasons why qcow2 should be preferred over
raw images? Compression comes to mind but that is at the cost of doing that
compression in the first place and it is not obvious to me whether cpu cycles
or disk space are more scarce. For example, my platform is an ARM Cortex A53.
To give you some idea how slow it is: it takes 30 seconds before a youtube
video starts playing. At the same time, the system has a 2 TB hard disk. I much
prefer a bigger disk image which needs less CPU when I want to create and use
it. What is your use-case?
Thanks!
cheers, josch
signature.asc
Description: signature
Bug#1061634: mmdebstrap: possible mmdebstrap-autopkgtest-build-qemu improvements
Package: mmdebstrap
Version: 1.4.1-1
Severity: normal
Hello Johannes!
I've been able to create a QEMU/KVM virtual machine image for autopkgtests:
$ mkdir -p ~/var/cache/sbuild/
$ cd /dev/shm
$ TMPDIR=/dev/shm mmdebstrap-autopkgtest-build-qemu \
--size=25G --boot=efi sid sid-amd64.img
$ chmod 660 sid-amd64.img
$ mv -i sid-amd64.img ~/var/cache/sbuild/
It is able to run autopkgtests:
$ autopkgtest --output-dir ./${SRCPKG}_autopkgtest.out \
--summary./${SRCPKG}_autopkgtest.summary \
--apt-upgrade -B ./${SRCPKG}_amd64.changes -- \
qemu --boot=efi --overlay-dir /dev/shm \
~/var/cache/sbuild/sid-amd64.img
My comments / suggestions for improvements:
* I had to manually fix the permissions, since the image file was
originally created with the following weird ones:
$ ls -l --si sid-amd64.img
-rw-r-xrwx 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img
I think the file permissions should be set (possibly at the end of the
mmdebstrap-autopkgtest-build-qemu run) as they would "naturally"
result from the user's umask:
$ cd /dev/shm
$ touch foo
$ ls -l --si foo
-rw-rw 1 $USER $USER 0 Jan 27 19:00 foo
That's why I manually changed the permissions at the end:
$ chmod 660 sid-amd64.img
$ ls -l --si sid_amd64.img
-rw-rw 1 $USER $USER 27G Jan 27 17:48 sid-amd64.img
I suggest that mmdebstrap-autopkgtest-build-qemu applies this fix
automatically.
* I was surprised to see a 25 GiB image fit into a 7.7 GiB filesystem:
$ cd /dev/shm
$ ls -l -h sid-amd64.img
-rw-rw 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img
$ df -h .
Filesystem Size Used Avail Use% Mounted on
tmpfs 7.7G 765M 7.0G 10% /dev/shm
Well, the mystery is solved by looking at the allocated size:
$ ls -l -h -s sid-amd64.img
662M -rw-rw 1 $USER $USER 26G Jan 27 17:48 sid-amd64.img
Would it be less confusing, if mmdebstrap-autopkgtest-build-qemu created
.qcow2 images?
-- System Information:
Debian Release: trixie/sid
APT prefers testing
APT policy: (800, 'testing'), (500, 'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 6.5.0-5-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages mmdebstrap depends on:
ii apt 2.7.10
ii perl 5.38.2-3
ii python3 3.11.6-1
Versions of packages mmdebstrap recommends:
ii arch-test0.21-1
ii fakechroot 2.20.1+ds-15
ii fakeroot 1.33-1
ii gpg 2.2.40-1.1+b1
ii libdistro-info-perl 1.7
ii libdpkg-perl 1.22.2
ii mount2.39.3-6
ii uidmap 1:4.13+dfsg1-3+b1
Versions of packages mmdebstrap suggests:
pn apt-transport-tor
ii apt-utils 2.7.10
ii ca-certificates20230311
ii debootstrap1.0.134
ii distro-info-data 0.60
ii dpkg-dev 1.22.2
pn genext2fs
ii perl-doc 5.38.2-3
pn qemu-user
pn qemu-user-static
pn squashfs-tools-ng
ii systemd255.2-4
-- no debconf information
