Bug#1033291: autopkgtest-virt-qemu: support booting images created using debvm-create
Control: tags -1 - patch Hi Simon, On Sun, Jun 25, 2023 at 12:00:21PM +0100, Simon McVittie wrote: > Control: forwarded -1 > https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/236 Thanks for the extensive review both here and on the MR. Please allow me to ignore all the pycodestyle and similar things until we have a solution that meets functional requirements. > The inability to reboot into an updated initrd or kernel is quite > significant for autopkgtest, and I'd be very tempted to call it a > blocker: I think many of the tests that specifically want a VM, and > are not satisfied with a lxc/podman container, are going to want that > *because* they are doing kernel/initrd things. On the merge request > (thanks for opening that!) I suggested a way to fix that limitation by > running `qemu-system-whatever -no-reboot` in a loop, so that the kernel > and initrd can be re-extracted after each shutdown (where reboot(8) > in the VM is turned into a shutdown as a result of -no-reboot). I think this together with the insight from your other mail that you want to reduce divergence of test environments is really helpful guidance. I was aware of the option of a -no-reboot loop (and that's even mentioned in the debvm-run manual page), but that comes with the complication of figuring out when to shut down (as you explained in detail already). I need deeper understanding of the context to reply in a useful way here. Let me also give a different view on this. You argued that the main reason to use the qemu backend is the ability to reboot. I do not agree with this. From my point of view, the main motivation is the better isolation (than lxc) and therefore lower risk of damage to my system. I also note that there is a particular test restriction "needs-reboot" and the new vmlinux boot mode could declare that it does not support this restriction. Do you thin that this would be an ok-ish alternative to properly implementing reboot? > You say "cloud image" here, but official Debian cloud images are > partitioned disks, and debvm uses a raw ext* filesystem filling the > entire disk (no partition table). Did you mean "cloud kernel"? Yes. > I'm already not very happy with -virt-qemu's use of screen-scraping > serial gettys, because screen-scraping an interactive shell isn't a > reliable thing to do, and the root shell on tty1/hvc1 means we end up > running the tests in a rather unrealistic environment that doesn't match > anyone's real Debian system: in practice real Debian systems are much > more likely to be server + ssh, or graphical console + display manager, > or graphical or serial console + getty, and they're certainly not what > we end up using with -virt-qemu (which is usually serial console + > passwordless root shell + su into user shell). Ok, that's news to me, but good to know. In a sense, debvm is in a similar spot here. Its main purpose is providing reproducing some kind of environment (e.g. for reproducing a bug). The realistic part is of lower importance though. So for debvm, I chose three main ways to interact with it: * qemu -nographic and a serial console is the default * Running qemu with a graphical interface (interactively) and this can use a regular tty login or a display manager * ssh > The way I'm hoping to resolve that is to introduce a -virt-ssh setup > script that shares code (the autopkgtest_qemu module) with -virt-qemu, > at which point we can use some sort of early-boot mechanism (perhaps > cloud-init, or the serial console, or debugfs, or a virtual CD-ROM > containing a script) to inject an authorized ssh key into the image and > (if necessary) install sshd, and then access it via ssh like an ordinary > server from then on. That'll make the test environment more realistic. I'm not sure how this is going to work for autopkgtest-virt-qemu. For debvm, ease of interactive use was a primary goal (as you basically can everything you can do with debvm directly with mmdebstrap + qemu). Therefore, I opted for adding the authorized_keys setup to the image construction and you can simply pass a key to debvm-create and it'll also install the ssh server then. This is a different approach than the one you describe here where the key is added retroactively. I think I can support the mechanisms you describe here in a debvm-create step at ease. So how do we proceed here? Quite evidently, our goals diverge here. While my goals are ease of unprivileged use, safe containment and broad architecture support, your goals are reduction of divergence of test beds and realistically reproducing regular environments if I capture things correctly. In any case, thanks for the extensive high quality feedback. Helmut
Bug#1033291: autopkgtest-virt-qemu: support booting images created using debvm-create
Control: forwarded -1 https://salsa.debian.org/ci-team/autopkgtest/-/merge_requests/236 On Tue, 21 Mar 2023 at 19:58:19 +0100, Helmut Grohne wrote: > * You cannot test bootloaders with this (obviously). > * You cannot test stuff that requires rebooting into an updated initrd >or Linux kernel. > * You cannot use a non-ext filesystem. The lack of bootloaders is a bit annoying, but not necessarily fatal: we'll often already have a different bootloader per architecture. The inability to reboot into an updated initrd or kernel is quite significant for autopkgtest, and I'd be very tempted to call it a blocker: I think many of the tests that specifically want a VM, and are not satisfied with a lxc/podman container, are going to want that *because* they are doing kernel/initrd things. On the merge request (thanks for opening that!) I suggested a way to fix that limitation by running `qemu-system-whatever -no-reboot` in a loop, so that the kernel and initrd can be re-extracted after each shutdown (where reboot(8) in the VM is turned into a shutdown as a result of -no-reboot). > We include python3 here, because autopkgtest-virt-qemu says so. We also > include a generic kernel image, because debvm prefers a cloud image, > which lacks support for the 9p filesystem used by autopkgtest-virt-qemu > (see #1027174). Then we must enable the additional serial gettys. You say "cloud image" here, but official Debian cloud images are partitioned disks, and debvm uses a raw ext* filesystem filling the entire disk (no partition table). Did you mean "cloud kernel"? I'm already not very happy with -virt-qemu's use of screen-scraping serial gettys, because screen-scraping an interactive shell isn't a reliable thing to do, and the root shell on tty1/hvc1 means we end up running the tests in a rather unrealistic environment that doesn't match anyone's real Debian system: in practice real Debian systems are much more likely to be server + ssh, or graphical console + display manager, or graphical or serial console + getty, and they're certainly not what we end up using with -virt-qemu (which is usually serial console + passwordless root shell + su into user shell). The way I'm hoping to resolve that is to introduce a -virt-ssh setup script that shares code (the autopkgtest_qemu module) with -virt-qemu, at which point we can use some sort of early-boot mechanism (perhaps cloud-init, or the serial console, or debugfs, or a virtual CD-ROM containing a script) to inject an authorized ssh key into the image and (if necessary) install sshd, and then access it via ssh like an ordinary server from then on. That'll make the test environment more realistic. I believe the only reason we need python3(-minimal) in the image is for the eofcat tool that is used to capture commands' stdout/stderr, which means we can discard that when using -virt-ssh. smcv
Bug#1033291: autopkgtest-virt-qemu: support booting images created using debvm-create
Hi Helmut,
thanks for implementing this!
* Helmut Grohne [2023-03-21 19:58]:
We include python3 here, because autopkgtest-virt-qemu says so. We also
include a generic kernel image, because debvm prefers a cloud image,
which lacks support for the 9p filesystem used by autopkgtest-virt-qemu
(see #1027174). Then we must enable the additional serial gettys. While
systemd has a getty generator, this generator cannot enable the consoles
that autopkgtest-virt-qemu needs. autopkgtest-virt-qemu really wants
boot messages one ttyS0 (or hvc0), so this is what we must pass as
console= boot parameter.
[..]
@@ -435,6 +488,18 @@
argv.append(
'if=pflash,format=raw,unit=1,file=%s/efivars.fd' % workdir
)
+elif boot == 'vmlinux':
+ext2_extract_boot_components(
+self.images[0].file, workdir + "/kernel", workdir + "/initrd"
+)
+label = subprocess.check_output(
+["/sbin/e2label", self.images[0].file], encoding="utf-8"
+).strip()
+argv.extend([
+"-kernel", workdir + "/kernel",
+"-initrd", workdir + "/initrd",
+"-append", "root=LABEL=%s rw console=ttyS0" % label,
+])
else:
adtlog.debug(
'Assuming nothing special needs to be done to set up '
Adding console=ttyS0 unconditionally seems to break arm64:
autopkgtest-virt-qemu --show-boot --qemu-architecture aarch64
autopkgtest_arm64.ext4
In debvm-run we add it for amd64|i386 only:
https://sources.debian.org/src/debvm/0.2.10/bin/debvm-run/#L356
Attached is a patch to do the same here.
Cheers Jochen
diff --git a/lib/autopkgtest_qemu.py b/lib/autopkgtest_qemu.py
index cdb34ec..201313b 100644
--- a/lib/autopkgtest_qemu.py
+++ b/lib/autopkgtest_qemu.py
@@ -495,10 +495,13 @@ class Qemu:
label = subprocess.check_output(
["/sbin/e2label", self.images[0].file], encoding="utf-8"
).strip()
+console = ""
+if self.qemu_architecture in ('i386', 'x86_64'):
+console = " console=ttyS0"
argv.extend([
"-kernel", workdir + "/kernel",
"-initrd", workdir + "/initrd",
-"-append", "root=LABEL=%s rw console=ttyS0" % label,
+"-append", "root=LABEL=%s rw%s" % (label, console),
])
else:
adtlog.debug(
signature.asc
Description: PGP signature
Bug#1033291: autopkgtest-virt-qemu: support booting images created using debvm-create
Hi Helmut, On 21-03-2023 19:58, Helmut Grohne wrote: you may have heared about this shiny new thing called debvm. At least Paul knows it as he helped me get its autopkgtests pass. Thank you. You're welcome. But for the rest of your bug, I'm too busy with Release Team work to think about it. If Antonio (or Simon) doesn't handle it, please ping this bug after the bookworm release. I'm trying hard to have a short freeze. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#1033291: autopkgtest-virt-qemu: support booting images created using debvm-create
Package: autopkgtest Version: 5.28 Severity: wishlist Control: affects -1 + debvm X-Debbugs-Cc: jo...@debian.org, jspri...@debian.org Hi Antonio and Paul, you may have heared about this shiny new thing called debvm. At least Paul knows it as he helped me get its autopkgtests pass. Thank you. This is a tool for creating and running ephemeral virtual machines. And autopkgtest-virt-qemu is a tool for running virtual machine images. Sounds like these two should somehow fit together. This is exactly what this bug is about. debvm has a quite special idea how a virtual machine image looks like: * A virtual machine image is a (sparse) raw image. * It contains an ext2 (or later) filesystem. * The filesystem has a non-empty filesystem label. * The filesystem contains Linux kernel and initrd, but not a bootloader. This makes some things very easy: * Since qemu becomes the actual bootloader, this removes a lot of architecture-specific issues and one can boot all architectures in a uniform way. * A raw image can easily be resized later * Generating such an image can be done entirely without root and this implies that running autopkgtests in qemu can be done entirely without root privileges. There also are downsides: * You cannot test bootloaders with this (obviously). * You cannot test stuff that requires rebooting into an updated initrd or Linux kernel. * You cannot use a non-ext filesystem. So this is not what everyone wants, but probably what some people want. It also isn't like every image created by debvm-create just works. You do need some options: debvm-create -o autopkgtest.ext4 -z 10G -- \ --include python3,linux-image-generic \ --customize-hook='systemctl --root="$1" enable serial-getty@ttyS1 serial-getty@hvc1' \ --customize-hook='sed "s/^deb /deb-src /" "$1/etc/apt/sources.list" > "$1/etc/apt/sources.list.d/src.list"' \ --customize-hook='APT_CONFIG=$MMDEBSTRAP_APT_CONFIG apt-get update' We include python3 here, because autopkgtest-virt-qemu says so. We also include a generic kernel image, because debvm prefers a cloud image, which lacks support for the 9p filesystem used by autopkgtest-virt-qemu (see #1027174). Then we must enable the additional serial gettys. While systemd has a getty generator, this generator cannot enable the consoles that autopkgtest-virt-qemu needs. autopkgtest-virt-qemu really wants boot messages one ttyS0 (or hvc0), so this is what we must pass as console= boot parameter. At the same time it wants a root shell on hvc1, which is not considered by the generator even when passing systemd.getty_auto, so we have to enable these at image construction time. By default debvm will perform a root autologin on serial gettys, so as soon as we enable them, they'll do the right thing for autopkgtest-virt-qemu. And finally, we need to add deb-src lines as debvm doesn't add them by default. Once we have this image, we need to teach autopkgtest this strange boot protocol of debvm and this is what this particular bug report is about. I'm attaching a patch to autopkgtest that adds the --boot=vmlinux value to implement this boot protocol as well as adding support for --boot=auto. And with this patch, you can say autopkgtest hello -- qemu autopkgtest.ext4 and have things just work. What do you think? Do you like the feature idea? Do you like the patch? This is a first iteration and probably not the last, but it definitely should be good to start the conversation. Helmut diff -Nru autopkgtest-5.28/debian/changelog autopkgtest-5.28+nmu1/debian/changelog --- autopkgtest-5.28/debian/changelog 2023-01-30 19:29:59.0 +0100 +++ autopkgtest-5.28+nmu1/debian/changelog 2023-03-21 13:52:17.0 +0100 @@ -1,3 +1,10 @@ +autopkgtest (5.28+nmu1) UNRELEASED; urgency=medium + + * Non-maintainer upload. + * virt-qemu: Support images created by debvm-create. (Closes: #-1) + + -- Helmut Grohne Tue, 21 Mar 2023 13:52:17 +0100 + autopkgtest (5.28) unstable; urgency=medium [ Jochen Sprickerhof ] diff -Nru autopkgtest-5.28/lib/autopkgtest_qemu.py autopkgtest-5.28+nmu1/lib/autopkgtest_qemu.py --- autopkgtest-5.28/lib/autopkgtest_qemu.py2023-01-30 19:29:59.0 +0100 +++ autopkgtest-5.28+nmu1/lib/autopkgtest_qemu.py 2023-03-21 13:52:17.0 +0100 @@ -36,6 +36,7 @@ import errno import fcntl +import itertools import json import os import re @@ -46,6 +47,7 @@ import tempfile import time from typing import ( +BinaryIO, List, Optional, Sequence, @@ -124,6 +126,52 @@ return [] +def call_debugfs( +image: str, command: str, stdout: Optional[BinaryIO] = None +) -> subprocess.CompletedProcess: +result = subprocess.run( +["/sbin/debugfs", "-R", command, image], +stdout=stdout or subprocess.PIPE, +stderr=subprocess.PIPE, +) +if result.returncode: +raise ValueError( +"command %r exited %d and stderr %r"
