On Sun, 05 Jun 2022 at 17:04:02 +0200, Johannes Schauer Marin Rodrigues wrote:
> I am offering to either rewrite autopkgtest-build-qemu with a --method option
> defaulting either to "guestfish" if a native image is requested, "initramfs"
> if
> a foreign architecture is requested and "kernel" if the image is neither i386,
> amd64, armhf, arm64 or ppc64el. If the user requires a system as close to a
> real system as possible, they can manually use --method=debian-installer.
This is extremely comprehensive, but I'm concerned that if we have this
many code paths for building a qemu VM, at least one of them will be
broken at any given time. We already have more virtualization backends
than maintainer bandwidth or routine testing (see my recent efforts to
get all our tests passing, and I haven't even got as far as qemu yet).
I appreciate your offer to add more code, but if we add all these code
paths and document them as something that should work, I'm concerned that
we'll end up in a situation where we can't remove any of them because
someone might be relying on them, but we don't actually know that they
*work* either, and nobody among the autopkgtest maintainers has the time
or knowledge to fix them when they don't. (This could be mitigated by
outsourcing as much as possible to some other system builder, like we
already do to some extent with vmdb2.)
As I believe I've already said in an earlier message to this bug report,
one way to get autopkgtest out of the business of being yet another
system-building tool is to start from a "cloud" VM image produced by
someone else (ideally, "have a qemu-runnable cloud image" would be one
of the jobs for the porting team for each architecture). One approach to
this is to take a generic cloud image, boot it with qemu and modify it
to provide the serial ports that autopkgtest-virt-qemu currently needs,
as autopkgtest-buildvm-ubuntu-cloud does.
However, the serial-port-based command execution is really quite
fragile: there are arbitrary timeouts that cause testbed startup to
fail randomly while under load (which is a large part of why I haven't
been able to get it systematically passing tests), and it's vulnerable
to architectures having different conventions for how the virtualized
"hardware" is represented in the guest user-space. I've tried to reduce
that problem by using the virtio hypervisor serial consoles /dev/hvc*, but
as you can see from the ppc64le special case in lib/autopkgtest_qemu.py,
even that is not completely predictable, because on ppc64le the virtio
serial consoles start from hvc1.
I think a less fragile approach than the serial ports would be to use
ssh, which in practice needs to work anyway, because it's how porters
typically interact with their test hardware. My goal in refactoring
part of autopkgtest-virt-qemu into lib/autopkgtest_qemu.py was to get an
autopkgtest-virt-ssh setup script that runs and manages a qemu VM, using
cloud-init to provision a trusted public key. Unfortunately, getting that
tested and working has been delayed by trying to clear enough technical
debt to make the tests pass on the backends we already have...
I think the proliferation of system-building tools that you have noted in
the past is partly because until recently they have always needed root,
partly because none of the existing ones is particularly comprehensive,
and partly because of the number of fiddly details involved in getting
the correct bootloader for each architecture and poking it into place.
If we had something analogous to grub-cloud (install on and boot from the
first disk, no attempt to support dual-booting or special configurations,
hard-code a serial console, etc.) as part of the minimal requirements
for supporting an architecture, ideally with a metapackage available on
all architectures that pulls in the correct grub-cloud-equivalent, then I
think that would help a lot - but unfortunately grub-cloud is amd64-only.
It also occurs to me that thanks to debootstrap and its clones (including
mmdebstrap), making a non-bootable Debian rootfs tree (in some transport
format like a tarball) is a solved problem; but one of the reasons people
have quality-of-implementation preferences between system-building tools
is that all of them try to implement the entire procedure, starting from
nothing and ending up with a bootable image. Might it help to re-scope
the problem to: start from a non-bootable rootfs tarball as produced by
debootstrap/mmdebstrap, and do whatever architecture-specific things
are necessary to turn it into a bootable disk image with a kernel,
a bootloader, whatever architecture-specific boot partitions are required,
and an expandable root partition populated from the tarball?
> Option 1 is to run guestfish inside another qemu instance. This is
> extremely slow though.
I agree this doesn't seem like a great trade-off.
> Option 2 is to do the partition setup and bootloader installation as an
> initramfs hook script
I think this has potential. I see it only works for certain architectures,
but we only know how to run certain architectures in qemu and give them the
necessary serial ports *anyway*, so...
> debian-installer (option 3 from above)
>
> - useful because this creates a system that is as close to a system as users
> would install it as possible -- but it takes longer to create than the
> others
Does this work on all architectures, or on all release architectures?
I think something more analogous to the official cloud images is perhaps
a better basis for CI systems than preseeding the interactive d-i
interface so much that it becomes non-interactive, but if this has wider
architecture coverage than (for example) the initramfs approach then it
seems maybe useful.
> Option 4 is to use u-boot-qemu
If we had u-boot-qemu on more architectures, I'd like this one a lot more.
> Option 5 is the same as above but will work for architectures without
> u-boot-qemu support. Without a bootloader, the kernel and initrd can be
> passed to qemu manually via -kernel and -initrd arguments.
The major disadvantage of booting with an external kernel and initrd is
that it can't test new kernels, and can't fully test new versions of a
component that contributes to the initramfs, which might break the
expectations of a test that has the isolation-machine restriction.
> vmdb2 needs root and this will not change
> mkosi needs root
As soon as you have one qemu-bootable image (perhaps from adapting a
cloud image made by someone else), you can boot into that to be root
and produce others. On amd64, you don't even need an image, because
fakemachine can reuse your running system's kernel and /usr (but that's
currently amd64-only).
> debos will only run on amd64 and not on the other arches
If I knew how to design APIs in Go, I'd have fixed that already:
it's not actually all that hard conceptually, but it does need a table
mapping Go architectures to various architecture-specific properties,
and I don't know how a Go programmer would represent that in a somewhat
future-proof API.
smcv
