Bug#908274: debvm for autopkgtests with multiple host?
Hi,
Quick followup given new insights.
On Sun, Sep 24, 2023 at 05:51:47PM +0200, Helmut Grohne wrote:
> Hi Johannes,
>
> On Sun, Sep 24, 2023 at 10:27:37AM +0200, Johannes Schauer Marin Rodrigues
> wrote:
> > There is really not much magic. The core of it is to pass this to your
> > mmdebstrap or debvm-create invocation:
> >
> > --setup-hook='for f in /etc/apt/sources.list /etc/apt/sources.list.d/*
> > /etc/apt/preferences.d/*;
> > do [ -e "$f" ] && { echo; sed "s| file://| copy://|"
> > "$f"; } | tee "$1/$f" >&2; done'
> > --hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount
>
> This sounds simple, but reality is a little more elaborate.
>
> For one thing, there also is
> /usr/share/mmdebstrap/hooks/copy-host-apt-sources-and-preferences. This
> hook directory is similar but subtly different from the above setup
> hook:
> * It does not perform the translation of file:// uris into copy://uris.
The gist is that accessing file:// URIs from within the mmdebstrap
chroot won't work out of the box. One can either turn the into copy://
URIs or use the file-mirror-automount hook to issue bind mounts for
them. According to Johannes, the latter is to be considered more
reliable.
> What seems to work is this:
>
> debvm-create
> --skip=usrmerge
> ...
> --
> --hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount
>
> --hook-dir=/usr/share/mmdebstrap/hooks/copy-host-apt-sources-and-preferences
> --hook-dir=/usr/share/mmdebstrap/hooks/maybe-merged-usr
> ""
>
> That final empty string supplies the apt sources. Does this sound about
> right? If yes, I'd like to add this as a non-flaky autopkgtest to debvm.
This is subtly wrong. The file-mirror-automount hook must come after
copy-host-apt-sources-and-preferences or it may miss URIs to mount and
it must come before maybe-merged-usr or it won't have done its job in
time. So rather use this pattern:
debvm-create
--skip=usrmerge
...
--
--hook-dir=/usr/share/mmdebstrap/hooks/copy-host-apt-sources-and-preferences
--hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount
--hook-dir=/usr/share/mmdebstrap/hooks/maybe-merged-usr
""
The sbuild autopkgtest uses something roughly like this and debvm's
autopkgtest also now use this (and actually pass that way).
Beware of one horny detail. When mmdebstrap fails resolving dependencies
(and that can happen during debci), it kills its process group as a
mechanism to get rid of its children. This works fine if your
autopkgtest does not have needs-root. If it does, this failure mode can
currently damage debci infrastructure (yes, really). So if you use this
together with needs-root, please also wrap it in "setsid -w" to keep
debci in a healthy state.
And with these instructions, I think I've also resolved #1036919!
Helmut
Bug#908274: debvm for autopkgtests with multiple host?
Hi Johannes,
On Sun, Sep 24, 2023 at 10:27:37AM +0200, Johannes Schauer Marin Rodrigues
wrote:
> There is really not much magic. The core of it is to pass this to your
> mmdebstrap or debvm-create invocation:
>
> --setup-hook='for f in /etc/apt/sources.list /etc/apt/sources.list.d/*
> /etc/apt/preferences.d/*;
> do [ -e "$f" ] && { echo; sed "s| file://| copy://|" "$f";
> } | tee "$1/$f" >&2; done'
> --hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount
This sounds simple, but reality is a little more elaborate.
For one thing, there also is
/usr/share/mmdebstrap/hooks/copy-host-apt-sources-and-preferences. This
hook directory is similar but subtly different from the above setup
hook:
* It does not perform the translation of file:// uris into copy://uris.
* It is more accurate in terms of following non-standard locations for
the various configuration items.
* Neither of these clear the sources.list created by mmdebstrap by
default.
* The latter one verifies that you have the same package versions
inside and outside.
Did I accurately represent the differences? Which one would you prefer
in which situation?
Adding any of this to debvm-create will not just work, because
debvm-create also adds the maybe-merged-usr hook and any pass-through
arguments you add come later. Therefore the maybe-merged-usr hook would
come before this hook and it fails if you pass an empty sources.list,
which would be most useful. As a workaround, you may add
--skip=usrmerge -- --hook-dir=/usr/share/mmdebstrap/hooks/maybe-merged-usr
to the debvm-create invocation to reorder the hooks to actually work.
What seems to work is this:
debvm-create
--skip=usrmerge
...
--
--hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount
--hook-dir=/usr/share/mmdebstrap/hooks/copy-host-apt-sources-and-preferences
--hook-dir=/usr/share/mmdebstrap/hooks/maybe-merged-usr
""
That final empty string supplies the apt sources. Does this sound about
right? If yes, I'd like to add this as a non-flaky autopkgtest to debvm.
Helmut
Bug#908274: debvm for autopkgtests with multiple host?
Hi, On 24-09-2023 10:27, Johannes Schauer Marin Rodrigues wrote: Is the apt configuration on those systems set to something that is not the default and should be considered as well? How the unstable to testing migration runs work is that they have a testing testbed (with apt pinning making testing the APT::Default-Release without using that configuration option for $REASONS) with unstable added as an available suite. autopkgtest will create apt pinning for only those packages that were requested on the interface (by britney2, our migration software) to be added to testing. That way, we try to see what happens in testing if we would migrate the candidate source package to testing without all the rest of unstable. Be aware, there's also an ugly fall back in autopkgtest where it will remove all the pinning if it can't install all test dependencies with the pinning set-upped as described above, effectively allowing all packages from unstable to be installed. However, for the current use-case that probably happens *before* debvm/mmdebstrap runs, so that detail should not matter. Paul OpenPGP_signature.asc Description: OpenPGP digital signature
Bug#908274: debvm for autopkgtests with multiple host?
Hi,
Quoting Helmut Grohne (2023-09-23 20:18:21)
> Quoting Ian Jackson (2023-09-23 12:19:27)
> > To summarise that discussion: at that time the best available solution that
> > worked in ci.d.n seemed to be to write an ad-hoc script to run the tests in
> > qemu; three packes had done that, each separately, with complex scripts
> > with many moving parts.
the three packages are probably sbuild, dropbear and cryptsetup?
> > I saw debvm, and wondered if it was suitable for this purpose. But, then I
> > looked at its debian/test/control and I see that the tests are marked as
> > flaky.[2] So maybe it isn't reliable enough :-/.
>
> The reliability of tests is ok. The reason for marking them flaky is that
> they currently test the "wrong" packages. ci.d.n sets up chroots in a
> delicate way to combine particular packages to see which combinations cause
> breakage. Then debvm just creates an unstable system and tests that. In
> effect, it currently tests unstable (inside those virtual machines) rather
> than what it is supposed to be testing.
>
> Johannes solved this problem on the mmdebstrap side and mmdebstrap's
> tests no longer are flaky in this way. Therefore this should be solvable
> on the debvm side. I just haven't gotten do figuring out the right runes
> thus far. Roughly speaking, the hosts' apt configuration, pinning and
> sources.lists should be used inside the created virtual machine.
right now the mmdebstrap autopkgtest only mimics sources and pinning of the
outside system. I had not considered that apt configuration on salsaci or debci
was set to something that influenced the tests. Is the apt configuration on
those systems set to something that is not the default and should be considered
as well?
There is really not much magic. The core of it is to pass this to your
mmdebstrap or debvm-create invocation:
--setup-hook='for f in /etc/apt/sources.list /etc/apt/sources.list.d/*
/etc/apt/preferences.d/*;
do [ -e "$f" ] && { echo; sed "s| file://| copy://|" "$f"; }
| tee "$1/$f" >&2; done'
--hook-dir=/usr/share/mmdebstrap/hooks/file-mirror-automount
The first will hook will make sure that the chroot receives the sources and
pinning values of the outside system. The second will do some bind-mount magic
which allows the chrooted processes to access even file:// repositories from
outside the chroot. Full script here:
https://sources.debian.org/src/sbuild/0.85.3/debian/tests/unshare-qemuwrapper/
> There is another practical problem. None of the autopkgtest nodes support
> kvm. Emulation will always use tcg. For one thing, tcg is slow. It can be so
> slow on some architectures that RCU becomes unhappy as its grace periods
> become too long. For another, tcg is buggy. It has emulation bugs even on
> release architectures that make some expected functionality fail. For
> instance, gdb reliably segfaults when run in s390x tcg emulation.
Architectures that are not amd64 or arm64 are a common source of problems. The
current sbuild autopkgtest hits another issue when running qemu on s390x:
514s ^M[ 28.399829] illegal operation: 0001 ilc:1 [#1] SMP
514s ^M[ 28.400490] Modules linked in: chacha_s390(+) libchacha virtio_pci
virtio_pci_legacy_dev virtio_pci_modern_dev virtio_blk
514s ^M[ 28.402977] CPU: 4 PID: 163 Comm: cryptomgr_test Not tainted
6.5.0-1-s390x #1 Debian 6.5.3-1
514s ^M[ 28.403150] Hardware name: QEMU 8561 QEMU (KVM/Linux)
514s ^M[ 28.403285] Krnl PSW : 0704c0018000 0042 (0x42)
514s ^M[ 28.403964]R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:3 CC:0
PM:0 RI:0 EA:3
[...]
514s ^M[ 28.408739] Last Breaking-Event-Address:
514s ^M[ 28.408758] [<03ff800fb084>] chacha_crypt_generic+0x54/0xfd0
[libchacha]
514s ^M[ 28.409610] ---[ end trace ]---
https://ci.debian.net/data/autopkgtest/testing/s390x/s/sbuild/38123402/log.gz
In addition to debvm, these tests should maybe directly depend on qemu so that
they get run on new qemu uploads that trigger these kinds of regressions.
Assuming this is really a qemu problem and not something a new kernel version
introduced...
Thanks!
cheers, josch
signature.asc
Description: signature
Bug#908274: debvm for autopkgtests with multiple host?
Hi Ian, On Sat, Sep 23, 2023 at 11:19:27AM +0100, Ian Jackson wrote: > To summarise that discussion: at that time the best available solution > that worked in ci.d.n seemed to be to write an ad-hoc script to run > the tests in qemu; three packes had done that, each separately, with > complex scripts with many moving parts. In principle, debvm is supposed to target that particular use case. There are two limitations that currently make this infeasible. > I saw debvm, and wondered if it was suitable for this purpose. > But, then I looked at its debian/test/control and I see that the tests > are marked as flaky.[2] So maybe it isn't reliable enough :-/. The reliability of tests is ok. The reason for marking them flaky is that they currently test the "wrong" packages. ci.d.n sets up chroots in a delicate way to combine particular packages to see which combinations cause breakage. Then debvm just creates an unstable system and tests that. In effect, it currently tests unstable (inside those virtual machines) rather than what it is supposed to be testing. Johannes solved this problem on the mmdebstrap side and mmdebstrap's tests no longer are flaky in this way. Therefore this should be solvable on the debvm side. I just haven't gotten do figuring out the right runes thus far. Roughly speaking, the hosts' apt configuration, pinning and sources.lists should be used inside the created virtual machine. > I have other questions too, particularly to do with the way I would > need autopkgtest to be able to influence package selection in the > nested testbeds. Exactly. That's currently the missing piece to remove the flakiness annotation. There is another practical problem. None of the autopkgtest nodes support kvm. Emulation will always use tcg. For one thing, tcg is slow. It can be so slow on some architectures that RCU becomes unhappy as its grace periods become too long. For another, tcg is buggy. It has emulation bugs even on release architectures that make some expected functionality fail. For instance, gdb reliably segfaults when run in s390x tcg emulation. That kvm aspect kinda seems like an unresolvable blocker. While most autopkgtest machines are physical machines, they use kvm for running the actual autopkgtest nodes and then lxc for individual test isolation. We'd have to use nested kvm here and somehow get it through lxc. > Everyone else: has there been any other progress on the multi-node > autopkgtest problem ? Disregarding these two aspects, debvm should get you quite far. You probably need to take network management into your own hands. I expect that vde2 would be a good way to implement this in an unprivileged way. Your debvm invocations may become a little longer that way, but that should be fine given that you store all of that in scripts. Helmut
Bug#908274: debvm for autopkgtests with multiple host?
Hi Helmut and others. Some time ago we had a conversation on debian-devel about how to make autopkgtests that spawn multiple nodes and communicate between them, eg for testing network protocols[1]. To summarise that discussion: at that time the best available solution that worked in ci.d.n seemed to be to write an ad-hoc script to run the tests in qemu; three packes had done that, each separately, with complex scripts with many moving parts. I saw debvm, and wondered if it was suitable for this purpose. But, then I looked at its debian/test/control and I see that the tests are marked as flaky.[2] So maybe it isn't reliable enough :-/. I have other questions too, particularly to do with the way I would need autopkgtest to be able to influence package selection in the nested testbeds. Everyone else: has there been any other progress on the multi-node autopkgtest problem ? Thanks, Ian. [1] https://lists.debian.org/debian-devel/2023/01/msg00059.html https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=908274 [2] https://salsa.debian.org/helmutg/debvm/-/blob/main/debian/tests/control -- Ian JacksonThese opinions are my own. Pronouns: they/he. If I emailed you from @fyvzl.net or @evade.org.uk, that is a private address which bypasses my fierce spamfilter.
