Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
17.05.2022 14:51, Michael Tokarev wrote: ... Now you have a reproducer that installed qemu-user-static on a real system and not a chroot and we observe the same effect. And now this is interesting. I finally were able to reproduce the issue but I don't think the prob is in qemu. It is the gcc, not qemu. When using -cpu host, the problem does not occur. Only when using the default qemu cpu (kvm64?) it is shown. It looks like it is the reason why it can not be reproduced on a bare metal too (in a chroot in a real system). This is.. wow. I experimented with various -cpu types of the "outer" qemu-system. It fails with -cpu Westmere but works with -cpu SandyBridge. It also fails with -cpu SandyBridge,-avx So it requires avx, and fails if avx is not available. When compiled with gcc-10, it does not happen. /mjt
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
16.05.2022 17:49, Johannes Schauer Marin Rodrigues wrote: ... $ qemu-img create -f qcow2 disk.qcow 4G $ curl --fail https://d-i.debian.org/daily-images/amd64/daily/netboot/debian-installer/amd64/initrd.gz > initrd.gz $ curl --fail https://d-i.debian.org/daily-images/amd64/daily/netboot/debian-installer/amd64/linux > linux $ /usr/bin/time -v qemu-system-x86_64 -nographic -no-reboot -enable-kvm -m 1G -initrd initrd.gz -kernel linux -append 'console=ttyS0,9600,n8 auto-install/enable=true debconf/priority=critical preseed/url=http://www.debian.org/releases/stable/example-preseed.txt netcfg/get_hostname=hostname netcfg/get_domain=domain passwd/root-password=r00tme passwd/root-password-again=r00tme passwd/user-fullname=user passwd/username=user passwd/user-password=insecure passwd/user-password-again=insecure pkgsel/run_tasksel=false popularity-contest:popularity-contest/participate=false grub-installer/bootdev=/dev/sda' -hda disk.qcow Instead of -hda disk.qcow, I'd suggest using this: -drive file=disk.qcow,if=virtio,cache=unsafe especially the cache thing, - it makes things *much* faster. But if you've a very fast SSD it should be okay too. This will create a bootable disk image using debian-installer with usernames and passwords given via preseed options. You can boot the whole thing like this: $ qemu-system-x86_64 -nographic -enable-kvm -m 1G -hda disk.qcow Log in with user:insecure, become root with password r00tme, edit /etc/apt/sources.list replacing bullseye with unstable, run apt-get update, then apt-get upgrade and then: I guess it's possible to login with root directly, when on the console. Also, the image is based on bookworm, not bullseye, but that doesn't really matter. $ apt install mmdebstrap arch-test mount uidmap binfmt-support qemu-user-static Then reboot the thing just to be sure and then run: $ mmdebstrap --arch=arm64 --variant=apt unstable /dev/null ... I: extracting archives... done I: installing essential packages... done qemu: uncaught target signal 11 (Segmentation fault) - core dumped E: run_chroot failed: E: env --unset=APT_CONFIG --unset=TMPDIR /usr/sbin/chrod W: listening on child socket failed: I: removing tempdir /tmp/mmdebstrap.nBUnpBBsox... E: mmdebstrap failed to run root@hostname:/home/user# dpkg -l | grep qemu ii qemu-guest-agent 1:7.0+dfsg-7 amd64 t ii qemu-user-static 1:7.0+dfsg-7 amd64 ) Now you have a reproducer that installed qemu-user-static on a real system and not a chroot and we observe the same effect. And now this is interesting. I finally were able to reproduce the issue but I don't think the prob is in qemu. It is the gcc, not qemu. When using -cpu host, the problem does not occur. Only when using the default qemu cpu (kvm64?) it is shown. It looks like it is the reason why it can not be reproduced on a bare metal too (in a chroot in a real system). This is.. wow. /mjt
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
17.05.2022 07:26, Johannes Schauer Marin Rodrigues wrote: ... I just learned about a much simpler way (classic: right after asking the expert). qemu-user emulation is also disabled by setting the QEMU_VERSION variable to something as this will mean that qemu-user exits without doing anything (other than printing version information): https://sources.debian.org/src/qemu/1:7.0+dfsg-7/linux-user/main.c/?hl=543#L480 This is unexpected to me :) But please do note it will exit successfully. So you don't know if it was qemu who was successful or a program you tried to run. Just something to be aware of. /mjt
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
Quoting Michael Tokarev (2022-05-16 07:46:01) > Yes this is possible, but only at the binfmt_misc level. > >echo 0 > /proc/sys/fs/binfmt_misc/qemu-aarch64 -- disable this particular > entry >echo 1 > /proc/sys/fs/binfmt_misc/qemu-aarch64 -- enable it > >echo 0 > /proc/sys/fs/binfmt_misc/status-- disable whole binfmt support >echo 1 > /proc/sys/fs/binfmt_misc/status-- enable it > > With systemd you can disable particular formats/architectures entirely > by masking /usr/lib/binfmt.d/qemu-foo.conf in /etc/binfmt.d/ (if you remove > binfmt-support). I just learned about a much simpler way (classic: right after asking the expert). qemu-user emulation is also disabled by setting the QEMU_VERSION variable to something as this will mean that qemu-user exits without doing anything (other than printing version information): https://sources.debian.org/src/qemu/1:7.0+dfsg-7/linux-user/main.c/?hl=543#L480 I'm just noting this here because I'm sure future-me will soon forget about this again. XD signature.asc Description: signature
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
Quoting Johannes Schauer Marin Rodrigues (2022-05-16 16:49:12) > Then reboot the thing just to be sure and then run: > > $ mmdebstrap --arch=arm64 --variant=apt unstable /dev/null > ... > I: extracting archives... > done > I: installing essential packages... > done > qemu: uncaught target signal 11 (Segmentation fault) - core dumped > E: run_chroot failed: E: env --unset=APT_CONFIG --unset=TMPDIR > /usr/sbin/chrod > W: listening on child socket failed: > I: removing tempdir /tmp/mmdebstrap.nBUnpBBsox... > E: mmdebstrap failed to run > root@hostname:/home/user# dpkg -l | grep qemu > ii qemu-guest-agent 1:7.0+dfsg-7 > amd64 t > ii qemu-user-static 1:7.0+dfsg-7 > amd64 ) > > Now you have a reproducer that installed qemu-user-static on a real system > and not a chroot and we observe the same effect. oh and just to make sure that this is not an mmdebstrap thing we can also use debootstrap on our freshly installed system and observe the same problem: root@hostname:/home/user# debootstrap --foreign --arch arm64 unstable debian-unstable ... I: Extracting libsmartcols1... I: Extracting libuuid1... I: Extracting mount... I: Extracting util-linux... I: Extracting util-linux-extra... I: Extracting libxxhash0... I: Extracting liblzma5... I: Extracting zlib1g... root@hostname:/home/user# chroot debian-unstable qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault root@hostname:/home/user# So now you have a reproducer where we don't even do anything special anymore. We use debian-installer to create the Debian system and we use debootstrap to trigger the problem. Thanks! cheers, josch signature.asc Description: signature
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
Hi,
Quoting Michael Tokarev (2022-05-16 07:46:01)
> > I'm not sure what you mean with "fall back". Maybe you mean that mmdebstrap
> > reads /proc/sys/fs/binfmt_misc/qemu-$arch and then looks for an F in the
> > "flags" field. If it finds that, then it will not copy qemu-user-static
> > into the chroot (as was the old way to make this work). As you also pointed
> > out, today this is automatic.
>
> No, I mean it is possible to try to execute a foreign binary directly, and
> if that fails, do it again but this time, prepend "qemu-foo[-static]' to the
> command line. Checking binfmt_misc flags is ofcourse too much. I'm asking
> because of this:
>
> 00:00:00.266 E: + mmdebstrap --mode=root --variant=apt --architectures=arm64
> unstable /tmp/debian-chroot.tar http://127.0.0.1/debian
> 00:00:00.599 E: I: arm64 cannot be executed, falling back to qemu-user
>
> I don't know what it will do with $options->{qemu} after this. I don't see
> any reason to mess with qemu-foo-static in tools like debootstrap, - you can
> run foreign binaries with it, but you can't - without binfmt_misc support -
> you can't run maintscripts or other stuff because this requires executing
> multiple binaries which you don't control.
Ah okay, yes this message wrong and misleading and a leftover from when
mmdebstrap still copied qemu-foo-static into the chroot. It doesn't do that
anymore. I'll fix the wording of this message in the next release.
But we still need to "mess" with some stuff:
- in proot mode, --qemu=qemu-$arch has to be passed to proot
- in fakechroot and chrootless mode we have to set QEMU_LD_PREFIX
In root and unshare mode we do nothing if the F flag is set in
/proc/sys/fs/binfmt_misc/qemu-$arch as explained earlier. The message should
indeed be adjusted to reflect reality. Thanks!
> > This brings me to a slightly related question: is it possible to disable
> > this transparent qemu-user-static support of running foreign architecture
> > binaries without the qemu-user-static inside the chroot? While this is
> > probably very useful in 99% of the cases, it is very annoying when I try to
> > cross build a source package and then I get a false positive when the build
> > (wrongly) tries to execute a foreign architecture binary and succeeds.
> > Thus, on my machine I'm constantly uninstalling and then re-installing
> > qemu-user whenever I want to do cross builds. Is there some config option I
> > can set to enable or disable transparent binfmt-misc emulation by qemu?
>
> Yes this is possible, but only at the binfmt_misc level.
>
>echo 0 > /proc/sys/fs/binfmt_misc/qemu-aarch64 -- disable this particular
> entry
>echo 1 > /proc/sys/fs/binfmt_misc/qemu-aarch64 -- enable it
>
>echo 0 > /proc/sys/fs/binfmt_misc/status-- disable whole binfmt support
>echo 1 > /proc/sys/fs/binfmt_misc/status-- enable it
>
> With systemd you can disable particular formats/architectures entirely
> by masking /usr/lib/binfmt.d/qemu-foo.conf in /etc/binfmt.d/ (if you remove
> binfmt-support).
Aha! I didn't know that one can write to those. I now see that the first line
of the contents of qemu-$foo changes from "enabled" to "disabled" and
vice-versa if I write 1 or 0 to it, respectively. Now this is also something
that I can check in other tools so that we don't try to cross build for
architectures on a platform that has qemu binfmt support enabled for the host
architecture. Thanks!
> > Then I also want to respond to some of the things you said in
> > #debian-devel:
> >
> >> 10:11 < mjt> but now I wonder if - after switching from binfmt-support to
> >> systemd-binfmt (and fixing
> >> this qemu-user-static bug), -- if the whole thing will work
> >> when installed in a chroot
> >> 10:13 < mjt> qemu-user should not register its binfmts when installed in
> >> chroot. binfmt-support didn't
> >> care about this, but I've no idea about systemd - maybe that
> >> one cares enough to actually
> >> fix this
> >> 10:15 < mjt> in other words: it worked in this context by accident not by1
> >> design
> >
> > What do I have to change to make it work again?
>
> This is a very good question. Two questions.
>
> First, I still don't know what happened. I tried to reproduce it locally but
> failed so far. Probably should try kernel from unstable to begin with, -
> I'm on bullseye. And I'm puzzled really, - because nothing had changed in
> this area in 7.0+dfsg-3 compared with the previous version, - as it turned
> out, even the thing which actually did change - the way it is being
> registered -
> does not work and hence does not affect this, and since you install
> binfmt-support
> explicitly, everything is exactly the same as before.
>
> And second, the whole thing of registering binfmts in a chroot is, um, wrong.
> This is because this affects whole system, inside this chroot, outside it, and
> in all other chroots too. And you install it in a chroot. It's okay to
>
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
Control: reopen -1
16.05.2022 01:07, Johannes Schauer Marin Rodrigues wrote:
Hi mjt,
Thank you very much for this excellent bug report!
thank you for your very quick reply! :D
Well.. it turned out my replies, while quick, aren't that helpful ;)
But let's see..
On Sun, 15 May 2022 10:43:43 +0300 Michael Tokarev wrote:
...
Speaking of mmdebstrap: why does it try to "fall back"? Did it try this
before, or is it intentional? The thing is that today, qemu-user-static
is transparent (or should be anyway), there's no need to run it explicitly,
it is done the binfmt-misc way so *bootstrap should not notice the arch
it is installing is foreign. It is okay if mmdebstrap does that since
historic times when this transparency didn't work, and it is not okay
if it expects such transparency now but it doesn't work.
I'm not sure what you mean with "fall back". Maybe you mean that mmdebstrap
reads /proc/sys/fs/binfmt_misc/qemu-$arch and then looks for an F in the
"flags" field. If it finds that, then it will not copy qemu-user-static into
the chroot (as was the old way to make this work). As you also pointed out,
today this is automatic.
No, I mean it is possible to try to execute a foreign binary directly, and
if that fails, do it again but this time, prepend "qemu-foo[-static]' to the
command line. Checking binfmt_misc flags is ofcourse too much. I'm asking
because of this:
00:00:00.266 E: + mmdebstrap --mode=root --variant=apt --architectures=arm64
unstable /tmp/debian-chroot.tar http://127.0.0.1/debian
00:00:00.599 E: I: arm64 cannot be executed, falling back to qemu-user
I don't know what it will do with $options->{qemu} after this. I don't see
any reason to mess with qemu-foo-static in tools like debootstrap, - you can
run foreign binaries with it, but you can't - without binfmt_misc support -
you can't run maintscripts or other stuff because this requires executing
multiple binaries which you don't control.
This brings me to a slightly related question: is it possible to disable this
transparent qemu-user-static support of running foreign architecture binaries
without the qemu-user-static inside the chroot? While this is probably very
useful in 99% of the cases, it is very annoying when I try to cross build a
source package and then I get a false positive when the build (wrongly) tries
to execute a foreign architecture binary and succeeds. Thus, on my machine I'm
constantly uninstalling and then re-installing qemu-user whenever I want to do
cross builds. Is there some config option I can set to enable or disable
transparent binfmt-misc emulation by qemu?
Yes this is possible, but only at the binfmt_misc level.
echo 0 > /proc/sys/fs/binfmt_misc/qemu-aarch64 -- disable this particular
entry
echo 1 > /proc/sys/fs/binfmt_misc/qemu-aarch64 -- enable it
echo 0 > /proc/sys/fs/binfmt_misc/status-- disable whole binfmt support
echo 1 > /proc/sys/fs/binfmt_misc/status-- enable it
With systemd you can disable particular formats/architectures entirely
by masking /usr/lib/binfmt.d/qemu-foo.conf in /etc/binfmt.d/ (if you remove
binfmt-support).
If you rename qemu-foo into qemu-foo.conf in /usr/lib/binfmt.d/ and restart
systemd-binfmt.service, it should work. I think.
I cannot confirm that this works (I tried with 1:7.0+dfsg-6).
I am now using 1:7.0+dfsg-7 which calls them *.conf and the problem persists.
Maybe you want to re-open this bug? Or should I open a new one?
Already done.
Then I also want to respond to some of the things you said in #debian-devel:
10:11 < mjt> but now I wonder if - after switching from binfmt-support to
systemd-binfmt (and fixing
this qemu-user-static bug), -- if the whole thing will work when
installed in a chroot
10:13 < mjt> qemu-user should not register its binfmts when installed in
chroot. binfmt-support didn't
care about this, but I've no idea about systemd - maybe that one
cares enough to actually
fix this
10:15 < mjt> in other words: it worked in this context by accident not by1
design
What do I have to change to make it work again?
This is a very good question. Two questions.
First, I still don't know what happened. I tried to reproduce it locally but
failed so far. Probably should try kernel from unstable to begin with, -
I'm on bullseye. And I'm puzzled really, - because nothing had changed in
this area in 7.0+dfsg-3 compared with the previous version, - as it turned
out, even the thing which actually did change - the way it is being registered -
does not work and hence does not affect this, and since you install
binfmt-support
explicitly, everything is exactly the same as before.
And second, the whole thing of registering binfmts in a chroot is, um, wrong.
This is because this affects whole system, inside this chroot, outside it, and
in all other chroots too. And you install it in a chroot. It's okay to install
qemu-user-static and register binfmts on the actual host
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
Hi mjt, > Thank you very much for this excellent bug report! thank you for your very quick reply! :D On Sun, 15 May 2022 10:43:43 +0300 Michael Tokarev wrote: > 15.05.2022 09:43, Michael Tokarev wrote: > > Meanwhile can you please try moving qemu bits away from /usr/lib/binfmt.d/ > > (this particular architecture should be enough), and see if that helps? I > > don't know what's needed for binfmt-support to re-register its fmt though, > > - a reboot should help but there is definitely a shorter way. > > Hm. It looks like there's an error in systemd binfmt registration, which > effectively does not work at all - the files in /usr/lib/binfmt.d/ should > have .conf extension. So there was no changes in qemu binfmt handling > between qemu -2 and -3, iirc. Except that now, for a new install, this > registration will not be done at all! > > Which makes the following even more important: > > > Speaking of mmdebstrap: why does it try to "fall back"? Did it try this > > before, or is it intentional? The thing is that today, qemu-user-static > > is transparent (or should be anyway), there's no need to run it explicitly, > > it is done the binfmt-misc way so *bootstrap should not notice the arch > > it is installing is foreign. It is okay if mmdebstrap does that since > > historic times when this transparency didn't work, and it is not okay > > if it expects such transparency now but it doesn't work. I'm not sure what you mean with "fall back". Maybe you mean that mmdebstrap reads /proc/sys/fs/binfmt_misc/qemu-$arch and then looks for an F in the "flags" field. If it finds that, then it will not copy qemu-user-static into the chroot (as was the old way to make this work). As you also pointed out, today this is automatic. This brings me to a slightly related question: is it possible to disable this transparent qemu-user-static support of running foreign architecture binaries without the qemu-user-static inside the chroot? While this is probably very useful in 99% of the cases, it is very annoying when I try to cross build a source package and then I get a false positive when the build (wrongly) tries to execute a foreign architecture binary and succeeds. Thus, on my machine I'm constantly uninstalling and then re-installing qemu-user whenever I want to do cross builds. Is there some config option I can set to enable or disable transparent binfmt-misc emulation by qemu? > If you rename qemu-foo into qemu-foo.conf in /usr/lib/binfmt.d/ and restart > systemd-binfmt.service, it should work. I think. I cannot confirm that this works (I tried with 1:7.0+dfsg-6). I am now using 1:7.0+dfsg-7 which calls them *.conf and the problem persists. Maybe you want to re-open this bug? Or should I open a new one? Then I also want to respond to some of the things you said in #debian-devel: > 10:11 < mjt> but now I wonder if - after switching from binfmt-support to > systemd-binfmt (and fixing > this qemu-user-static bug), -- if the whole thing will work when > installed in a chroot > 10:13 < mjt> qemu-user should not register its binfmts when installed in > chroot. binfmt-support didn't > care about this, but I've no idea about systemd - maybe that one > cares enough to actually > fix this > 10:15 < mjt> in other words: it worked in this context by accident not by1 > design What do I have to change to make it work again? > 10:39 < mjt> josch: so, in this your test, do you explicitly install > binfmt-support together with > qemu-user-static, or do you rely on qemu-user-static > dependencies? > 10:40 < mjt> josch: if the latter, can you verify if additionally installing > binfmt-support fixes this? > 10:42 < mjt> josch: I *guess* you'll have to add binfmt-support for now, > because even when this my bug > is fixed, you still can't rely on systemd properly registering > binfmts in a chroot Yes, I am explicitly installing binfmt-support. You can see this in the --depends argument I'm passing to debbisect in my initial report. Same in my mmdebstrap test cases where I observed this bug first. The binfmt-support package is explicitly installed there as well: https://sources.debian.org/src/mmdebstrap/0.8.6-2/make_mirror.sh/#L499 Thanks! cheers, josch signature.asc Description: signature
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
15.05.2022 09:43, Michael Tokarev wrote: .. Meanwhile can you please try moving qemu bits away from /usr/lib/binfmt.d/ (this particular architecture should be enough), and see if that helps? I don't know what's needed for binfmt-support to re-register its fmt though, - a reboot should help but there is definitely a shorter way. Hm. It looks like there's an error in systemd binfmt registration, which effectively does not work at all - the files in /usr/lib/binfmt.d/ should have .conf extension. So there was no changes in qemu binfmt handling between qemu -2 and -3, iirc. Except that now, for a new install, this registration will not be done at all! Which makes the following even more important: Speaking of mmdebstrap: why does it try to "fall back"? Did it try this before, or is it intentional? The thing is that today, qemu-user-static is transparent (or should be anyway), there's no need to run it explicitly, it is done the binfmt-misc way so *bootstrap should not notice the arch it is installing is foreign. It is okay if mmdebstrap does that since historic times when this transparency didn't work, and it is not okay if it expects such transparency now but it doesn't work. If you rename qemu-foo into qemu-foo.conf in /usr/lib/binfmt.d/ and restart systemd-binfmt.service, it should work. I think. Thanks, /mjt
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
15.05.2022 09:06, Johannes Schauer Marin Rodrigues wrote: ... Hello! Thank you very much for this excellent bug report! From the output above it is quite clear that the upgrade of qemu-user-static from 1:7.0+dfsg-2+b1 to 1:7.0+dfsg-3 is the culprit. Looking at d/changelog indeed reveals some changes to qemu-user-static between these versions. Here's the change which went into -3: * d/binfmt-install: also generate binfmt.d/ entries for systemd * d/control: use systemd as preferred alternative to binfmt-support hopefully Closes: #789011 (Minimal dependencies to register binfmt) Closes: #985889 (make binfmt setup configurable) Basically I switched from binfmt-misc to systemd for binfmt registration, or, actually, made the two coexist. I will take a look at this hopefully tomorrow (I don't have access to my usual test systems right now). Meanwhile can you please try moving qemu bits away from /usr/lib/binfmt.d/ (this particular architecture should be enough), and see if that helps? I don't know what's needed for binfmt-support to re-register its fmt though, - a reboot should help but there is definitely a shorter way. Speaking of mmdebstrap: why does it try to "fall back"? Did it try this before, or is it intentional? The thing is that today, qemu-user-static is transparent (or should be anyway), there's no need to run it explicitly, it is done the binfmt-misc way so *bootstrap should not notice the arch it is installing is foreign. It is okay if mmdebstrap does that since historic times when this transparency didn't work, and it is not okay if it expects such transparency now but it doesn't work. Thanks, /mjt
Bug#1011003: regression: qemu-user-static produces segfaults in foreign arch mmdebstrap
Package: qemu-user-static Version: 1:7.0+dfsg-3 Severity: important X-Debbugs-Cc: jo...@debian.org Control: affects -1 + mmdebstrap Hi, a few days ago, the mmdebstrap testsuite started to break like this: 00:00:00.206 E: + ./mmdebstrap --mode=root --variant=apt --architectures=arm64 unstable /tmp/debian-chroot.tar http://127.0.0.1/debian 00:00:00.456 E: I: arm64 cannot be executed, falling back to qemu-user 00:00:00.515 E: I: automatically chosen format: tar 00:00:00.526 E: I: using /tmp/mmdebstrap.X4yO9M_TEG as tempdir 00:00:00.603 E: I: running apt-get update... 00:00:02.923 E: I: downloading packages with apt... 00:00:04.589 E: I: extracting archives... 00:00:08.066 E: I: installing essential packages... 00:00:08.099 E: qemu: uncaught target signal 11 (Segmentation fault) - core dumped Our daily jenkins jobs confirms this observation (grep for Segmentation fault): https://jenkins.debian.net/job/fakeroot-foreign-worker/626/console I used debbisect to find out the package responsible for the issue like this: $ debbisect --cache=/tmp/cache --qemu=defaults \ --depends=mmdebstrap,arch-test,mount,uidmap,binfmt-support,qemu-user-static \ 2022-05-06T20:54:02Z 2022-05-07T03:42:36Z \ 'ssh -F "$1" qemu mmdebstrap --arch=arm64 --variant=apt --aptopt=Acquire::Check-Valid-Until\\\ \\\"false\\\" unstable /dev/null $DEBIAN_BISECT_MIRROR' snapshot timestamp difference: 0.283727 days approximately 2 steps left to test #1: trying known good 2022-05-06 20:54:02+00:00... computation time left: 0:02:54.646190 approximately 1 steps left to test #2: trying known bad 2022-05-07 03:42:36+00:00... bisection finished successfully last good timestamp: 2022-05-06 20:54:02+00:00 first bad timestamp: 2022-05-07 03:42:36+00:00 the following packages differ between the last good and first bad timestamp: libssl1.1:amd64 1.1.1n-1 -> 1.1.1o-1 qemu-user-static 1:7.0+dfsg-2+b1 -> 1:7.0+dfsg-3 test upgrading libssl1.1:amd64 1.1.1n-1 -> 1.1.1o-1... upgrading libssl1.1:amd64 does not cause the problem test upgrading qemu-user-static 1:7.0+dfsg-2+b1 -> 1:7.0+dfsg-3... upgrading qemu-user-static triggered the problem >From the output above it is quite clear that the upgrade of qemu-user-static from 1:7.0+dfsg-2+b1 to 1:7.0+dfsg-3 is the culprit. Looking at d/changelog indeed reveals some changes to qemu-user-static between these versions. Could you please investigate? This currently breaks my software mmdebstrap. Thanks! cheers, josch
