[Bug 1921664] Re: Coroutines are racy for risc64 emu on arm64 - crash on Assertion
Upstream QEMU bugs are now tracked on https://gitlab.com/qemu- project/qemu/-/issues - so if you can reproduce it with the latest version from upstream QEMU, please report it there. ** No longer affects: qemu -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921664 Title: Coroutines are racy for risc64 emu on arm64 - crash on Assertion Status in qemu package in Ubuntu: Incomplete Bug description: Note: this could as well be "riscv64 on arm64" for being slow@slow and affect other architectures as well. The following case triggers on a Raspberry Pi4 running with arm64 on Ubuntu 21.04 [1][2]. It might trigger on other environments as well, but that is what we have seen it so far. $ wget https://github.com/carlosedp/riscv-bringup/releases/download/v1.0/UbuntuFocal-riscv64-QemuVM.tar.gz $ tar xzf UbuntuFocal-riscv64-QemuVM.tar.gz $ ./run_riscvVM.sh (wait ~2 minutes) [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting udev Kernel Device Manager... qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. This is often, but not 100% reproducible and the cases differ slightly we see either of: - qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. - qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. Rebuilding working cases has shown to make them fail, as well as rebulding (or even reinstalling) bad cases has made them work. Also the same builds on different arm64 CPUs behave differently. TL;DR: The full list of conditions influencing good/bad case here are not yet known. [1]: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview [2]: http://cdimage.ubuntu.com/daily-preinstalled/pending/hirsute-preinstalled-desktop-arm64+raspi.img.xz --- --- original report --- --- I regularly run a RISC-V (RV64GC) QEMU VM, but an update a few days ago broke it. Now when I launch it, it hits an assertion: OpenSBI v0.6 _ _ / __ \ / | _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |) | |_) || |_ \/| .__/ \___|_| |_|_/|/_| | | |_| ... Found /boot/extlinux/extlinux.conf Retrieving file: /boot/extlinux/extlinux.conf 618 bytes read in 2 ms (301.8 KiB/s) RISC-V Qemu Boot Options 1: Linux kernel-5.5.0-dirty 2: Linux kernel-5.5.0-dirty (recovery mode) Enter choice: 1:Linux kernel-5.5.0-dirty Retrieving file: /boot/initrd.img-5.5.0-dirty qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. ./run.sh: line 31: 1604 Aborted (core dumped) qemu-system-riscv64 -machine virt -nographic -smp 8 -m 8G -bios fw_payload.bin -device virtio-blk-devi ce,drive=hd0 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -drive file=riscv64-UbuntuFocal-qemu.qcow2,format=qcow2,id=hd0 -devi ce virtio-net-device,netdev=usernet -netdev user,id=usernet,$ports Interestingly this doesn't happen on the AMD64 version of Ubuntu 21.04 (fully updated). Think you have everything already, but just in case: $ lsb_release -rd Description:Ubuntu Hirsute Hippo (development branch) Release:21.04 $ uname -a Linux minimacvm 5.11.0-11-generic #12-Ubuntu SMP Mon Mar 1 19:27:36 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux (note this is a VM running on macOS/M1) $ apt-cache policy qemu qemu: Installed: 1:5.2+dfsg-9ubuntu1 Candidate: 1:5.2+dfsg-9ubuntu1 Version table: *** 1:5.2+dfsg-9ubuntu1 500 500 http://ports.ubuntu.com/ubuntu-ports hirsute/universe arm64 Packages 100 /var/lib/dpkg/status ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: qemu 1:5.2+dfsg-9ubuntu1 ProcVersionSignature: Ubuntu 5.11.0-11.12-generic 5.11.0 Uname: Linux 5.11.0-11-generic aarch64 ApportVersion: 2.20.11-0ubuntu61 Architecture: arm64 CasperMD5CheckResult: unknown CurrentDmesg: Error: command ['pkexec', 'dmesg'] failed with exit code 127: polkit-agent-helper-1: error response to PolicyKit daemon: GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: No session for cookie Error executing command as another user: Not authorized This incident has been reported. Date: Mon Mar 29 02:33:25 2021 Dependencies: KvmCmdLine: COMMAND STAT EUID RUID PIDPPID %CPU COMMAND Lspci-vt: -[:00]-+-00.0 Apple Inc. Device f020 +-01.0
[Bug 1921664] Re: Coroutines are racy for risc64 emu on arm64 - crash on Assertion
I am consistently hitting this when trying to install the Ubuntu arm64 ISO image in a VM. A minimal command line that reproduces the problem is (host system is jammy arm64): qemu-system-aarch64 -enable-kvm -m 2048 -M virt -cpu host -nographic -drive file=flash0.img,if=pflash,format=raw -drive file=flash1.img,if=pflash,format=raw -drive file=image2.qcow2,if=virtio -cdrom jammy-live-server-arm64.iso The installation never gets to an end, always crashing. ** Changed in: qemu Status: Expired => Incomplete ** Changed in: qemu (Ubuntu) Status: Expired => Incomplete -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921664 Title: Coroutines are racy for risc64 emu on arm64 - crash on Assertion Status in QEMU: Incomplete Status in qemu package in Ubuntu: Incomplete Bug description: Note: this could as well be "riscv64 on arm64" for being slow@slow and affect other architectures as well. The following case triggers on a Raspberry Pi4 running with arm64 on Ubuntu 21.04 [1][2]. It might trigger on other environments as well, but that is what we have seen it so far. $ wget https://github.com/carlosedp/riscv-bringup/releases/download/v1.0/UbuntuFocal-riscv64-QemuVM.tar.gz $ tar xzf UbuntuFocal-riscv64-QemuVM.tar.gz $ ./run_riscvVM.sh (wait ~2 minutes) [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting udev Kernel Device Manager... qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. This is often, but not 100% reproducible and the cases differ slightly we see either of: - qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. - qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. Rebuilding working cases has shown to make them fail, as well as rebulding (or even reinstalling) bad cases has made them work. Also the same builds on different arm64 CPUs behave differently. TL;DR: The full list of conditions influencing good/bad case here are not yet known. [1]: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview [2]: http://cdimage.ubuntu.com/daily-preinstalled/pending/hirsute-preinstalled-desktop-arm64+raspi.img.xz --- --- original report --- --- I regularly run a RISC-V (RV64GC) QEMU VM, but an update a few days ago broke it. Now when I launch it, it hits an assertion: OpenSBI v0.6 _ _ / __ \ / | _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |) | |_) || |_ \/| .__/ \___|_| |_|_/|/_| | | |_| ... Found /boot/extlinux/extlinux.conf Retrieving file: /boot/extlinux/extlinux.conf 618 bytes read in 2 ms (301.8 KiB/s) RISC-V Qemu Boot Options 1: Linux kernel-5.5.0-dirty 2: Linux kernel-5.5.0-dirty (recovery mode) Enter choice: 1:Linux kernel-5.5.0-dirty Retrieving file: /boot/initrd.img-5.5.0-dirty qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. ./run.sh: line 31: 1604 Aborted (core dumped) qemu-system-riscv64 -machine virt -nographic -smp 8 -m 8G -bios fw_payload.bin -device virtio-blk-devi ce,drive=hd0 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -drive file=riscv64-UbuntuFocal-qemu.qcow2,format=qcow2,id=hd0 -devi ce virtio-net-device,netdev=usernet -netdev user,id=usernet,$ports Interestingly this doesn't happen on the AMD64 version of Ubuntu 21.04 (fully updated). Think you have everything already, but just in case: $ lsb_release -rd Description:Ubuntu Hirsute Hippo (development branch) Release:21.04 $ uname -a Linux minimacvm 5.11.0-11-generic #12-Ubuntu SMP Mon Mar 1 19:27:36 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux (note this is a VM running on macOS/M1) $ apt-cache policy qemu qemu: Installed: 1:5.2+dfsg-9ubuntu1 Candidate: 1:5.2+dfsg-9ubuntu1 Version table: *** 1:5.2+dfsg-9ubuntu1 500 500 http://ports.ubuntu.com/ubuntu-ports hirsute/universe arm64 Packages 100 /var/lib/dpkg/status ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: qemu 1:5.2+dfsg-9ubuntu1 ProcVersionSignature: Ubuntu 5.11.0-11.12-generic 5.11.0 Uname: Linux 5.11.0-11-generic aarch64 ApportVersion: 2.20.11-0ubuntu61 Architecture: arm64 CasperMD5CheckResult: unknown CurrentDmesg: Error: command ['pkexec', 'dmesg'] failed with exit code 127: polkit-agent-helper-1: error
[Bug 1921664] Re: Coroutines are racy for risc64 emu on arm64 - crash on Assertion
No, as I described in great detail it has nothing to do with the attached devices. I just noticed that the bug was excused away as being do to the “slow” RPi 4. I’ll share that I originally hit it on Apple’s M1 but as I expect my environment might be too unusual I replicated it on RPi 4. I have since switched to building qemu from source so I don’t know if it still happens. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921664 Title: Coroutines are racy for risc64 emu on arm64 - crash on Assertion Status in QEMU: Expired Status in qemu package in Ubuntu: Expired Bug description: Note: this could as well be "riscv64 on arm64" for being slow@slow and affect other architectures as well. The following case triggers on a Raspberry Pi4 running with arm64 on Ubuntu 21.04 [1][2]. It might trigger on other environments as well, but that is what we have seen it so far. $ wget https://github.com/carlosedp/riscv-bringup/releases/download/v1.0/UbuntuFocal-riscv64-QemuVM.tar.gz $ tar xzf UbuntuFocal-riscv64-QemuVM.tar.gz $ ./run_riscvVM.sh (wait ~2 minutes) [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting udev Kernel Device Manager... qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. This is often, but not 100% reproducible and the cases differ slightly we see either of: - qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. - qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. Rebuilding working cases has shown to make them fail, as well as rebulding (or even reinstalling) bad cases has made them work. Also the same builds on different arm64 CPUs behave differently. TL;DR: The full list of conditions influencing good/bad case here are not yet known. [1]: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview [2]: http://cdimage.ubuntu.com/daily-preinstalled/pending/hirsute-preinstalled-desktop-arm64+raspi.img.xz --- --- original report --- --- I regularly run a RISC-V (RV64GC) QEMU VM, but an update a few days ago broke it. Now when I launch it, it hits an assertion: OpenSBI v0.6 _ _ / __ \ / | _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |) | |_) || |_ \/| .__/ \___|_| |_|_/|/_| | | |_| ... Found /boot/extlinux/extlinux.conf Retrieving file: /boot/extlinux/extlinux.conf 618 bytes read in 2 ms (301.8 KiB/s) RISC-V Qemu Boot Options 1: Linux kernel-5.5.0-dirty 2: Linux kernel-5.5.0-dirty (recovery mode) Enter choice: 1:Linux kernel-5.5.0-dirty Retrieving file: /boot/initrd.img-5.5.0-dirty qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. ./run.sh: line 31: 1604 Aborted (core dumped) qemu-system-riscv64 -machine virt -nographic -smp 8 -m 8G -bios fw_payload.bin -device virtio-blk-devi ce,drive=hd0 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -drive file=riscv64-UbuntuFocal-qemu.qcow2,format=qcow2,id=hd0 -devi ce virtio-net-device,netdev=usernet -netdev user,id=usernet,$ports Interestingly this doesn't happen on the AMD64 version of Ubuntu 21.04 (fully updated). Think you have everything already, but just in case: $ lsb_release -rd Description:Ubuntu Hirsute Hippo (development branch) Release:21.04 $ uname -a Linux minimacvm 5.11.0-11-generic #12-Ubuntu SMP Mon Mar 1 19:27:36 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux (note this is a VM running on macOS/M1) $ apt-cache policy qemu qemu: Installed: 1:5.2+dfsg-9ubuntu1 Candidate: 1:5.2+dfsg-9ubuntu1 Version table: *** 1:5.2+dfsg-9ubuntu1 500 500 http://ports.ubuntu.com/ubuntu-ports hirsute/universe arm64 Packages 100 /var/lib/dpkg/status ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: qemu 1:5.2+dfsg-9ubuntu1 ProcVersionSignature: Ubuntu 5.11.0-11.12-generic 5.11.0 Uname: Linux 5.11.0-11-generic aarch64 ApportVersion: 2.20.11-0ubuntu61 Architecture: arm64 CasperMD5CheckResult: unknown CurrentDmesg: Error: command ['pkexec', 'dmesg'] failed with exit code 127: polkit-agent-helper-1: error response to PolicyKit daemon: GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: No session for cookie Error executing command as another user: Not authorized This incident has been reported. Date:
[Bug 1921664] Re: Coroutines are racy for risc64 emu on arm64 - crash on Assertion
I've been having crashes with the same assertion message, when trying to run Windows 10 ARM under a VM. But I finally figured out that what's actually crashing it is not the fact that it's Windows, it's the fact that I was attaching the virtual drive via virtual USB. If I do the same thing to an Ubuntu ARM64 guest, it *also* crashes. qemu-system-aarch64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed With the RISC-V guest, does your crash change if you change the type of attachment that's used for the virtual disk? Also, I tried enabling core dumps in libvirt, but it didn't seem to dump cores to apport. Enabling core dumps would be useful for issues like this. -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921664 Title: Coroutines are racy for risc64 emu on arm64 - crash on Assertion Status in QEMU: Expired Status in qemu package in Ubuntu: Expired Bug description: Note: this could as well be "riscv64 on arm64" for being slow@slow and affect other architectures as well. The following case triggers on a Raspberry Pi4 running with arm64 on Ubuntu 21.04 [1][2]. It might trigger on other environments as well, but that is what we have seen it so far. $ wget https://github.com/carlosedp/riscv-bringup/releases/download/v1.0/UbuntuFocal-riscv64-QemuVM.tar.gz $ tar xzf UbuntuFocal-riscv64-QemuVM.tar.gz $ ./run_riscvVM.sh (wait ~2 minutes) [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting udev Kernel Device Manager... qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. This is often, but not 100% reproducible and the cases differ slightly we see either of: - qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. - qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. Rebuilding working cases has shown to make them fail, as well as rebulding (or even reinstalling) bad cases has made them work. Also the same builds on different arm64 CPUs behave differently. TL;DR: The full list of conditions influencing good/bad case here are not yet known. [1]: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview [2]: http://cdimage.ubuntu.com/daily-preinstalled/pending/hirsute-preinstalled-desktop-arm64+raspi.img.xz --- --- original report --- --- I regularly run a RISC-V (RV64GC) QEMU VM, but an update a few days ago broke it. Now when I launch it, it hits an assertion: OpenSBI v0.6 _ _ / __ \ / | _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |) | |_) || |_ \/| .__/ \___|_| |_|_/|/_| | | |_| ... Found /boot/extlinux/extlinux.conf Retrieving file: /boot/extlinux/extlinux.conf 618 bytes read in 2 ms (301.8 KiB/s) RISC-V Qemu Boot Options 1: Linux kernel-5.5.0-dirty 2: Linux kernel-5.5.0-dirty (recovery mode) Enter choice: 1:Linux kernel-5.5.0-dirty Retrieving file: /boot/initrd.img-5.5.0-dirty qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. ./run.sh: line 31: 1604 Aborted (core dumped) qemu-system-riscv64 -machine virt -nographic -smp 8 -m 8G -bios fw_payload.bin -device virtio-blk-devi ce,drive=hd0 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -drive file=riscv64-UbuntuFocal-qemu.qcow2,format=qcow2,id=hd0 -devi ce virtio-net-device,netdev=usernet -netdev user,id=usernet,$ports Interestingly this doesn't happen on the AMD64 version of Ubuntu 21.04 (fully updated). Think you have everything already, but just in case: $ lsb_release -rd Description:Ubuntu Hirsute Hippo (development branch) Release:21.04 $ uname -a Linux minimacvm 5.11.0-11-generic #12-Ubuntu SMP Mon Mar 1 19:27:36 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux (note this is a VM running on macOS/M1) $ apt-cache policy qemu qemu: Installed: 1:5.2+dfsg-9ubuntu1 Candidate: 1:5.2+dfsg-9ubuntu1 Version table: *** 1:5.2+dfsg-9ubuntu1 500 500 http://ports.ubuntu.com/ubuntu-ports hirsute/universe arm64 Packages 100 /var/lib/dpkg/status ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: qemu 1:5.2+dfsg-9ubuntu1 ProcVersionSignature: Ubuntu 5.11.0-11.12-generic 5.11.0 Uname: Linux 5.11.0-11-generic aarch64 ApportVersion: 2.20.11-0ubuntu61 Architecture:
[Bug 1921664] Re: Coroutines are racy for risc64 emu on arm64 - crash on Assertion
[Expired for QEMU because there has been no activity for 60 days.] ** Changed in: qemu Status: Incomplete => Expired -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921664 Title: Coroutines are racy for risc64 emu on arm64 - crash on Assertion Status in QEMU: Expired Status in qemu package in Ubuntu: Expired Bug description: Note: this could as well be "riscv64 on arm64" for being slow@slow and affect other architectures as well. The following case triggers on a Raspberry Pi4 running with arm64 on Ubuntu 21.04 [1][2]. It might trigger on other environments as well, but that is what we have seen it so far. $ wget https://github.com/carlosedp/riscv-bringup/releases/download/v1.0/UbuntuFocal-riscv64-QemuVM.tar.gz $ tar xzf UbuntuFocal-riscv64-QemuVM.tar.gz $ ./run_riscvVM.sh (wait ~2 minutes) [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting udev Kernel Device Manager... qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. This is often, but not 100% reproducible and the cases differ slightly we see either of: - qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. - qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. Rebuilding working cases has shown to make them fail, as well as rebulding (or even reinstalling) bad cases has made them work. Also the same builds on different arm64 CPUs behave differently. TL;DR: The full list of conditions influencing good/bad case here are not yet known. [1]: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview [2]: http://cdimage.ubuntu.com/daily-preinstalled/pending/hirsute-preinstalled-desktop-arm64+raspi.img.xz --- --- original report --- --- I regularly run a RISC-V (RV64GC) QEMU VM, but an update a few days ago broke it. Now when I launch it, it hits an assertion: OpenSBI v0.6 _ _ / __ \ / | _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |) | |_) || |_ \/| .__/ \___|_| |_|_/|/_| | | |_| ... Found /boot/extlinux/extlinux.conf Retrieving file: /boot/extlinux/extlinux.conf 618 bytes read in 2 ms (301.8 KiB/s) RISC-V Qemu Boot Options 1: Linux kernel-5.5.0-dirty 2: Linux kernel-5.5.0-dirty (recovery mode) Enter choice: 1:Linux kernel-5.5.0-dirty Retrieving file: /boot/initrd.img-5.5.0-dirty qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. ./run.sh: line 31: 1604 Aborted (core dumped) qemu-system-riscv64 -machine virt -nographic -smp 8 -m 8G -bios fw_payload.bin -device virtio-blk-devi ce,drive=hd0 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -drive file=riscv64-UbuntuFocal-qemu.qcow2,format=qcow2,id=hd0 -devi ce virtio-net-device,netdev=usernet -netdev user,id=usernet,$ports Interestingly this doesn't happen on the AMD64 version of Ubuntu 21.04 (fully updated). Think you have everything already, but just in case: $ lsb_release -rd Description:Ubuntu Hirsute Hippo (development branch) Release:21.04 $ uname -a Linux minimacvm 5.11.0-11-generic #12-Ubuntu SMP Mon Mar 1 19:27:36 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux (note this is a VM running on macOS/M1) $ apt-cache policy qemu qemu: Installed: 1:5.2+dfsg-9ubuntu1 Candidate: 1:5.2+dfsg-9ubuntu1 Version table: *** 1:5.2+dfsg-9ubuntu1 500 500 http://ports.ubuntu.com/ubuntu-ports hirsute/universe arm64 Packages 100 /var/lib/dpkg/status ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: qemu 1:5.2+dfsg-9ubuntu1 ProcVersionSignature: Ubuntu 5.11.0-11.12-generic 5.11.0 Uname: Linux 5.11.0-11-generic aarch64 ApportVersion: 2.20.11-0ubuntu61 Architecture: arm64 CasperMD5CheckResult: unknown CurrentDmesg: Error: command ['pkexec', 'dmesg'] failed with exit code 127: polkit-agent-helper-1: error response to PolicyKit daemon: GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: No session for cookie Error executing command as another user: Not authorized This incident has been reported. Date: Mon Mar 29 02:33:25 2021 Dependencies: KvmCmdLine: COMMAND STAT EUID RUID PIDPPID %CPU COMMAND Lspci-vt: -[:00]-+-00.0 Apple Inc. Device f020 +-01.0 Red Hat, Inc. Virtio network device +-05.0
[Bug 1921664] Re: Coroutines are racy for risc64 emu on arm64 - crash on Assertion
[Expired for qemu (Ubuntu) because there has been no activity for 60 days.] ** Changed in: qemu (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921664 Title: Coroutines are racy for risc64 emu on arm64 - crash on Assertion Status in QEMU: Expired Status in qemu package in Ubuntu: Expired Bug description: Note: this could as well be "riscv64 on arm64" for being slow@slow and affect other architectures as well. The following case triggers on a Raspberry Pi4 running with arm64 on Ubuntu 21.04 [1][2]. It might trigger on other environments as well, but that is what we have seen it so far. $ wget https://github.com/carlosedp/riscv-bringup/releases/download/v1.0/UbuntuFocal-riscv64-QemuVM.tar.gz $ tar xzf UbuntuFocal-riscv64-QemuVM.tar.gz $ ./run_riscvVM.sh (wait ~2 minutes) [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting udev Kernel Device Manager... qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. This is often, but not 100% reproducible and the cases differ slightly we see either of: - qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. - qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. Rebuilding working cases has shown to make them fail, as well as rebulding (or even reinstalling) bad cases has made them work. Also the same builds on different arm64 CPUs behave differently. TL;DR: The full list of conditions influencing good/bad case here are not yet known. [1]: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview [2]: http://cdimage.ubuntu.com/daily-preinstalled/pending/hirsute-preinstalled-desktop-arm64+raspi.img.xz --- --- original report --- --- I regularly run a RISC-V (RV64GC) QEMU VM, but an update a few days ago broke it. Now when I launch it, it hits an assertion: OpenSBI v0.6 _ _ / __ \ / | _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |) | |_) || |_ \/| .__/ \___|_| |_|_/|/_| | | |_| ... Found /boot/extlinux/extlinux.conf Retrieving file: /boot/extlinux/extlinux.conf 618 bytes read in 2 ms (301.8 KiB/s) RISC-V Qemu Boot Options 1: Linux kernel-5.5.0-dirty 2: Linux kernel-5.5.0-dirty (recovery mode) Enter choice: 1:Linux kernel-5.5.0-dirty Retrieving file: /boot/initrd.img-5.5.0-dirty qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. ./run.sh: line 31: 1604 Aborted (core dumped) qemu-system-riscv64 -machine virt -nographic -smp 8 -m 8G -bios fw_payload.bin -device virtio-blk-devi ce,drive=hd0 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -drive file=riscv64-UbuntuFocal-qemu.qcow2,format=qcow2,id=hd0 -devi ce virtio-net-device,netdev=usernet -netdev user,id=usernet,$ports Interestingly this doesn't happen on the AMD64 version of Ubuntu 21.04 (fully updated). Think you have everything already, but just in case: $ lsb_release -rd Description:Ubuntu Hirsute Hippo (development branch) Release:21.04 $ uname -a Linux minimacvm 5.11.0-11-generic #12-Ubuntu SMP Mon Mar 1 19:27:36 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux (note this is a VM running on macOS/M1) $ apt-cache policy qemu qemu: Installed: 1:5.2+dfsg-9ubuntu1 Candidate: 1:5.2+dfsg-9ubuntu1 Version table: *** 1:5.2+dfsg-9ubuntu1 500 500 http://ports.ubuntu.com/ubuntu-ports hirsute/universe arm64 Packages 100 /var/lib/dpkg/status ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: qemu 1:5.2+dfsg-9ubuntu1 ProcVersionSignature: Ubuntu 5.11.0-11.12-generic 5.11.0 Uname: Linux 5.11.0-11-generic aarch64 ApportVersion: 2.20.11-0ubuntu61 Architecture: arm64 CasperMD5CheckResult: unknown CurrentDmesg: Error: command ['pkexec', 'dmesg'] failed with exit code 127: polkit-agent-helper-1: error response to PolicyKit daemon: GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: No session for cookie Error executing command as another user: Not authorized This incident has been reported. Date: Mon Mar 29 02:33:25 2021 Dependencies: KvmCmdLine: COMMAND STAT EUID RUID PIDPPID %CPU COMMAND Lspci-vt: -[:00]-+-00.0 Apple Inc. Device f020 +-01.0 Red Hat, Inc. Virtio network device
[Bug 1921664] Re: Coroutines are racy for risc64 emu on arm64 - crash on Assertion
Hmm, thanks for the hint Thomas. Of the two formerly referenced same-source different result builds: [1] => built 2021-03-23 in Hirsute => works [2] => built 2021-04-12 in Hirsute => fails [1]: https://launchpad.net/ubuntu/+source/qemu/1:5.2+dfsg-9ubuntu1/+build/21196422 [2]: https://launchpad.net/~paelzer/+archive/ubuntu/lp-1921664-testbuilds-rebuildold/+build/21392458 The default flags changed in https://launchpad.net/ubuntu/+source/dpkg/1.20.7.1ubuntu4 and according to the build logs both ran with that. Copy-Pasta from the log: dpkg (= 1.20.7.1ubuntu4), => In between those we did not switch the LTO default flags For clarification LTO is the default nowadays and we are not disabling it generally in qemu. So - yes the builds are with LTO, but both the good and the bad one are. Although looking at versions I see we have: - good case 10.2.1-23ubuntu2 - bad case 10.3.0-1ubuntu1 So maybe - while it wasn't LTO - something in 10.3 maybe even LTO- since-10.3 is what is broken? @Tommy - I don't have any of the test systems around anymore, if I'd build you a no-LTO qemu for testing what would you these days need - Hirsute, Impish, ... ? -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921664 Title: Coroutines are racy for risc64 emu on arm64 - crash on Assertion Status in QEMU: Incomplete Status in qemu package in Ubuntu: Incomplete Bug description: Note: this could as well be "riscv64 on arm64" for being slow@slow and affect other architectures as well. The following case triggers on a Raspberry Pi4 running with arm64 on Ubuntu 21.04 [1][2]. It might trigger on other environments as well, but that is what we have seen it so far. $ wget https://github.com/carlosedp/riscv-bringup/releases/download/v1.0/UbuntuFocal-riscv64-QemuVM.tar.gz $ tar xzf UbuntuFocal-riscv64-QemuVM.tar.gz $ ./run_riscvVM.sh (wait ~2 minutes) [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting udev Kernel Device Manager... qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. This is often, but not 100% reproducible and the cases differ slightly we see either of: - qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. - qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. Rebuilding working cases has shown to make them fail, as well as rebulding (or even reinstalling) bad cases has made them work. Also the same builds on different arm64 CPUs behave differently. TL;DR: The full list of conditions influencing good/bad case here are not yet known. [1]: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview [2]: http://cdimage.ubuntu.com/daily-preinstalled/pending/hirsute-preinstalled-desktop-arm64+raspi.img.xz --- --- original report --- --- I regularly run a RISC-V (RV64GC) QEMU VM, but an update a few days ago broke it. Now when I launch it, it hits an assertion: OpenSBI v0.6 _ _ / __ \ / | _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |) | |_) || |_ \/| .__/ \___|_| |_|_/|/_| | | |_| ... Found /boot/extlinux/extlinux.conf Retrieving file: /boot/extlinux/extlinux.conf 618 bytes read in 2 ms (301.8 KiB/s) RISC-V Qemu Boot Options 1: Linux kernel-5.5.0-dirty 2: Linux kernel-5.5.0-dirty (recovery mode) Enter choice: 1:Linux kernel-5.5.0-dirty Retrieving file: /boot/initrd.img-5.5.0-dirty qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. ./run.sh: line 31: 1604 Aborted (core dumped) qemu-system-riscv64 -machine virt -nographic -smp 8 -m 8G -bios fw_payload.bin -device virtio-blk-devi ce,drive=hd0 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -drive file=riscv64-UbuntuFocal-qemu.qcow2,format=qcow2,id=hd0 -devi ce virtio-net-device,netdev=usernet -netdev user,id=usernet,$ports Interestingly this doesn't happen on the AMD64 version of Ubuntu 21.04 (fully updated). Think you have everything already, but just in case: $ lsb_release -rd Description:Ubuntu Hirsute Hippo (development branch) Release:21.04 $ uname -a Linux minimacvm 5.11.0-11-generic #12-Ubuntu SMP Mon Mar 1 19:27:36 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux (note this is a VM running on macOS/M1) $ apt-cache policy qemu qemu: Installed: 1:5.2+dfsg-9ubuntu1
[Bug 1921664] Re: Coroutines are racy for risc64 emu on arm64 - crash on Assertion
@Christian & Tommy : Could you please check whether the problematic binaries were built with link-time optimization, i.e. with -flto ? If so, does the problem go away when you rebuild the package without LTO? ** Changed in: qemu Status: New => Incomplete ** Changed in: qemu (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of qemu- devel-ml, which is subscribed to QEMU. https://bugs.launchpad.net/bugs/1921664 Title: Coroutines are racy for risc64 emu on arm64 - crash on Assertion Status in QEMU: Incomplete Status in qemu package in Ubuntu: Incomplete Bug description: Note: this could as well be "riscv64 on arm64" for being slow@slow and affect other architectures as well. The following case triggers on a Raspberry Pi4 running with arm64 on Ubuntu 21.04 [1][2]. It might trigger on other environments as well, but that is what we have seen it so far. $ wget https://github.com/carlosedp/riscv-bringup/releases/download/v1.0/UbuntuFocal-riscv64-QemuVM.tar.gz $ tar xzf UbuntuFocal-riscv64-QemuVM.tar.gz $ ./run_riscvVM.sh (wait ~2 minutes) [ OK ] Reached target Local File Systems (Pre). [ OK ] Reached target Local File Systems. Starting udev Kernel Device Manager... qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. This is often, but not 100% reproducible and the cases differ slightly we see either of: - qemu-system-riscv64: ../../util/qemu-coroutine-lock.c:57: qemu_co_queue_wait_impl: Assertion `qemu_in_coroutine()' failed. - qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. Rebuilding working cases has shown to make them fail, as well as rebulding (or even reinstalling) bad cases has made them work. Also the same builds on different arm64 CPUs behave differently. TL;DR: The full list of conditions influencing good/bad case here are not yet known. [1]: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi#1-overview [2]: http://cdimage.ubuntu.com/daily-preinstalled/pending/hirsute-preinstalled-desktop-arm64+raspi.img.xz --- --- original report --- --- I regularly run a RISC-V (RV64GC) QEMU VM, but an update a few days ago broke it. Now when I launch it, it hits an assertion: OpenSBI v0.6 _ _ / __ \ / | _ \_ _| | | | |_ __ ___ _ __ | (___ | |_) || | | | | | '_ \ / _ \ '_ \ \___ \| _ < | | | |__| | |_) | __/ | | |) | |_) || |_ \/| .__/ \___|_| |_|_/|/_| | | |_| ... Found /boot/extlinux/extlinux.conf Retrieving file: /boot/extlinux/extlinux.conf 618 bytes read in 2 ms (301.8 KiB/s) RISC-V Qemu Boot Options 1: Linux kernel-5.5.0-dirty 2: Linux kernel-5.5.0-dirty (recovery mode) Enter choice: 1:Linux kernel-5.5.0-dirty Retrieving file: /boot/initrd.img-5.5.0-dirty qemu-system-riscv64: ../../block/aio_task.c:64: aio_task_pool_wait_one: Assertion `qemu_coroutine_self() == pool->main_co' failed. ./run.sh: line 31: 1604 Aborted (core dumped) qemu-system-riscv64 -machine virt -nographic -smp 8 -m 8G -bios fw_payload.bin -device virtio-blk-devi ce,drive=hd0 -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -drive file=riscv64-UbuntuFocal-qemu.qcow2,format=qcow2,id=hd0 -devi ce virtio-net-device,netdev=usernet -netdev user,id=usernet,$ports Interestingly this doesn't happen on the AMD64 version of Ubuntu 21.04 (fully updated). Think you have everything already, but just in case: $ lsb_release -rd Description:Ubuntu Hirsute Hippo (development branch) Release:21.04 $ uname -a Linux minimacvm 5.11.0-11-generic #12-Ubuntu SMP Mon Mar 1 19:27:36 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux (note this is a VM running on macOS/M1) $ apt-cache policy qemu qemu: Installed: 1:5.2+dfsg-9ubuntu1 Candidate: 1:5.2+dfsg-9ubuntu1 Version table: *** 1:5.2+dfsg-9ubuntu1 500 500 http://ports.ubuntu.com/ubuntu-ports hirsute/universe arm64 Packages 100 /var/lib/dpkg/status ProblemType: Bug DistroRelease: Ubuntu 21.04 Package: qemu 1:5.2+dfsg-9ubuntu1 ProcVersionSignature: Ubuntu 5.11.0-11.12-generic 5.11.0 Uname: Linux 5.11.0-11-generic aarch64 ApportVersion: 2.20.11-0ubuntu61 Architecture: arm64 CasperMD5CheckResult: unknown CurrentDmesg: Error: command ['pkexec', 'dmesg'] failed with exit code 127: polkit-agent-helper-1: error response to PolicyKit daemon: GDBus.Error:org.freedesktop.PolicyKit1.Error.Failed: No session for cookie Error executing command as another user: Not authorized This incident has been reported. Date: Mon Mar 29 02:33:25 2021 Dependencies: