We've done some additional analysis (of a qemu coredump) and determined that vCPU threads also cache coroutines; this means that the predicted hard max is significantly higher than the original [Impact] section suggested (and better explains the behavior observed when executing the test plan).
I've updated the [Impact] section with a number of tweaks that take this finding into account. ** Description changed: [ Impact ] QEMU 8.2.2's coroutine pool implementation can hit the Linux vm.max_map_count limit (`65530` in Jammy, `1048576` in Noble), causing QEMU to abort with "failed to allocate memory for stack" or "failed to set up stack guard page" during coroutine creation. Each coroutine creates two anonymous mappings with mmap(2). When a coroutine terminates it is cached in a global "release pool"; once that is full (2 x max pool size), terminated coroutines are cached in a thread-local "alloc pool" (up to the max pool size, per thread). When - the thread-local pool is empty, the global pool is moved to the thread- + the thread-local pool is empty, the global pool is moved to a thread- local pool. Pooled coroutines are only freed once both pools are full. + + There are 3 kinds of threads which run coroutines (have non-empty + thread-local pools): the main thread, vCPU threads, and iothreads. In (unpatched) QEMU 8.2.2 the theoretical maximum of idle coroutines in each thread-local pool is given by: (see hw/block/virtio-blk.c:1638) - 64 + Σ (1 to i of virtio-blk devices) (num_queues_i × queue_size_i / 2) + pool_max_size = 64 + Σ (1 to i of virtio-blk devices) (num_queues_i × + queue_size_i / 2) Defaults: - queue_size_i = 256 (hw/block/virtio-blk.c:1722) - num_queues_i = vCPUs (hw/virtio/virtio-pci.c:2466) - - The global pool is twice as large as the max pool size (util/qemu- - coroutine.c:102), so: - - theoretical maximum mappings = 2 × (iothreads + 2) × thread-local max + num_queues_i = vCPUs (hw/virtio/virtio-pci.c:2466) + + The global pool is twice as large as `pool_max_size` (util/qemu- + coroutine.c:102); since thread-local pools steal the global pool, they + may have at most 2 × pool_max_size coroutines: + + theoretical maximum mappings = 2 × (vCPUs + iothreads + 2) × 2 × + pool_max_size A VM with 32 vCPUs and two virtio-blk disks gets a theoretical maximum - mappings of 49920; add a third disk or a few more vCPUs and it is over - 65530 for older kernels. + mappings of 1122816, higher than the Noble kernel default. In practice the number of coroutines that will actually be allocated is typically much lower than this, since the total number of allocated - coroutines is a high water mark of all concurrently active coroutines. - Many small IOs all at once drain both pools, causing more coroutines to - be allocated. + coroutines depends on concurrent IO. Many small IOs all at once drain + both pools, causing more coroutines to be allocated. Coroutine growth + after VM start is hypothesized to behave logarithmically. This behavior was introduced in upstream 4c41c69e05fe28c (v7.0.0-rc0) and therefore only affects Ubuntu 24.04. The user who reported this experienced crashes while running Noble's QEMU in a container on a 5.15 kernel (with the old `vm.max_map_count=65530`). Even with a relatively modest VM - configuration (36 vCPUs, 3 disks), the theoretical max of coroutine VMAs - is higher than this limit. + configuration (36 vCPUs, 3 disks), we’ve observed numbers of cached VMAs + in the 30,000 - 70,000 range. Crashes are expected to be significantly less likely with Noble's sysctl - config since reaching the default max_map_count would require the pool's - theoretical max size to be increased substantially (using 128+ IOThreads - or adding many additional virtio-blk devices). Crashes with this - configuration have not been reported. + config since reaching the default vm.max_map_count would require a + significant fraction of the total allocated coroutines to be active + concurrently. Crashes with the expected Noble configuration have not + been reported. While crashes can be worked around by adjusting the sysctl, this is still a resource leak that can leave a large number of unused coroutines - allocated but unusable. 86a637e4810 allows for better reuse of of those - cached idle coroutines. + allocated but unusable. 86a637e4810 allows for better reuse of those + cached idle coroutines (2,000 - 5,000 as opposed to the 30,000 - 70,000 + cited above). + [ Test Plan ] 1. Run the upstream iotests (in a virtual machine) to exercise coroutines (not all of these tests are run during package build): ```sh gbp pq switch sudo apt build-dep ./ sudo apt install seabios ipxe-qemu make -f debian/rules configure-qemu # Binaries needed for iotests ninja -C b/qemu modules make -f debian/rules build-x86-optionrom cp -a b/optionrom/*.bin /usr/share/seabios/* /usr/lib/ipxe/qemu/* b/qemu/qemu-bundle/usr/share/qemu/ make -C b/qemu/ check-block cd b/qemu/ ./tests/qemu-iotests/check -qcow2 # Expected failures 151 281 307 graph-changes-while-io ``` 2. Run the Ubuntu Server team's qemu-migration-test [1] 3. The following procedure reproduces mmap exhaustion: ``` lxc launch ubuntu:noble --vm -c limits.cpu=8 -c limits.memory=16GiB n0 ``` In the VM: ```sh sudo apt install libvirt-daemon-system virtinst wget https://cloud-images.ubuntu.com/daily/server/noble/current/noble-server-cloudimg-amd64.img sudo cp noble-server-cloudimg-amd64.img /var/lib/libvirt/images/testvm-root.qcow2 sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data2.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data3.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data4.raw 10G cat > user-data <<EOF #cloud-config password: ubuntu chpasswd: expire: false ssh_pwauth: true package_update: true packages: - fio EOF sudo virt-install \ -n testvm \ --os-variant=ubuntu24.04 \ --ram=8192 --vcpus=8 \ --iothreads 8 \ --import \ --disk path=/var/lib/libvirt/images/testvm-root.qcow2,bus=virtio,cache=writethrough \ --disk path=/var/lib/libvirt/images/testvm-data.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data2.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data3.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data4.raw,bus=virtio,cache=none,driver.io=io_uring \ --graphics none \ --network network=default \ --cloud-init user-data=user-data # Ctrl+Shift+a+] sudo virsh shutdown testvm ``` Limit write throughput in the block layer so that writes allocate a coroutine but aren't able to complete right away: ```sh for disk in vdb vdc vdd vde; do sudo virsh blkdeviotune testvm "${disk}" --total-iops-sec 2000 --current done ``` Use `sudo virsh edit testvm` to pin each disk to its own iothread; otherwise all IO will occur on the main thread. Add `iothread='1'` to the `<driver>` section of each 'testvm-data' disk, incrementing the number for each disk. ```sh # Reduce the max map count by a factor of four to match the reduction in # used IOThreads in the test environment sudo sysctl -w vm.max_map_count=16384 sudo virsh start testvm sudo virsh console testvm ``` On the host, you can monitor the maps with: ```sh pid=$(pidof qemu-system-x86_64) while sleep 1; do sudo grep -c '' "/proc/${pid}/maps"; done ``` In the testvm: ```sh while true; do for dev in vdb vdc vdd vde; do sudo fio "--name=/dev/${dev}" \ --ioengine=io_uring \ --direct=1 \ --runtime=10 \ --time_based=1 \ --rw=randread \ --bs=4k \ --iodepth=128 \ --numjobs=32 \ --group_reporting \ "--filename=/dev/${dev}" done done ``` Patched behavior: /proc/pid/maps levels off and stays consistent through the fio run. (With these parameters we observed approximately 5376). Unpatched behavior: /proc/pid/maps increases with each fio run until the VM crashes with one of: - failed to allocate memory for stack: Cannot allocate memory - failed to set up stack guard page: Cannot allocate memory 4. Performance regression test (results should be provided both before & after): ```sh for bs in 4k 128k; do for iodepth in 1 8 16 32 64; do sudo fio --output-format=json \ --output=fio-$bs-$iodepth.json \ --ioengine=libaio \ --direct=1 \ --numjobs=$(nproc) \ --runtime=30 \ --ramp_time=5 \ --time_based=1 \ --rw=randread \ --bs=$bs \ --iodepth=$iodepth \ --name=job1 \ --filename=/dev/vdb done done ``` [1] https://code.launchpad.net/~ubuntu-server/ubuntu/+source/qemu- migration-test/+git/qemu-migration-test [ Where problems could occur ] The patches restructure the way coroutines are allocated and deallocated. Coroutines are used extensively in the QMP/HMP protocol implementation, block drivers and live migrations; this change is somewhat fundamental to QEMU's coroutine infrastructure. If the change were wrong/broken, it could cause: - failing block device drivers, causing failed boots or broken I/O operations within affected guests - incorrect/strange behavior when interacting with a QEMU process over QMP/HMP - failed/broken live migrations - performance regressions as the patch introduces a lock in the coroutine pool hot path (coroutine_pool_get) Upstream has two follow-ups to the refactor, 9352f80cd926fe2dde7c89b93ee33bb0356ff40e and 25bc7d16fa96b0ff881c83ed225ea380fe427c78. 9352f80cd will be included in the upload; 25bc7d16fa fixes a false-positive compiler warning. 86a637e4 parses the contents of `/proc/sys/vm/max_map_count` incorrectly [1]. This means that without Hector's linked follow-up, the patch removes any limitation on the maximum size of the coroutine idle pool. The introduction of a read() from procfs will introduce an apparmor denial for VMs using libvirt's AppArmor security driver [2]. If AppArmor is active for a VM, this change will remove any limitation of the maximum size of the coroutine idle pool. If [1] and [2] are also applied, workloads which require many coroutines may experience a performance regression (as the patches set a hard limit on the size of the coroutine pool, so coroutines needed above the pool size are allocated/deallocated on demand). The coroutine pool size is a function of `vm.max_map_count`, so increasing the max map count is the recommended workaround: ``` sysctl -w vm.max_map_count=1048576 ``` It's likely that workloads running into this performance regression would also have been affected by the original bug. [1] https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html [2] https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 [ Other Info ] QEMU's coroutines are a userspace threading implementation. They are used frequently in QEMU's virtual disk drivers. Noble's QEMU 8.2.2 keeps a thread-local list of coroutines `alloc_pool` (util/qemu-coroutine.c:40); this list can grow to thousands or even tens of thousands of coroutines _for each thread_. Each coroutine has two VMAs created with mmap(2). If there are enough threads available to the QEMU process, the number of coroutines may cause the total number of mappings to exceed `vm.max_map_count`, causing the QEMU process to fail to allocate memory and crash. From upstream, this bug occurs because "per-thread pools can grow to tens of thousands of coroutines. Each coroutine causes 2 virtual memory areas to be created. Eventually vm.max_map_count is reached and memory- related syscalls fail. The per-thread pool sizes are non-uniform and depend on past coroutine usage in each thread, so it's possible for one thread to have a large pool while another thread's pool is empty." The new approach "does not leave large numbers of coroutines pooled in a thread that may not use them again. In order to perform well it amortizes the cost of global pool accesses by working in batches of coroutines instead of individual coroutines." Upstream commit (coroutine: cap per-thread local pool size): https://gitlab.com/qemu-project/qemu/-/commit/86a637e4 Upstream commit (coroutine: reserve 5,000 mappings): https://gitlab.com/qemu-project/qemu/-/commit/9352f80c Upstream commit (util/coroutine: fix -Werror=maybe-uninitialized false-positive): https://gitlab.com/qemu-project/qemu/-/commit/25bc7d Additionally, the following bugs were identified in 86a637e4 which require follow-ups for the patch to work as intended: Missing AppArmor rule: https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 Incorrect string parsing: https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html ** Description changed: [ Impact ] QEMU 8.2.2's coroutine pool implementation can hit the Linux vm.max_map_count limit (`65530` in Jammy, `1048576` in Noble), causing QEMU to abort with "failed to allocate memory for stack" or "failed to set up stack guard page" during coroutine creation. Each coroutine creates two anonymous mappings with mmap(2). When a coroutine terminates it is cached in a global "release pool"; once that is full (2 x max pool size), terminated coroutines are cached in a thread-local "alloc pool" (up to the max pool size, per thread). When the thread-local pool is empty, the global pool is moved to a thread- local pool. Pooled coroutines are only freed once both pools are full. There are 3 kinds of threads which run coroutines (have non-empty thread-local pools): the main thread, vCPU threads, and iothreads. In (unpatched) QEMU 8.2.2 the theoretical maximum of idle coroutines in each thread-local pool is given by: (see hw/block/virtio-blk.c:1638) pool_max_size = 64 + Σ (1 to i of virtio-blk devices) (num_queues_i × queue_size_i / 2) Defaults: queue_size_i = 256 (hw/block/virtio-blk.c:1722) - num_queues_i = vCPUs (hw/virtio/virtio-pci.c:2466) + num_queues_i = vCPUs (hw/virtio/virtio-pci.c:2466) The global pool is twice as large as `pool_max_size` (util/qemu- coroutine.c:102); since thread-local pools steal the global pool, they may have at most 2 × pool_max_size coroutines: theoretical maximum mappings = 2 × (vCPUs + iothreads + 2) × 2 × pool_max_size A VM with 32 vCPUs and two virtio-blk disks gets a theoretical maximum mappings of 1122816, higher than the Noble kernel default. In practice the number of coroutines that will actually be allocated is typically much lower than this, since the total number of allocated - coroutines depends on concurrent IO. Many small IOs all at once drain - both pools, causing more coroutines to be allocated. Coroutine growth - after VM start is hypothesized to behave logarithmically. + coroutines depends on concurrent IO forcing an increase in the size of + the thread-local caches. Many small IOs all at once drain both pools, + causing more coroutines to be allocated. Coroutine growth after VM start + is hypothesized to behave logarithmically. This behavior was introduced in upstream 4c41c69e05fe28c (v7.0.0-rc0) and therefore only affects Ubuntu 24.04. The user who reported this experienced crashes while running Noble's QEMU in a container on a 5.15 kernel (with the old `vm.max_map_count=65530`). Even with a relatively modest VM configuration (36 vCPUs, 3 disks), we’ve observed numbers of cached VMAs in the 30,000 - 70,000 range. Crashes are expected to be significantly less likely with Noble's sysctl config since reaching the default vm.max_map_count would require a significant fraction of the total allocated coroutines to be active concurrently. Crashes with the expected Noble configuration have not been reported. While crashes can be worked around by adjusting the sysctl, this is still a resource leak that can leave a large number of unused coroutines allocated but unusable. 86a637e4810 allows for better reuse of those cached idle coroutines (2,000 - 5,000 as opposed to the 30,000 - 70,000 cited above). - [ Test Plan ] 1. Run the upstream iotests (in a virtual machine) to exercise coroutines (not all of these tests are run during package build): ```sh gbp pq switch sudo apt build-dep ./ sudo apt install seabios ipxe-qemu make -f debian/rules configure-qemu # Binaries needed for iotests ninja -C b/qemu modules make -f debian/rules build-x86-optionrom cp -a b/optionrom/*.bin /usr/share/seabios/* /usr/lib/ipxe/qemu/* b/qemu/qemu-bundle/usr/share/qemu/ make -C b/qemu/ check-block cd b/qemu/ ./tests/qemu-iotests/check -qcow2 # Expected failures 151 281 307 graph-changes-while-io ``` 2. Run the Ubuntu Server team's qemu-migration-test [1] 3. The following procedure reproduces mmap exhaustion: ``` lxc launch ubuntu:noble --vm -c limits.cpu=8 -c limits.memory=16GiB n0 ``` In the VM: ```sh sudo apt install libvirt-daemon-system virtinst wget https://cloud-images.ubuntu.com/daily/server/noble/current/noble-server-cloudimg-amd64.img sudo cp noble-server-cloudimg-amd64.img /var/lib/libvirt/images/testvm-root.qcow2 sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data2.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data3.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data4.raw 10G cat > user-data <<EOF #cloud-config password: ubuntu chpasswd: expire: false ssh_pwauth: true package_update: true packages: - fio EOF sudo virt-install \ -n testvm \ --os-variant=ubuntu24.04 \ --ram=8192 --vcpus=8 \ --iothreads 8 \ --import \ --disk path=/var/lib/libvirt/images/testvm-root.qcow2,bus=virtio,cache=writethrough \ --disk path=/var/lib/libvirt/images/testvm-data.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data2.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data3.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data4.raw,bus=virtio,cache=none,driver.io=io_uring \ --graphics none \ --network network=default \ --cloud-init user-data=user-data # Ctrl+Shift+a+] sudo virsh shutdown testvm ``` Limit write throughput in the block layer so that writes allocate a coroutine but aren't able to complete right away: ```sh for disk in vdb vdc vdd vde; do sudo virsh blkdeviotune testvm "${disk}" --total-iops-sec 2000 --current done ``` Use `sudo virsh edit testvm` to pin each disk to its own iothread; otherwise all IO will occur on the main thread. Add `iothread='1'` to the `<driver>` section of each 'testvm-data' disk, incrementing the number for each disk. ```sh # Reduce the max map count by a factor of four to match the reduction in # used IOThreads in the test environment sudo sysctl -w vm.max_map_count=16384 sudo virsh start testvm sudo virsh console testvm ``` On the host, you can monitor the maps with: ```sh pid=$(pidof qemu-system-x86_64) while sleep 1; do sudo grep -c '' "/proc/${pid}/maps"; done ``` In the testvm: ```sh while true; do for dev in vdb vdc vdd vde; do sudo fio "--name=/dev/${dev}" \ --ioengine=io_uring \ --direct=1 \ --runtime=10 \ --time_based=1 \ --rw=randread \ --bs=4k \ --iodepth=128 \ --numjobs=32 \ --group_reporting \ "--filename=/dev/${dev}" done done ``` Patched behavior: /proc/pid/maps levels off and stays consistent through the fio run. (With these parameters we observed approximately 5376). Unpatched behavior: /proc/pid/maps increases with each fio run until the VM crashes with one of: - failed to allocate memory for stack: Cannot allocate memory - failed to set up stack guard page: Cannot allocate memory 4. Performance regression test (results should be provided both before & after): ```sh for bs in 4k 128k; do for iodepth in 1 8 16 32 64; do sudo fio --output-format=json \ --output=fio-$bs-$iodepth.json \ --ioengine=libaio \ --direct=1 \ --numjobs=$(nproc) \ --runtime=30 \ --ramp_time=5 \ --time_based=1 \ --rw=randread \ --bs=$bs \ --iodepth=$iodepth \ --name=job1 \ --filename=/dev/vdb done done ``` [1] https://code.launchpad.net/~ubuntu-server/ubuntu/+source/qemu- migration-test/+git/qemu-migration-test [ Where problems could occur ] The patches restructure the way coroutines are allocated and deallocated. Coroutines are used extensively in the QMP/HMP protocol implementation, block drivers and live migrations; this change is somewhat fundamental to QEMU's coroutine infrastructure. If the change were wrong/broken, it could cause: - failing block device drivers, causing failed boots or broken I/O operations within affected guests - incorrect/strange behavior when interacting with a QEMU process over QMP/HMP - failed/broken live migrations - performance regressions as the patch introduces a lock in the coroutine pool hot path (coroutine_pool_get) Upstream has two follow-ups to the refactor, 9352f80cd926fe2dde7c89b93ee33bb0356ff40e and 25bc7d16fa96b0ff881c83ed225ea380fe427c78. 9352f80cd will be included in the upload; 25bc7d16fa fixes a false-positive compiler warning. 86a637e4 parses the contents of `/proc/sys/vm/max_map_count` incorrectly [1]. This means that without Hector's linked follow-up, the patch removes any limitation on the maximum size of the coroutine idle pool. The introduction of a read() from procfs will introduce an apparmor denial for VMs using libvirt's AppArmor security driver [2]. If AppArmor is active for a VM, this change will remove any limitation of the maximum size of the coroutine idle pool. If [1] and [2] are also applied, workloads which require many coroutines may experience a performance regression (as the patches set a hard limit on the size of the coroutine pool, so coroutines needed above the pool size are allocated/deallocated on demand). The coroutine pool size is a function of `vm.max_map_count`, so increasing the max map count is the recommended workaround: ``` sysctl -w vm.max_map_count=1048576 ``` It's likely that workloads running into this performance regression would also have been affected by the original bug. [1] https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html [2] https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 [ Other Info ] QEMU's coroutines are a userspace threading implementation. They are used frequently in QEMU's virtual disk drivers. Noble's QEMU 8.2.2 keeps a thread-local list of coroutines `alloc_pool` (util/qemu-coroutine.c:40); this list can grow to thousands or even tens of thousands of coroutines _for each thread_. Each coroutine has two VMAs created with mmap(2). If there are enough threads available to the QEMU process, the number of coroutines may cause the total number of mappings to exceed `vm.max_map_count`, causing the QEMU process to fail to allocate memory and crash. From upstream, this bug occurs because "per-thread pools can grow to tens of thousands of coroutines. Each coroutine causes 2 virtual memory areas to be created. Eventually vm.max_map_count is reached and memory- related syscalls fail. The per-thread pool sizes are non-uniform and depend on past coroutine usage in each thread, so it's possible for one thread to have a large pool while another thread's pool is empty." The new approach "does not leave large numbers of coroutines pooled in a thread that may not use them again. In order to perform well it amortizes the cost of global pool accesses by working in batches of coroutines instead of individual coroutines." Upstream commit (coroutine: cap per-thread local pool size): https://gitlab.com/qemu-project/qemu/-/commit/86a637e4 Upstream commit (coroutine: reserve 5,000 mappings): https://gitlab.com/qemu-project/qemu/-/commit/9352f80c Upstream commit (util/coroutine: fix -Werror=maybe-uninitialized false-positive): https://gitlab.com/qemu-project/qemu/-/commit/25bc7d Additionally, the following bugs were identified in 86a637e4 which require follow-ups for the patch to work as intended: Missing AppArmor rule: https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 Incorrect string parsing: https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html ** Description changed: [ Impact ] QEMU 8.2.2's coroutine pool implementation can hit the Linux vm.max_map_count limit (`65530` in Jammy, `1048576` in Noble), causing QEMU to abort with "failed to allocate memory for stack" or "failed to set up stack guard page" during coroutine creation. Each coroutine creates two anonymous mappings with mmap(2). When a coroutine terminates it is cached in a global "release pool"; once that is full (2 x max pool size), terminated coroutines are cached in a thread-local "alloc pool" (up to the max pool size, per thread). When the thread-local pool is empty, the global pool is moved to a thread- local pool. Pooled coroutines are only freed once both pools are full. There are 3 kinds of threads which run coroutines (have non-empty thread-local pools): the main thread, vCPU threads, and iothreads. In (unpatched) QEMU 8.2.2 the theoretical maximum of idle coroutines in each thread-local pool is given by: (see hw/block/virtio-blk.c:1638) pool_max_size = 64 + Σ (1 to i of virtio-blk devices) (num_queues_i × queue_size_i / 2) Defaults: queue_size_i = 256 (hw/block/virtio-blk.c:1722) num_queues_i = vCPUs (hw/virtio/virtio-pci.c:2466) The global pool is twice as large as `pool_max_size` (util/qemu- coroutine.c:102); since thread-local pools steal the global pool, they may have at most 2 × pool_max_size coroutines: theoretical maximum mappings = 2 × (vCPUs + iothreads + 2) × 2 × pool_max_size A VM with 32 vCPUs and two virtio-blk disks gets a theoretical maximum mappings of 1122816, higher than the Noble kernel default. In practice the number of coroutines that will actually be allocated is typically much lower than this, since the total number of allocated coroutines depends on concurrent IO forcing an increase in the size of the thread-local caches. Many small IOs all at once drain both pools, causing more coroutines to be allocated. Coroutine growth after VM start is hypothesized to behave logarithmically. This behavior was introduced in upstream 4c41c69e05fe28c (v7.0.0-rc0) and therefore only affects Ubuntu 24.04. The user who reported this experienced crashes while running Noble's QEMU in a container on a 5.15 kernel (with the old `vm.max_map_count=65530`). Even with a relatively modest VM configuration (36 vCPUs, 3 disks), we’ve observed numbers of cached VMAs in the 30,000 - 70,000 range. Crashes are expected to be significantly less likely with Noble's sysctl config since reaching the default vm.max_map_count would require a significant fraction of the total allocated coroutines to be active - concurrently. Crashes with the expected Noble configuration have not - been reported. + concurrently (or a 10x increase in the number of vCPUs). Crashes with + the expected Noble configuration have not been reported. While crashes can be worked around by adjusting the sysctl, this is still a resource leak that can leave a large number of unused coroutines allocated but unusable. 86a637e4810 allows for better reuse of those cached idle coroutines (2,000 - 5,000 as opposed to the 30,000 - 70,000 cited above). [ Test Plan ] 1. Run the upstream iotests (in a virtual machine) to exercise coroutines (not all of these tests are run during package build): ```sh gbp pq switch sudo apt build-dep ./ sudo apt install seabios ipxe-qemu make -f debian/rules configure-qemu # Binaries needed for iotests ninja -C b/qemu modules make -f debian/rules build-x86-optionrom cp -a b/optionrom/*.bin /usr/share/seabios/* /usr/lib/ipxe/qemu/* b/qemu/qemu-bundle/usr/share/qemu/ make -C b/qemu/ check-block cd b/qemu/ ./tests/qemu-iotests/check -qcow2 # Expected failures 151 281 307 graph-changes-while-io ``` 2. Run the Ubuntu Server team's qemu-migration-test [1] 3. The following procedure reproduces mmap exhaustion: ``` lxc launch ubuntu:noble --vm -c limits.cpu=8 -c limits.memory=16GiB n0 ``` In the VM: ```sh sudo apt install libvirt-daemon-system virtinst wget https://cloud-images.ubuntu.com/daily/server/noble/current/noble-server-cloudimg-amd64.img sudo cp noble-server-cloudimg-amd64.img /var/lib/libvirt/images/testvm-root.qcow2 sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data2.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data3.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data4.raw 10G cat > user-data <<EOF #cloud-config password: ubuntu chpasswd: expire: false ssh_pwauth: true package_update: true packages: - fio EOF sudo virt-install \ -n testvm \ --os-variant=ubuntu24.04 \ --ram=8192 --vcpus=8 \ --iothreads 8 \ --import \ --disk path=/var/lib/libvirt/images/testvm-root.qcow2,bus=virtio,cache=writethrough \ --disk path=/var/lib/libvirt/images/testvm-data.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data2.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data3.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data4.raw,bus=virtio,cache=none,driver.io=io_uring \ --graphics none \ --network network=default \ --cloud-init user-data=user-data # Ctrl+Shift+a+] sudo virsh shutdown testvm ``` Limit write throughput in the block layer so that writes allocate a coroutine but aren't able to complete right away: ```sh for disk in vdb vdc vdd vde; do sudo virsh blkdeviotune testvm "${disk}" --total-iops-sec 2000 --current done ``` Use `sudo virsh edit testvm` to pin each disk to its own iothread; otherwise all IO will occur on the main thread. Add `iothread='1'` to the `<driver>` section of each 'testvm-data' disk, incrementing the number for each disk. ```sh # Reduce the max map count by a factor of four to match the reduction in # used IOThreads in the test environment sudo sysctl -w vm.max_map_count=16384 sudo virsh start testvm sudo virsh console testvm ``` On the host, you can monitor the maps with: ```sh pid=$(pidof qemu-system-x86_64) while sleep 1; do sudo grep -c '' "/proc/${pid}/maps"; done ``` In the testvm: ```sh while true; do for dev in vdb vdc vdd vde; do sudo fio "--name=/dev/${dev}" \ --ioengine=io_uring \ --direct=1 \ --runtime=10 \ --time_based=1 \ --rw=randread \ --bs=4k \ --iodepth=128 \ --numjobs=32 \ --group_reporting \ "--filename=/dev/${dev}" done done ``` Patched behavior: /proc/pid/maps levels off and stays consistent through the fio run. (With these parameters we observed approximately 5376). Unpatched behavior: /proc/pid/maps increases with each fio run until the VM crashes with one of: - failed to allocate memory for stack: Cannot allocate memory - failed to set up stack guard page: Cannot allocate memory 4. Performance regression test (results should be provided both before & after): ```sh for bs in 4k 128k; do for iodepth in 1 8 16 32 64; do sudo fio --output-format=json \ --output=fio-$bs-$iodepth.json \ --ioengine=libaio \ --direct=1 \ --numjobs=$(nproc) \ --runtime=30 \ --ramp_time=5 \ --time_based=1 \ --rw=randread \ --bs=$bs \ --iodepth=$iodepth \ --name=job1 \ --filename=/dev/vdb done done ``` [1] https://code.launchpad.net/~ubuntu-server/ubuntu/+source/qemu- migration-test/+git/qemu-migration-test [ Where problems could occur ] The patches restructure the way coroutines are allocated and deallocated. Coroutines are used extensively in the QMP/HMP protocol implementation, block drivers and live migrations; this change is somewhat fundamental to QEMU's coroutine infrastructure. If the change were wrong/broken, it could cause: - failing block device drivers, causing failed boots or broken I/O operations within affected guests - incorrect/strange behavior when interacting with a QEMU process over QMP/HMP - failed/broken live migrations - performance regressions as the patch introduces a lock in the coroutine pool hot path (coroutine_pool_get) Upstream has two follow-ups to the refactor, 9352f80cd926fe2dde7c89b93ee33bb0356ff40e and 25bc7d16fa96b0ff881c83ed225ea380fe427c78. 9352f80cd will be included in the upload; 25bc7d16fa fixes a false-positive compiler warning. 86a637e4 parses the contents of `/proc/sys/vm/max_map_count` incorrectly [1]. This means that without Hector's linked follow-up, the patch removes any limitation on the maximum size of the coroutine idle pool. The introduction of a read() from procfs will introduce an apparmor denial for VMs using libvirt's AppArmor security driver [2]. If AppArmor is active for a VM, this change will remove any limitation of the maximum size of the coroutine idle pool. If [1] and [2] are also applied, workloads which require many coroutines may experience a performance regression (as the patches set a hard limit on the size of the coroutine pool, so coroutines needed above the pool size are allocated/deallocated on demand). The coroutine pool size is a function of `vm.max_map_count`, so increasing the max map count is the recommended workaround: ``` sysctl -w vm.max_map_count=1048576 ``` It's likely that workloads running into this performance regression would also have been affected by the original bug. [1] https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html [2] https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 [ Other Info ] QEMU's coroutines are a userspace threading implementation. They are used frequently in QEMU's virtual disk drivers. Noble's QEMU 8.2.2 keeps a thread-local list of coroutines `alloc_pool` (util/qemu-coroutine.c:40); this list can grow to thousands or even tens of thousands of coroutines _for each thread_. Each coroutine has two VMAs created with mmap(2). If there are enough threads available to the QEMU process, the number of coroutines may cause the total number of mappings to exceed `vm.max_map_count`, causing the QEMU process to fail to allocate memory and crash. From upstream, this bug occurs because "per-thread pools can grow to tens of thousands of coroutines. Each coroutine causes 2 virtual memory areas to be created. Eventually vm.max_map_count is reached and memory- related syscalls fail. The per-thread pool sizes are non-uniform and depend on past coroutine usage in each thread, so it's possible for one thread to have a large pool while another thread's pool is empty." The new approach "does not leave large numbers of coroutines pooled in a thread that may not use them again. In order to perform well it amortizes the cost of global pool accesses by working in batches of coroutines instead of individual coroutines." Upstream commit (coroutine: cap per-thread local pool size): https://gitlab.com/qemu-project/qemu/-/commit/86a637e4 Upstream commit (coroutine: reserve 5,000 mappings): https://gitlab.com/qemu-project/qemu/-/commit/9352f80c Upstream commit (util/coroutine: fix -Werror=maybe-uninitialized false-positive): https://gitlab.com/qemu-project/qemu/-/commit/25bc7d Additionally, the following bugs were identified in 86a637e4 which require follow-ups for the patch to work as intended: Missing AppArmor rule: https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 Incorrect string parsing: https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html ** Description changed: [ Impact ] QEMU 8.2.2's coroutine pool implementation can hit the Linux vm.max_map_count limit (`65530` in Jammy, `1048576` in Noble), causing QEMU to abort with "failed to allocate memory for stack" or "failed to set up stack guard page" during coroutine creation. Each coroutine creates two anonymous mappings with mmap(2). When a coroutine terminates it is cached in a global "release pool"; once that is full (2 x max pool size), terminated coroutines are cached in a thread-local "alloc pool" (up to the max pool size, per thread). When the thread-local pool is empty, the global pool is moved to a thread- local pool. Pooled coroutines are only freed once both pools are full. There are 3 kinds of threads which run coroutines (have non-empty thread-local pools): the main thread, vCPU threads, and iothreads. In (unpatched) QEMU 8.2.2 the theoretical maximum of idle coroutines in each thread-local pool is given by: (see hw/block/virtio-blk.c:1638) pool_max_size = 64 + Σ (1 to i of virtio-blk devices) (num_queues_i × queue_size_i / 2) Defaults: queue_size_i = 256 (hw/block/virtio-blk.c:1722) num_queues_i = vCPUs (hw/virtio/virtio-pci.c:2466) The global pool is twice as large as `pool_max_size` (util/qemu- coroutine.c:102); since thread-local pools steal the global pool, they may have at most 2 × pool_max_size coroutines: theoretical maximum mappings = 2 × (vCPUs + iothreads + 2) × 2 × pool_max_size A VM with 32 vCPUs and two virtio-blk disks gets a theoretical maximum mappings of 1122816, higher than the Noble kernel default. In practice the number of coroutines that will actually be allocated is typically much lower than this, since the total number of allocated coroutines depends on concurrent IO forcing an increase in the size of the thread-local caches. Many small IOs all at once drain both pools, causing more coroutines to be allocated. Coroutine growth after VM start is hypothesized to behave logarithmically. This behavior was introduced in upstream 4c41c69e05fe28c (v7.0.0-rc0) and therefore only affects Ubuntu 24.04. The user who reported this experienced crashes while running Noble's QEMU in a container on a 5.15 kernel (with the old `vm.max_map_count=65530`). Even with a relatively modest VM configuration (36 vCPUs, 3 disks), we’ve observed numbers of cached VMAs in the 30,000 - 70,000 range. Crashes are expected to be significantly less likely with Noble's sysctl config since reaching the default vm.max_map_count would require a significant fraction of the total allocated coroutines to be active concurrently (or a 10x increase in the number of vCPUs). Crashes with - the expected Noble configuration have not been reported. + the default Noble configuration have not been reported. While crashes can be worked around by adjusting the sysctl, this is still a resource leak that can leave a large number of unused coroutines allocated but unusable. 86a637e4810 allows for better reuse of those cached idle coroutines (2,000 - 5,000 as opposed to the 30,000 - 70,000 cited above). [ Test Plan ] 1. Run the upstream iotests (in a virtual machine) to exercise coroutines (not all of these tests are run during package build): ```sh gbp pq switch sudo apt build-dep ./ sudo apt install seabios ipxe-qemu make -f debian/rules configure-qemu # Binaries needed for iotests ninja -C b/qemu modules make -f debian/rules build-x86-optionrom cp -a b/optionrom/*.bin /usr/share/seabios/* /usr/lib/ipxe/qemu/* b/qemu/qemu-bundle/usr/share/qemu/ make -C b/qemu/ check-block cd b/qemu/ ./tests/qemu-iotests/check -qcow2 # Expected failures 151 281 307 graph-changes-while-io ``` 2. Run the Ubuntu Server team's qemu-migration-test [1] 3. The following procedure reproduces mmap exhaustion: ``` lxc launch ubuntu:noble --vm -c limits.cpu=8 -c limits.memory=16GiB n0 ``` In the VM: ```sh sudo apt install libvirt-daemon-system virtinst wget https://cloud-images.ubuntu.com/daily/server/noble/current/noble-server-cloudimg-amd64.img sudo cp noble-server-cloudimg-amd64.img /var/lib/libvirt/images/testvm-root.qcow2 sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data2.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data3.raw 10G sudo qemu-img create -f raw /var/lib/libvirt/images/testvm-data4.raw 10G cat > user-data <<EOF #cloud-config password: ubuntu chpasswd: expire: false ssh_pwauth: true package_update: true packages: - fio EOF sudo virt-install \ -n testvm \ --os-variant=ubuntu24.04 \ --ram=8192 --vcpus=8 \ --iothreads 8 \ --import \ --disk path=/var/lib/libvirt/images/testvm-root.qcow2,bus=virtio,cache=writethrough \ --disk path=/var/lib/libvirt/images/testvm-data.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data2.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data3.raw,bus=virtio,cache=none,driver.io=io_uring \ --disk path=/var/lib/libvirt/images/testvm-data4.raw,bus=virtio,cache=none,driver.io=io_uring \ --graphics none \ --network network=default \ --cloud-init user-data=user-data # Ctrl+Shift+a+] sudo virsh shutdown testvm ``` Limit write throughput in the block layer so that writes allocate a coroutine but aren't able to complete right away: ```sh for disk in vdb vdc vdd vde; do sudo virsh blkdeviotune testvm "${disk}" --total-iops-sec 2000 --current done ``` Use `sudo virsh edit testvm` to pin each disk to its own iothread; otherwise all IO will occur on the main thread. Add `iothread='1'` to the `<driver>` section of each 'testvm-data' disk, incrementing the number for each disk. ```sh # Reduce the max map count by a factor of four to match the reduction in # used IOThreads in the test environment sudo sysctl -w vm.max_map_count=16384 sudo virsh start testvm sudo virsh console testvm ``` On the host, you can monitor the maps with: ```sh pid=$(pidof qemu-system-x86_64) while sleep 1; do sudo grep -c '' "/proc/${pid}/maps"; done ``` In the testvm: ```sh while true; do for dev in vdb vdc vdd vde; do sudo fio "--name=/dev/${dev}" \ --ioengine=io_uring \ --direct=1 \ --runtime=10 \ --time_based=1 \ --rw=randread \ --bs=4k \ --iodepth=128 \ --numjobs=32 \ --group_reporting \ "--filename=/dev/${dev}" done done ``` Patched behavior: /proc/pid/maps levels off and stays consistent through the fio run. (With these parameters we observed approximately 5376). Unpatched behavior: /proc/pid/maps increases with each fio run until the VM crashes with one of: - failed to allocate memory for stack: Cannot allocate memory - failed to set up stack guard page: Cannot allocate memory 4. Performance regression test (results should be provided both before & after): ```sh for bs in 4k 128k; do for iodepth in 1 8 16 32 64; do sudo fio --output-format=json \ --output=fio-$bs-$iodepth.json \ --ioengine=libaio \ --direct=1 \ --numjobs=$(nproc) \ --runtime=30 \ --ramp_time=5 \ --time_based=1 \ --rw=randread \ --bs=$bs \ --iodepth=$iodepth \ --name=job1 \ --filename=/dev/vdb done done ``` [1] https://code.launchpad.net/~ubuntu-server/ubuntu/+source/qemu- migration-test/+git/qemu-migration-test [ Where problems could occur ] The patches restructure the way coroutines are allocated and deallocated. Coroutines are used extensively in the QMP/HMP protocol implementation, block drivers and live migrations; this change is somewhat fundamental to QEMU's coroutine infrastructure. If the change were wrong/broken, it could cause: - failing block device drivers, causing failed boots or broken I/O operations within affected guests - incorrect/strange behavior when interacting with a QEMU process over QMP/HMP - failed/broken live migrations - performance regressions as the patch introduces a lock in the coroutine pool hot path (coroutine_pool_get) Upstream has two follow-ups to the refactor, 9352f80cd926fe2dde7c89b93ee33bb0356ff40e and 25bc7d16fa96b0ff881c83ed225ea380fe427c78. 9352f80cd will be included in the upload; 25bc7d16fa fixes a false-positive compiler warning. 86a637e4 parses the contents of `/proc/sys/vm/max_map_count` incorrectly [1]. This means that without Hector's linked follow-up, the patch removes any limitation on the maximum size of the coroutine idle pool. The introduction of a read() from procfs will introduce an apparmor denial for VMs using libvirt's AppArmor security driver [2]. If AppArmor is active for a VM, this change will remove any limitation of the maximum size of the coroutine idle pool. If [1] and [2] are also applied, workloads which require many coroutines may experience a performance regression (as the patches set a hard limit on the size of the coroutine pool, so coroutines needed above the pool size are allocated/deallocated on demand). The coroutine pool size is a function of `vm.max_map_count`, so increasing the max map count is the recommended workaround: ``` sysctl -w vm.max_map_count=1048576 ``` It's likely that workloads running into this performance regression would also have been affected by the original bug. [1] https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html [2] https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 [ Other Info ] QEMU's coroutines are a userspace threading implementation. They are used frequently in QEMU's virtual disk drivers. Noble's QEMU 8.2.2 keeps a thread-local list of coroutines `alloc_pool` (util/qemu-coroutine.c:40); this list can grow to thousands or even tens of thousands of coroutines _for each thread_. Each coroutine has two VMAs created with mmap(2). If there are enough threads available to the QEMU process, the number of coroutines may cause the total number of mappings to exceed `vm.max_map_count`, causing the QEMU process to fail to allocate memory and crash. From upstream, this bug occurs because "per-thread pools can grow to tens of thousands of coroutines. Each coroutine causes 2 virtual memory areas to be created. Eventually vm.max_map_count is reached and memory- related syscalls fail. The per-thread pool sizes are non-uniform and depend on past coroutine usage in each thread, so it's possible for one thread to have a large pool while another thread's pool is empty." The new approach "does not leave large numbers of coroutines pooled in a thread that may not use them again. In order to perform well it amortizes the cost of global pool accesses by working in batches of coroutines instead of individual coroutines." Upstream commit (coroutine: cap per-thread local pool size): https://gitlab.com/qemu-project/qemu/-/commit/86a637e4 Upstream commit (coroutine: reserve 5,000 mappings): https://gitlab.com/qemu-project/qemu/-/commit/9352f80c Upstream commit (util/coroutine: fix -Werror=maybe-uninitialized false-positive): https://gitlab.com/qemu-project/qemu/-/commit/25bc7d Additionally, the following bugs were identified in 86a637e4 which require follow-ups for the patch to work as intended: Missing AppArmor rule: https://gitlab.com/libvirt/libvirt/-/commit/85e07fb1ceee7943879f8a374cabfa8ab858a3c6 Incorrect string parsing: https://lists.gnu.org/archive/html/qemu-devel/2026-10/msg01192.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2168803 Title: QEMU 8.2.2 guests crash by exhausting vm.max_map_count with coroutines To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2168803/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
