Hello, I've attached a debdiff for noble-security (lp2172882_noble- security_on-1.20.debdiff).
- Based on 1:8.2.2+ds-0ubuntu1.20 (noble-proposed), version 1:8.2.2+ds-0ubuntu1.21, as the earlier qemu security uploads were built on top of the SRU in -proposed. 1.20 carries four SRU fixes that are still verification-needed-noble (LP: #2156307, #2158673, #2166503, #2168803). Thanks. ** Patch added: "lp2172882_noble-security_on-1.20.debdiff" https://bugs.launchpad.net/bugs/2172882/+attachment/6019602/+files/lp2172882_noble-security_on-1.20.debdiff ** Description changed: + [Impact] + + On Noble, QEMU aborts when an NBD client disconnects from an export whose disk runs in an IOThread, and the guest is killed. + This happens on every libvirt pull-mode backup (virsh backup-begin) of a domain that uses IOThreads on virtio-blk or virtio-scsi: + + qemu-system-x86_64: blockdev-nbd.c:62: nbd_blockdev_client_closed: + Assertion `qemu_in_main_thread() && nbd_server' failed. + + It is a regression from the CVE-2024-7409 fix in 1:8.2.2+ds-0ubuntu1.10 (USN-7744-1). + d/p/CVE-2024-7409-3.patch assumes the NBD client close callback runs in the main loop, but in 8.2.2 nbd_trip() calls it from the export's IOThread. + The fix backports the two upstream commits that move it to the main loop. + + [Test Plan] + + On a Noble host with KVM (no guest OS needed): + + sudo apt install -y qemu-system-x86 qemu-utils libvirt-daemon-system + sudo qemu-img create -f qcow2 /var/lib/libvirt/images/lp2172882.qcow2 1G + sudo qemu-io -f qcow2 -c 'write -P 0xab 0 64M' /var/lib/libvirt/images/lp2172882.qcow2 + echo "<domain type='kvm'><name>lp2172882</name><memory unit='MiB'>256</memory><vcpu>1</vcpu><iothreads>1</iothreads><os><type arch='x86_64' machine='q35'>hvm</type></os><devices><disk type='file' device='disk'><driver name='qemu' type='qcow2' iothread='1'/><source file='/var/lib/libvirt/images/lp2172882.qcow2'/><target dev='vda' bus='virtio'/></disk></devices></domain>" > lp2172882.xml + sudo virsh define lp2172882.xml + sudo virsh start lp2172882 + # AppArmor only lets QEMU create the socket in its per-domain directory + SOCK=/var/lib/libvirt/qemu/domain-$(sudo virsh domid lp2172882)-lp2172882/backup.sock + echo "<domainbackup mode='pull'><server transport='unix' socket='$SOCK'/><disks><disk name='vda' backup='yes' type='file'><scratch file='/var/lib/libvirt/images/lp2172882-scratch.qcow2'/></disk></disks></domainbackup>" > backup.xml + sudo virsh backup-begin lp2172882 backup.xml + sudo qemu-img convert -f raw -O qcow2 "nbd+unix:///vda?socket=$SOCK" /tmp/lp2172882-out.qcow2 + sleep 2 + sudo virsh domstate lp2172882 + sudo tail -2 /var/log/libvirt/qemu/lp2172882.log + + Before the fix, the domain is "shut off" and the log shows the assertion above with "shutting down, reason=crashed". + After the fix, virsh domstate must print "running", the log must have no assertion, and "sudo virsh domjobabort lp2172882" must succeed with the domain still running. + + With the fix, ending the backup job while a client is still connected must also keep the domain running. + This is the CVE-2024-7409 path where nbd-server-stop closes the remaining clients: + + sudo virsh backup-begin lp2172882 backup.xml + sudo qemu-io -r -f raw -c 'sleep 8000' "nbd+unix:///vda?socket=$SOCK" & + sleep 2 + sudo virsh domjobabort lp2172882 + sudo virsh domstate lp2172882 + + [Where problems could occur] + + A regression could show up as an abort on the main-thread assert the fix adds to nbd_client_put() if another caller still runs in an IOThread, or as nbd-server-stop or block-export-del waiting if a disconnecting client is not released. + Upstream iotests 281 and 307 (IOThread exports) fail with the current package and pass with the fix, and the -nbd iotests pass. + + [Original Description] + Since the CVE-2024-7409 security update (1:8.2.2+ds-0ubuntu1.10), QEMU on Noble aborts when an NBD client disconnects from an export whose block node runs in an IOThread. This is the normal flow of a libvirt pull-mode backup (virsh backup-begin / virDomainBackupBegin) of any VM that uses IOThreads (<iothreads> with <driver iothread=...> on virtio- blk, or <controller type='scsi' model='virtio-scsi'><driver iothread=.../>). The guest is killed: qemu-system-x86_64: blockdev-nbd.c:62: nbd_blockdev_client_closed: Assertion `qemu_in_main_thread() && nbd_server' failed. ... shutting down, reason=crashed Backtrace of the aborting thread (stock 1:8.2.2+ds-0ubuntu1.18, symbols from debuginfod; this is an IOThread, not the main thread: the main thread is LWP 7356 = PID; full backtrace attached): #5 __assert_fail_base (assertion="qemu_in_main_thread() && nbd_server", file="blockdev-nbd.c", line=62, function="nbd_blockdev_client_closed") #7 nbd_blockdev_client_closed (client=...) at blockdev-nbd.c:62 #8 client_close (negotiated=true, client=...) at nbd/server.c:1550 #10 nbd_trip (opaque=...) at nbd/server.c:3029 #11 coroutine_trampoline () at util/coroutine-ucontext.c:175 [Cause] debian/patches/CVE-2024-7409-3.patch (upstream 3e7ef738c8, "nbd/server: CVE-2024-7409: Close stray clients at server-stop") adds assert(qemu_in_main_thread() && nbd_server) to nbd_blockdev_client_closed(). Its commit message states that it "relies heavily on the fact that nbd/server.c guarantees that it only calls nbd_blockdev_client_closed() from the main loop". That guarantee was only introduced upstream in the QEMU 9.0 cycle (f816310d0c32 "nbd/server: only traverse NBDExport->clients from main loop thread", together with 7075d235114b, efade66d5872 and 9c707525cbb1). Noble's 8.2.2 does not have it. In 8.2.2, nbd_trip() calls client_close() from the export's AioContext, so for an IOThread- bound export the close callback runs on the IOThread and the new assertion fails. Taking the upstream stable-8.2 versions of those commits (aee1039c, 13fc21ae, a69a002d, as in qemu v8.2.3+) does not help. Both a Noble build with them added and unmodified upstream v8.2.7 abort instead in nbd_client_put(): Assertion `qemu_in_main_thread()' failed, and the client gets block-status EIO. Unmodified upstream 11.1.3 is not affected. Affected: 1:8.2.2+ds-0ubuntu1.10 through 1:8.2.2+ds-0ubuntu1.16 (noble- security), 1.18 (noble-updates) and 1.19 (noble-proposed); none of their changelogs touch NBD. Not affected: 1.9 and earlier, which are vulnerable to CVE-2024-7409. Reproduced on: - a clean Ubuntu 24.04.5 VM: qemu-system-x86 1:8.2.2+ds-0ubuntu1.18, libvirt-daemon-system 10.0.0-2ubuntu8.19, kernel 6.8.0-147-generic; - a 24.04 host with qemu 1:8.2.2+ds-0ubuntu1.16 and libvirt 10.0.0-2ubuntu8.13. Both reproduce with virtio-blk + IOThread, virtio-scsi + IOThread, and a two-disk domain (crash when the first disk's client disconnects while the second export is still open). [Impact] - Any Noble host that takes NBD backups of running VMs whose disks use IOThreads loses the VM on every backup. That covers libvirt pull-mode backup, and therefore backup products built on it. - IOThreads are a common performance configuration for virtio-blk and virtio-scsi, so users must currently choose between IOThreads and being able to back up the VM. - The crash is a regression introduced by a security update. [Test Plan] The attached nbd_iothread_repro.sh automates all of this (run as root on a Noble libvirt host; no guest OS needed): bash nbd_iothread_repro.sh setup bash nbd_iothread_repro.sh start blk # virtio-blk disk with <driver iothread='1'/> bash nbd_iothread_repro.sh repro blk # pull backup, qemu-img reads the export and disconnects bash nbd_iothread_repro.sh start scsi; bash nbd_iothread_repro.sh repro scsi bash nbd_iothread_repro.sh start multi; bash nbd_iothread_repro.sh repro multi Manual steps: 1. Create a disk: qemu-img create -f qcow2 disk.qcow2 2G; qemu-io -f qcow2 -c 'write -P 0x5a 0 256M' disk.qcow2 2. Start a transient domain with <iothreads>1</iothreads> and the disk on virtio with <driver name='qemu' type='qcow2' cache='none' io='native' iothread='1'/>. 3. Begin a pull backup: <domainbackup mode='pull'><server transport='unix' socket='SOCK'/><disks><disk name='vda' backup='yes' type='file'><scratch file='SCRATCH'/></disk></disks></domainbackup> virsh backup-begin DOM backup.xml Under the default AppArmor profile, put SOCK and SCRATCH in the domain's /var/lib/libvirt/qemu/domain-<id>-<name>/ directory. Paths such as /var/tmp get EACCES at nbd-server-start; the script does this. 4. Read the export and disconnect: qemu-img convert -f raw -O qcow2 'nbd+unix:///vda?socket=SOCK' out.qcow2 5. Expected: the domain keeps running, and virsh domjobabort DOM ends the job. Actual: the domain is gone, and /var/log/libvirt/qemu/DOM.log shows the assertion and reason=crashed. Regression checks for the CVE fix, with the fixed package: - "bash nbd_iothread_repro.sh conns blk 150" runs 150 connect/disconnect cycles within one job. All must succeed, and the final read must work (no leak of the connection counter against the default cap of 100). - "bash nbd_iothread_repro.sh stray blk" opens a client that never negotiates, then ends the job. The stray client must be closed promptly and the domain must keep running. - "bash nbd_iothread_repro.sh repeat blk|scsi|multi 50" runs repeated full backup cycles. Results with the attached patch applied to 1:8.2.2+ds-0ubuntu1.16: - read-and-disconnect passes on virtio-blk, virtio-scsi and the two-disk domain; - repeat: 150/150 cycles; - conns: 150 connect/disconnect cycles, no failures; - stray: client closed in 0.01 s, domain still running. [Where problems could occur] - The patch changes only blockdev-nbd.c:nbd_blockdev_client_closed(), the close callback for NBD servers started with nbd-server-start (QMP/HMP). - The connection-list and counter bookkeeping added by the CVE fix still runs only in the main loop. It runs directly when the client closes on the main thread, and otherwise through a one-shot bottom half on the main AioContext. - nbd_server_free() already waits with AIO_WAIT_WHILE_UNLOCKED() for the counter to reach zero, so pending bottom halves run before the server is freed. - The nbd_client_put() reference drop stays on the thread that closed the client. That is exactly the behaviour of 1:8.2.2+ds-0ubuntu1.9 and of upstream 8.2.2. - qemu-nbd uses its own close callback and is unchanged. - If something went wrong, the risks would be a regression in the CVE-2024-7409 stray-client handling (for example nbd-server-stop hanging if the counter were not decremented), or exhaustion of the connection limit. The conns and stray checks above cover both. [Other Info] - The patch is Noble-specific. Upstream master already guarantees main-loop close through the 9.0 rework, which is too large to backport. The upstream stable-8.2 line has the related failure described above. - Debugging note: apport ignores crashes in libvirt's private mount namespace ("crashed in a separate mount namespace, ignoring"). To capture the .crash, set namespaces = [ ] and max_core = "unlimited" in /etc/libvirt/qemu.conf on the test host. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2172882 Title: qemu 1:8.2.2+ds-0ubuntu1.10 and later aborts in nbd_blockdev_client_closed() when an NBD client of an IOThread-bound export disconnects (CVE-2024-7409 backport assumes main-loop close) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2172882/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
