Public bug reported: [Impact]
QEMU can hit an assertion and crash when attempting to write to a virtual disk image when the following conditions are met: 1. disk type is "raw" 2. disk cache type set to "none" 3. disk is shared over NFS 4. disk size is not a multiple of 4 kiB In this case, QEMU assumes that the image needs to be aligned to 4kib, and that writing to the disk which is not a multiple of 4kib will lead to writing past the end of the disk image, and will hit the following assert: qemu-system-x86_64: /build/qemu-AB62EU/qemu-4.2/block/io.c:1885: bdrv_co_write_req_prepare: Assertion `end_sector <= bs->total_sectors || child->perm & BLK_PERM_RESIZE' failed. Aborted (core dumped) This is particularly prevalent if you have a disk of size 10000000000 on the NFS server. You can work-around the problem by making the disk image a multiple of 4kib, in order to not hit the assert. [Test case] This bug is straightforward to reproduce on Focal and Groovy. Start with a fresh install of Ubuntu Server, install KVM stack: $ sudo apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils $ sudo reboot Next, install and configure a NFS server: $ sudo -s $ sudo apt install nfs-kernel-server $ mkdir -p /export $ chown libvirt-qemu:kvm /export $ echo "/export 127.0.0.1(rw,sync,no_subtree_check)" >> /etc/exports $ systemctl restart nfs-server Create a disk image: $ truncate -s 10000000000 /export/reproducer-centos.img $ chown libvirt-qemu:kvm /export/reproducer-centos.img $ chmod 666 /export/reproducer-centos.img Mount the NFS server to /mnt: $ mount 127.0.0.1:/export /mnt -o bg,noacl,noatime,nolock,proto=udp,vers=3,noauto (for Groovy / 5.8 kernel, drop the proto=udp option) Download the CentOS image: $ wget https://vault.centos.org/7.2.1511/isos/x86_64/CentOS-7-x86_64-Minimal-1511.iso Start the VM: $ qemu-system-x86_64 -cdrom ./CentOS-7-x86_64-Minimal-1511.iso -m 1024 -blockdev '{"driver":"file","filename":"/mnt/reproducer-centos.img ","node-name":"disk0","cache":{"direct":true}}' -device virtio-blk- pci,drive=disk0 -vnc 0.0.0.0:0 -enable-kvm Connect to the VM with Reminna, configured for VNC connection to the server on <ip address>:5900 Click Continue at the language/keyboard selection screen. Click Installation Destination. Click Done (no changes needed on that screen). Click Begin Installation. It will crash after displaying "Setting up the installation environment" (which is the second thing printed) or the message about creating the disk label. If it gets any farther than that (i.e. starts creating filesystems), it's going to work and you can stop the test. This is the error I see: qemu-system-x86_64: /build/qemu-AB62EU/qemu-4.2/block/io.c:1885: bdrv_co_write_req_prepare: Assertion `end_sector <= bs->total_sectors || child->perm & BLK_PERM_RESIZE' failed. Aborted (core dumped) Test packages are available for Focal and Groovy in the following PPA: https://launchpad.net/~mruffell/+archive/ubuntu/sf298252-test When you use the QEMU packages from this PPA, the issue is fixed and the CentOS installation works properly. [Where problems could occur] There are two places where problems could occur. The first, is around the handling of byte aligned disk images, in the block subsystem of QEMU.A new check is added to see if we have the write permission, and not the resize permission, and if that is the case, then we throw an error. This error is more graceful than hitting an assert, but it introduces error handling and other risks for regression. The second, is that NFS is explicitly cleared for byte aligned writes, and this is enforced by testing for the NFS magic bytes on the start of the filesystem the disk image is being loaded from. By checking magic bytes, we ensure no other filesystem type could get mixed up and allow byte aligned writes when they would not be supported, which reduces the risk of regression. If a regression were to occur, it would likely only affect users with non 4kib aligned disk images, and a workaround would be to resize the virtual disk image to 4kib alignment, or create new VMs with disk images as a multiple of 4kib. [Other] The commits which fix the problem landed in QEMU 5.1, and are: commit 9c60a5d1978e6dcf85c0e01b50e6f7f54ca09104 From: Kevin Wolf <[email protected]> Date: Thu, 16 Jul 2020 16:26:00 +0200 Subject: block: Require aligned image size to avoid assertion failure Link: https://git.qemu.org/?p=qemu.git;a=commit;h=9c60a5d1978e6dcf85c0e01b50e6f7f54ca09104 commit 5edc85571e7b7269dce408735eba7507f18ac666 From: Kevin Wolf <[email protected]> Date: Thu, 16 Jul 2020 16:26:01 +0200 Subject: file-posix: Allow byte-aligned O_DIRECT with NFS Link: https://git.qemu.org/?p=qemu.git;a=commit;h=5edc85571e7b7269dce408735eba7507f18ac666 Mailing list discussion: https://www.mail-archive.com/[email protected]/msg721982.html ** Affects: qemu (Ubuntu) Importance: Undecided Status: Fix Released ** Affects: qemu (Ubuntu Focal) Importance: Medium Assignee: Matthew Ruffell (mruffell) Status: In Progress ** Affects: qemu (Ubuntu Groovy) Importance: Medium Assignee: Matthew Ruffell (mruffell) Status: In Progress ** Tags: focal groovy sts ** Also affects: qemu (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: qemu (Ubuntu Groovy) Importance: Undecided Status: New ** Changed in: qemu (Ubuntu) Status: New => Fix Released ** Changed in: qemu (Ubuntu Focal) Status: New => In Progress ** Changed in: qemu (Ubuntu Groovy) Status: New => In Progress ** Changed in: qemu (Ubuntu Focal) Importance: Undecided => Medium ** Changed in: qemu (Ubuntu Groovy) Importance: Undecided => Medium ** Changed in: qemu (Ubuntu Focal) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Changed in: qemu (Ubuntu Groovy) Assignee: (unassigned) => Matthew Ruffell (mruffell) ** Tags added: focal groovy sts -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1921665 Title: QEMU hits assertion when virtual disk is stored on NFS server and is not 4 kib byte aligned To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1921665/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
