** Description changed: [ Impact ] Users running VMs with `virtiofs` mounts where there is a page-size mismatch between the host and guest kernels (e.g., a 4K page-size host and a 64K page-size guest) experience "Cannot allocate memory" (`ENOMEM`) errors when attempting to read directories. This occurs because `virtiofsd` computes `max_pages` based on the host's page size, but the guest converts `max_pages` back to bytes using its own (larger) page size. As a result, the guest sends a `READDIR` request that exceeds `virtiofsd`'s `MAX_BUFFER_SIZE`, causing the daemon to reject the request. The fix resolves this by capping the amount of directory data generated locally to `MAX_BUFFER_SIZE` instead of rejecting the oversized request, as `READDIR` is permitted to return fewer bytes than requested. [ Test Plan ] # Set up repos and download packages 0. Grab a fresh arm64 machine 1. add deb-src to ubuntu.sources 2. sudo apt build-dep linux 3. sudo apt install virtme-ng qemu-system-arm libncurses-dev 4. sudo apt install virtiofsd # the package in question # 24.04 EXTRA SETUP (NOT NECESSARY FOR 26.04+) 4.1. sudo apt install python3-pip flake8 pylint cargo rustc 4.2. sudo apt remove virtme-ng # gone too soon :( 4.3. pip install --break-system-packages uv # now it's getting spicy 4.4. git clone https://github.com/arighi/virtme-ng.git && cd virtme-ng 4.5. BUILD_VIRTME_NG_INIT=1 uv tool install . # Set up vng and build kernel 5. sudo usermod -aG kvm ubuntu && newgrp kvm # replace `ubuntu` with your user 6. git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ && cd linux 7. wait 10000 years 8. set up kernel config: a. vng --kconfig b. make menuconfig c. kernel features -> page size -> set to 64k -> save -> exit 9. vng --build # Run the reproducer (from upstream merge request) 10. getconf PAGE_SIZE # should print 4096 on host 11. vng --run arch/arm64/boot/Image --memory 16G --cpus 4 --overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"' 12. There should be no errors in the output of 11. # Reproducer output 12a. Unpatched package output (fail): ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4 --overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"' 7.1.0-rc7-virtme 65536 ls: general io error: Cannot allocate memory (os error 12) ls rc=1 Traceback (most recent call last): File "<string>", line 1, in <module> import os; print(len(os.listdir("/tmp"))) ~~~~~~~~~~^^^^^^^^ OSError: [Errno 12] Cannot allocate memory: '/tmp' 12b. Patched package output (success): ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4 --overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"' 7.1.0-rc7-virtme 65536 hsperfdata_root snap-private-tmp systemd-private-119639c5e17c430ca33b233f937d2924-ModemManager.service-EmreAg systemd-private-119639c5e17c430ca33b233f937d2924-chrony.service-DJCwIj systemd-private-119639c5e17c430ca33b233f937d2924-fwupd.service-DSIi3M systemd-private-119639c5e17c430ca33b233f937d2924-polkit.service-w28tHD systemd-private-119639c5e17c430ca33b233f937d2924-systemd-logind.service-bWi2S2 virtme_retvjz9uo9n ls rc=0 12 [ Where problems could occur ] - This patch modifies the returned value of `READDIR`. If other programs - incorrectly assume `READDIR` will never give short reads and do not handle - the short read case correctly, this patch could lead to data corruption. - However, this hypothetical depends on OTHER privileged programs behaving - incorrectly, which is not the responsibility of virtiofsd. - This patch strictly improves code correctness, so problems are unlikely. + This patch modifies the returned value of `READDIR`. This patch only + intends to change behavior on architectures with variable page sizes; + ARM64 is the given example here. Architectures with fixed page sizes + like AMD64 should not be affected (hugepages notwithstanding). + + In the case of the same page size between the guest and the host, + `READDIR` will not run into the case where it is too large and needs to + be clamped, as the conversion of `max_pages` to bytes yields the same + result in the guest and the host. For example, 4KiB AMD64 hosts running + 4KiB AMD64 guests will not run into this issue, and ideally will not be + affected by this patch. + + In the case of the guest having a lesser page size than the host, + `READDIR` will not need to be clamped, either, as the conversion of + `max_pages` back to bytes yields a greater number in the host than in + the guest. For example, a 16KiB ARM64 host running a 4KiB ARM64 guest. + This case would ideally be unaffected by this patch, as well. + + This patch should only affect the third case, where the guest has a + greater page size than the host. `max_pages` will yield a greater number + of bytes in the guest than in the host, causing the need to clamp the + value of 'READDIR' lest the directory reading fail with 'Cannot allocate + memory'. + + As such, regressions are most likely limited to the case where the guest + has a larger page size than the host, but all three cases carry the same + inherent regression risk of modifying the returned value of a directory + reading: data corruption, truncation, etc. + + Considering it's the guest reading data from the host, this risk should + limit the data corruption to the guest itself (unless the guest is + allowed to write back to the host). + + If a regression is discovered, a suitable workaround is to remove + virtiofsd, which will fall back to the 9p driver on virtme-ng and + libvirt. [ Other Info ] The whole rigmarole with building virtme-ng on 24.04 is necessary because the version available in the Noble repos (1.22) is too old to support virtiofs mounts in the first place. 1.22 uses 9p exclusively on ARM64; virtiofs support was added in 1.30 for ARM64. Additionally, we can't install virtme-ng via pip directly because the provided `virtme-ng-init` initializer is compiled for AMD64, not ARM64. Building vng from source was the most reproducible way I found to test this package. Finally, for Noble, a newer kernel may be required. See below as to why. `sudo apt install linux-image-7.0.0-14-generic --install-suggests` Extra info from upstream MR: --- The virtiofsd-side failure is easiest to reproduce after applying the kernel patch that backs uncached readdir output with pages: https://lore.kernel.org/all/[email protected]/ Without that kernel patch, the guest may fail earlier in the kernel before the oversized READDIR request reaches virtiofsd. ... (omitted for brevity) ... The page-size mismatch is what exposes the issue. virtiofsd computes max_pages from the host page size, while the guest converts max_pages back to bytes using the guest page size. With a 4K host and 64K guest, the guest can send a READDIR size larger than virtiofsd's MAX_BUFFER_SIZE. READDIR can return less than requested, so this patch caps the amount of directory data generated locally instead of rejecting the request. --- Target Releases: Ubuntu 24.04, 26.04, and 26.10. Upstream Commit: d24cda8a325d server: do not reject oversized readdir requests Upstream Link: https://gitlab.com/virtio-fs/virtiofsd/-/commit/d24cda8a325d2a9ae1adc5acda57515ed2e8e1d2 Upstream Merge Request: https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/316
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2155048 Title: [SRU] Backport request - server: do not reject oversized readdir requests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rust-virtiofsd/+bug/2155048/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
