Debdiff for 24.04 ** Patch added: "lp2155048-noble-4.debdiff" https://bugs.launchpad.net/ubuntu/+source/rust-virtiofsd/+bug/2155048/+attachment/5980732/+files/lp2155048-noble-4.debdiff
** Description changed: [ Impact ] Users running VMs with `virtiofs` mounts where there is a page-size mismatch between the host and guest kernels (e.g., a 4K page-size host and a 64K page-size guest) experience "Cannot allocate memory" (`ENOMEM`) errors when attempting to read directories. This occurs because `virtiofsd` computes `max_pages` based on the host's page size, but the guest converts `max_pages` back to bytes using its own (larger) page size. As a result, the guest sends a `READDIR` request that exceeds `virtiofsd`'s `MAX_BUFFER_SIZE`, causing the daemon to reject the request. The fix resolves this by capping the amount of directory data generated locally to `MAX_BUFFER_SIZE` instead of rejecting the oversized request, as `READDIR` is permitted to return fewer bytes than requested. [ Test Plan ] # Set up repos and download packages 0. Grab a fresh arm64 machine 1. add deb-src to ubuntu.sources 2. sudo apt build-dep linux 3. sudo apt install virtme-ng qemu-system-arm libncurses-dev 4. sudo apt install virtiofsd # the package in question # 24.04 EXTRA SETUP (NOT NECESSARY FOR 26.04+) 4.1. sudo apt install python3-pip flake8 pylint cargo rustc 4.2. sudo apt remove virtme-ng # gone too soon :( 4.3. pip install --break-system-packages uv # now it's getting spicy 4.4. git clone https://github.com/arighi/virtme-ng.git && cd virtme-ng 4.5. BUILD_VIRTME_NG_INIT=1 uv tool install . # Set up vng and build kernel 5. sudo usermod -aG kvm ubuntu && newgrp kvm # replace `ubuntu` with your user 6. git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ && cd linux 7. wait 10000 years 8. set up kernel config: a. vng --kconfig b. make menuconfig c. kernel features -> page size -> set to 64k -> save -> exit 9. vng --build # Run the reproducer (from upstream merge request) 10. getconf PAGE_SIZE # should print 4096 on host 11. vng --run arch/arm64/boot/Image --memory 16G --cpus 4 --overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"' 12. There should be no errors in the output of 11. # Reproducer output 12a. Unpatched package output (fail): ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4 --overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"' 7.1.0-rc7-virtme 65536 ls: general io error: Cannot allocate memory (os error 12) ls rc=1 Traceback (most recent call last): File "<string>", line 1, in <module> import os; print(len(os.listdir("/tmp"))) ~~~~~~~~~~^^^^^^^^ OSError: [Errno 12] Cannot allocate memory: '/tmp' 12b. Patched package output (success): ubuntu@kamek:~/linux$ vng --run arch/arm64/boot/Image --memory 16G --cpus 4 --overlay-rwdir /tmp --exec 'uname -r; getconf PAGE_SIZE; ls /tmp; echo "ls rc=$?"; python3 -c "import os; print(len(os.listdir(\"/tmp\")))"' 7.1.0-rc7-virtme 65536 hsperfdata_root snap-private-tmp systemd-private-119639c5e17c430ca33b233f937d2924-ModemManager.service-EmreAg systemd-private-119639c5e17c430ca33b233f937d2924-chrony.service-DJCwIj systemd-private-119639c5e17c430ca33b233f937d2924-fwupd.service-DSIi3M systemd-private-119639c5e17c430ca33b233f937d2924-polkit.service-w28tHD systemd-private-119639c5e17c430ca33b233f937d2924-systemd-logind.service-bWi2S2 virtme_retvjz9uo9n ls rc=0 12 [ Where problems could occur ] This patch modifies the returned value of `READDIR`. This patch only intends to change behavior on architectures with variable page sizes; ARM64 is the given example here. Architectures with fixed page sizes like AMD64 should not be affected (hugepages notwithstanding). In the case of the same page size between the guest and the host, `READDIR` will not run into the case where it is too large and needs to be clamped, as the conversion of `max_pages` to bytes yields the same result in the guest and the host. For example, 4KiB AMD64 hosts running 4KiB AMD64 guests will not run into this issue, and ideally will not be affected by this patch. In the case of the guest having a lesser page size than the host, `READDIR` will not need to be clamped, either, as the conversion of `max_pages` back to bytes yields a greater number in the host than in the guest. For example, a 16KiB ARM64 host running a 4KiB ARM64 guest. This case would ideally be unaffected by this patch, as well. This patch should only affect the third case, where the guest has a greater page size than the host. `max_pages` will yield a greater number of bytes in the guest than in the host, causing the need to clamp the value of 'READDIR' lest the directory reading fail with 'Cannot allocate memory'. As such, regressions are most likely limited to the case where the guest has a larger page size than the host, but all three cases carry the same inherent regression risk of modifying the returned value of a directory reading: data corruption, truncation, etc. Considering it's the guest reading data from the host, this risk should limit the data corruption to the guest itself (unless the guest is allowed to write back to the host). If a regression is discovered, a suitable workaround is to remove virtiofsd, which will fall back to the 9p driver on virtme-ng and libvirt. [ Other Info ] The whole rigmarole with building virtme-ng on 24.04 is necessary because the version available in the Noble repos (1.22) is too old to support virtiofs mounts in the first place. 1.22 uses 9p exclusively on ARM64; virtiofs support was added in 1.30 for ARM64. Additionally, we can't install virtme-ng via pip directly because the provided `virtme-ng-init` initializer is compiled for AMD64, not ARM64. Building vng from source was the most reproducible way I found to test this package. Finally, for Noble, a newer kernel may be required. See below as to why. `sudo apt install linux-image-7.0.0-14-generic --install-suggests` Extra info from upstream MR: --- The virtiofsd-side failure is easiest to reproduce after applying the kernel patch that backs uncached readdir output with pages: https://lore.kernel.org/all/[email protected]/ Without that kernel patch, the guest may fail earlier in the kernel before the oversized READDIR request reaches virtiofsd. ... (omitted for brevity) ... The page-size mismatch is what exposes the issue. virtiofsd computes max_pages from the host page size, while the guest converts max_pages back to bytes using the guest page size. With a 4K host and 64K guest, the guest can send a READDIR size larger than virtiofsd's MAX_BUFFER_SIZE. READDIR can return less than requested, so this patch caps the amount of directory data generated locally instead of rejecting the request. --- - Target Releases: Ubuntu 24.04, 26.04, and 26.10. + Target Releases: Ubuntu 24.04 and 26.04 Upstream Commit: d24cda8a325d server: do not reject oversized readdir requests Upstream Link: https://gitlab.com/virtio-fs/virtiofsd/-/commit/d24cda8a325d2a9ae1adc5acda57515ed2e8e1d2 Upstream Merge Request: https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/316 + Upstream Release: https://gitlab.com/virtio-fs/virtiofsd/-/releases/v1.14.0 + Debian Tracker: https://tracker.debian.org/pkg/rust-virtiofsd + Ubuntu Tracker: https://launchpad.net/ubuntu/+source/rust-virtiofsd -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2155048 Title: [SRU] Backport request - server: do not reject oversized readdir requests To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/rust-virtiofsd/+bug/2155048/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
