Public bug reported:
[ Impact ]
QEMU's userspace virtio-net RX path can permanently stall in an affected
guest network device. The guest and QEMU process remain alive, and
guest-to-host traffic can still be observed after bypassing ARP with a
static neighbour entry, but host-to-guest delivery stops until the VM is
restarted.
The issue is a race in virtio_net_has_buffers(). The old code enables RX
queue notification and then re-checks available buffers, but that re-
check reused a stale shadow copy of the available index instead of re-
reading the guest's available index. With event-index notification
suppression, the guest can add buffers in the small window between
enabling notification and the re-check without sending another kick, and
because the re-check never refreshes that shadow copy it misses those
new buffers. QEMU then waits for a kick that will not arrive, leaving
the host-to-guest RX path wedged.
This affects workloads using QEMU userspace virtio-net, especially under
sustained host-to-guest traffic and configurations that make RX buffer
exhaustion more frequent.
[ Test Plan ]
The test must use QEMU userspace virtio-net. Do not use vhost-net for
this test, because vhost-net moves the data path out of the QEMU
userspace code being fixed.
How to run this plan:
- Run steps 1-6 twice: once on the affected qemu (expect step 7) and once on
the patched qemu carrying the fix (expect step 8). Concrete bridge/tap, build
and run commands used here are in the detailed sections below.
- The stall is a race, so it is not deterministic. A single clean run does
not prove the fix. On the affected qemu, repeat the run (and if needed raise -P
/ -t or lower the ring) until the stall is observed; on the patched qemu,
repeat the same condition that reliably reproduced on the affected qemu several
times and confirm it never stalls.
1. Prepare a KVM host and one Ubuntu guest connected through a host
bridge and tap device.
Host:
bridge: 10.77.0.1/24
tap: attached to the bridge
Guest:
static IP: 10.77.0.2/24
kernel command line includes: swiotlb=force
(with iommu_platform=on this forces the guest onto the bounce-buffer DMA
path, which makes RX buffer exhaustion and the notification race more frequent)
QEMU NIC:
virtio-net-pci-non-transitional
iommu_platform=on
mrg_rxbuf=on
rx_queue_size=256
2. Boot the guest with the qemu-system-x86_64 binary under test.
In the guest, reduce the virtio-net ring when the QEMU version
supports it:
sudo ethtool -G GUEST_IFACE rx 64 tx 64 || true
Record the actual ring size:
ethtool -g GUEST_IFACE
3. In the guest, start two iperf3 servers:
iperf3 -s -p 8001
iperf3 -s -p 8002
4. From the host, start simultaneous host-to-guest traffic to both
ports.
Short run used for Noble, where RX/TX could be reduced to 64:
iperf3 -c 10.77.0.2 -p 8001 -P 20 -w 256k -t 120 -i 5 -f m
iperf3 -c 10.77.0.2 -p 8002 -P 20 -w 256k -t 120 -i 5 -f m
Stronger run used for Jammy, where QEMU 6.2 only allowed RX/TX 256.
Because the ring cannot be shrunk, raise the stream count instead to
make RX buffer exhaustion frequent (P80 reproduces reliably; the earlier
P40 was intermittent):
iperf3 -c 10.77.0.2 -p 8001 -P 80 -w 256k -t 120 -i 5 -f m
iperf3 -c 10.77.0.2 -p 8002 -P 80 -w 256k -t 120 -i 5 -f m
5. After the load, verify host-to-guest connectivity:
ping -c 10 -W 1 10.77.0.2
timeout 15s iperf3 -c 10.77.0.2 -p 8001 -P 1 -t 5 -i 1 -f m
6. If host-to-guest traffic stalls, verify that the VM and guest TX path
are still alive:
- the guest serial console still accepts commands
- QMP/HMP reports the VM as running and virtio-net as not broken
- add a static neighbour entry for 10.77.0.1 inside the guest
- send one UDP packet from the guest to 10.77.0.1
- observe that UDP packet on the host with a UDP listener or tcpdump
7. Expected result on an affected qemu:
The host-to-guest RX path eventually stalls. This is observed as
either:
- an iperf3 receiver summary of 0.00 Bytes / 0.00 Mbits/sec, or
- a post-load TCP probe whose receiver side drops to 0.00 Mbits/sec.
The VM remains running. The guest serial console still works. Guest-
to-host UDP can still be observed after static ARP, showing that this is
not a full VM hang.
8. Expected result on the patched qemu:
The same workload completes without host-to-guest RX stall:
- iperf3 receiver throughput remains non-zero
- host-to-guest ping succeeds
- the post-load TCP probe succeeds
- serial-console guest-to-host ping succeeds
Validation examples from the test environment:
Noble stock qemu 1:8.2.2+ds-0ubuntu1.17:
P20 / 120s / RX 64 TX 64 reproduced the stall.
Jammy stock qemu 1:6.2+dfsg-2ubuntu6.31:
P80 / 120s / RX 256 TX 256 with a Jammy 22.04 guest reproduced the
host-to-guest RX stall in 5/5 runs.
The earlier P40 / 240s condition reproduced only intermittently (about 1
in 12). See section 13 for the rate sweep.
Patched QEMU 8.2.2 with the stable-8.2 fix:
P20 / 120s completed with non-zero receiver throughput, successful
post-load ping, successful TCP probe, and successful serial-console ping.
[ Where problems could occur ]
The change touches virtio-net RX buffer availability checking and a
virtio core notification helper. Regressions would most likely appear as
virtio-net RX performance changes, missed or extra virtqueue
notifications, or guest network connectivity problems under high
throughput.
The fix makes the notification-enable path more conservative by re-
reading the available index after enabling notification, instead of
relying on the older check that can miss a guest buffer update. The risk
is mitigated by the patch being present in newer upstream/stable QEMU
versions and by focused regression testing with sustained virtio-net
traffic.
[ Other Info ]
Upstream fix:
f937309fbdbb48c354220a3e7110c202ae4aa7fa
virtio-net: Fix network stall at the host side waiting for kick
** Affects: qemu (Ubuntu)
Importance: Undecided
Status: Fix Released
** Affects: qemu (Ubuntu Jammy)
Importance: Undecided
Assignee: Seyeong Kim (seyeongkim)
Status: In Progress
** Affects: qemu (Ubuntu Noble)
Importance: Undecided
Assignee: Seyeong Kim (seyeongkim)
Status: In Progress
** Also affects: qemu (Ubuntu Noble)
Importance: Undecided
Status: New
** Also affects: qemu (Ubuntu Jammy)
Importance: Undecided
Status: New
** Changed in: qemu (Ubuntu)
Status: New => Fix Released
** Changed in: qemu (Ubuntu Jammy)
Assignee: (unassigned) => Seyeong Kim (seyeongkim)
** Changed in: qemu (Ubuntu Noble)
Assignee: (unassigned) => Seyeong Kim (seyeongkim)
** Changed in: qemu (Ubuntu Noble)
Status: New => In Progress
** Changed in: qemu (Ubuntu Jammy)
Status: New => In Progress
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2158673
Title:
qemu: virtio-net host-to-guest RX stalls under sustained traffic
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/2158673/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
