Public bug reported: DESCRIPTION ===========
[Impact] On systems with a high CPU count, creating a veth pair emits one add and one remove uevent for every queue of every possible CPU, on both ends of the pair. udev processes the resulting storm, and the sysfs metadata work serialises on the per-kernfs-root semaphores, which throttles all later sysfs activity on the machine. Container runtimes hit this on every container, because Docker's default bridge networking creates a veth pair per container and its netlink library does not set IFLA_NUM_TX_QUEUES or IFLA_NUM_RX_QUEUES. Measured on a Vera NVL72 node, 2 sockets x 88 cores x 2 threads = 352 logical CPUs, 64k page kernel, Ubuntu 26.04, udev 259.5-0ubuntu3.4 with the default 720 workers, Docker 29.7.2 on the default bridge: - A single veth pair generates 2816 uevents on the queues subsystem: 704 adds and 702 immediate removals per device, plus 2 removals at delete, times two devices. - Creating and deleting 200 veth pairs takes 116.89 s wall time, of which 107.01 s is udev draining the backlog after creation. - Container churn throughput is capped at roughly 3,100-3,850 cycles/h from 8 concurrent containers upward, completely insensitive to offered load. Stopping udev raises it to 14,000-19,500. The penalty ranges from +276% to +514% across the loaded concurrency levels. - With udev running, kernel CPU sits at 19.9-26.0% while user CPU is around 1%, i.e. the machine is largely idle while delivering a fifth of its achievable throughput. - Mean container cycle time at 32 concurrent containers is 30.75-33.01 s, against 5.79-5.94 s with udev stopped. The cost scales with num_possible_cpus(), so larger systems are affected more severely. The originally reported figures were from a 176-CPU node and showed a smaller penalty. [Fix] Two upstream patches to drivers/net/veth.c: veth: decide peer queue count from the peer's own attributes commit 267bc3f113d8c441f57014c502443686f5d33d57, in v7.4-rc1 veth: set the real number of queues before registering the devices posted to netdev on 2026-10-16, reviewed and tested, repost after the merge window requested; carried as NVIDIA: SAUCE until it lands https://lore.kernel.org/netdev/20261016-bug-veth-init-queues-before-register-v1-1-f48eb2bb2...@gmail.com/ The second patch is the fix for this bug. veth_newlink() calls veth_init_queues() after register_netdevice() for both ends of the pair, so every queue that register_netdevice() publishes in sysfs is immediately destroyed again when the real queue count is reduced to one. Moving the calls before registration means the extra queues are never published: register_queue_kobjects() creates objects for real_num_rx_queues and real_num_tx_queues rather than num_*_queues, and for a device that is not yet registered netif_set_real_num_tx_queues() and netif_set_real_num_rx_queues() only assign the counter, skipping all kobject work and the expedited RCU grace period the tx path otherwise takes. The first patch is a prerequisite only in the sense of ordering: it corrects which attribute set decides the peer's queue count, and the two patches touch the same line upstream. ** Affects: linux-nvidia-bos (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2173592 Title: Container churn throughput drops ~5x with udev enabled because veth publishes and then removes one sysfs queue object per possible CPU To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux-nvidia-bos/+bug/2173592/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
