From: Chris Arges <[email protected]> Date: Thu, 17 Sep 2026 19:38:40 -0500 > On 2026-09-17 22:10:22, Kuniyuki Iwashima wrote: > > From: Chris J Arges <[email protected]> > > Date: Thu, 17 Sep 2026 14:38:21 -0500 > > > We have observed hung tasks blocked on rtnl_mutex while network namespaces > > > were being removed. The namespaces contained many network devices, and the > > > host had accumulated a large population of entries on the global per-CPU > > > uncached route lists. A perf profile collected during one incident > > > attributed most of the cleanup worker's samples to rt_flush_dev(): > > > > > > ``` > > > 99.92% kworker/u384:3- worker_thread > > > `-88.71% process_one_work > > > `-81.02% cleanup_net > > > `-81.00% unregister_netdevice_many_notify > > > `-79.42% notifier_call_chain > > > `-78.05% fib_netdev_event > > > `-77.92% rt_flush_dev > > > ``` > > > > > > For each device, rt_flush_dev() visits every possible CPU and scans the > > > global uncached route population while its caller holds rtnl_mutex. If N > > > is > > > the number of devices, C the number of possible CPUs, and R the number of > > > uncached routes, the cost is O(N * (C + R)). > > > > > > During namespace cleanup, other processes that issue RTNETLINK operations > > > requiring the RTNL lock can stall until cleanup releases the lock. > > > > > > A minimal reproducer is available here: > > > https://github.com/arges/linux-reproducers/tree/main/rtnl-flush-storm > > > > > > This series replaces each per-CPU uncached route list with a hash table > > > using the network device as its key. Each table uses 64 buckets. > > > > This sounds a bit overkill. Also, this series still leaves > > O(N * C) loops. > > > > Given unregistering a single device is less common than > > destroying netns, I think the right approach should be to > > make the route flush once in cleanup_net() + outside RTNL. > > > > Could you try this change ? (only compile-tested) > > > Excellent, I'll test this and report back.
I found a pre-existing issue, which affects the previous diff, so on top of it, please apply this patch https://lore.kernel.org/netdev/[email protected]/T/#u and this diff : ---8<--- diff --git a/net/ipv4/route.c b/net/ipv4/route.c index d35b66b33bbc..c12e20e07749 100644 --- a/net/ipv4/route.c +++ b/net/ipv4/route.c @@ -1567,12 +1567,13 @@ void rt_add_uncached_list(struct rtable *rt) { struct uncached_list *ul = raw_cpu_ptr(&rt_uncached_list); + rt->dst.rt_uncached_list = ul; + spin_lock_bh(&ul->lock); if (!check_net(dst_dev_net_rcu(&rt->dst))) { rt_replace_uncached_list(rt); } else { - rt->dst.rt_uncached_list = ul; list_add_tail(&rt->dst.rt_uncached, &ul->head); } diff --git a/net/ipv6/route.c b/net/ipv6/route.c index 6cffe8440b44..f22793abbbb8 100644 --- a/net/ipv6/route.c +++ b/net/ipv6/route.c @@ -155,12 +155,13 @@ void rt6_uncached_list_add(struct rt6_info *rt) { struct uncached_list *ul = raw_cpu_ptr(&rt6_uncached_list); + rt->dst.rt_uncached_list = ul; + spin_lock_bh(&ul->lock); if (!check_net(dst_dev_net_rcu(&rt->dst))) { rt6_uncached_list_replace(rt); } else { - rt->dst.rt_uncached_list = ul; list_add_tail(&rt->dst.rt_uncached, &ul->head); } ---8<--- Thanks !
