Public bug reported: [Impact]
On Ubuntu 26.04 (zfs-linux 2.4.1-1ubuntu5.1, kernel 7.0.0-34-generic) a pool with many cached inodes and many concurrent lookups can livelock inside ZFS. On one of our NAS hosts (509 TB pool, about 16M cached inodes, NFS/rsync readers) this caused load above 1100, about 1000 stuck tasks, zero pool I/O for about 2 hours, and a forced power cycle. Cause: 1. When dnode_size exceeds arc_dnode_limit, the ARC asks every dataset to prune a large share of its cached dnodes. 2. zfs_prune() passes that whole count to a single super_cache_scan() call. The kernel's own reclaim (do_shrink_slab) works in shrinker->batch chunks of 1024; this call instead marks hundreds of thousands to millions of inodes I_FREEING up front and evicts them one at a time on the single arc_prune thread. 3. Every lookup that lands on one of those inodes fails igrab() in zfs_zget() and retries with "cond_resched(); goto again;", a busy loop that takes and drops the znode hold locks on every pass. 4. Hundreds of spinning lookups starve the evicting thread of CPU and of those locks. Eviction stalls, so the lookups never finish. The fix was merged upstream unchanged on 2026-10-09 (reviewed by Brian Behlendorf; closes openzfs/zfs#19266, PR #19275): - e95c55285dc8 "Linux: Feed the superblock shrinker in batches from zfs_prune()" - 98527e086f0d "Linux: Sleep instead of spinning when zfs_zget() races eviction" (delay(1) instead of cond_resched(), as xfs_iget() does) Both are Linux-only (41 lines added, 6 removed), change no on-disk format, ioctl or tunable, and apply cleanly to 2.4.1-1ubuntu5.1. [Test Plan] Setup: a 26.04 VM with 24 vCPUs and the ARC capped at 24 GiB; 8M files in 4000 directories (xattr=sa, atime on); 1024-2048 threads doing readdir + stat in a loop, plus a 40 GB streaming read to churn the ARC. Optionally amplify with zfs_arc_dnode_reduce_percent=100. Measure failed igrab retries (bpftrace on zfs_zget), inodes evicted per 10 s, and threads making no progress. Our results, stock vs patched: - failed igrab retries per 10 s: 1.5M -> about 100 - slowest 10 s window for the evictor: 504 inodes evicted (stalled) -> 195k - tasks waiting on znode hold locks: 206 -> 0 - worst stat(): 25.9 s -> 7.4 s zfs-test sanity.run: 826 of 827 tests give the same result on both; the one difference passed on the patched module. Write-path churn at default tunables showed no regression. The patched module has run in production on the affected host since 2026-10-08. [Where problems could occur] - Pruning now happens in shrinker->batch chunks, so a prune request makes more super_cache_scan() calls. The total number of objects scanned is unchanged. - A lookup that races eviction now sleeps one tick instead of yielding, which could add latency on that rare path. Measured p99.9 latency was unchanged. - A regression would show up as slower inode reclaim under memory pressure, or as higher latency for lookups of inodes that are being evicted. [Other Info] - The kernel's linux-main-modules-zfs-* packages are built from the kernel's own copy of zfs, so the linux source package needs the two patches as well. - Upstream issue: https://github.com/openzfs/zfs/issues/19266 - Upstream PR: https://github.com/openzfs/zfs/pull/19275 - Commits: https://github.com/openzfs/zfs/commit/e95c55285dc889ba945483361701c2ac22d47e58 and https://github.com/openzfs/zfs/commit/98527e086f0df6dd3845edac29f3e19866ac61f0 ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Affects: zfs-linux (Ubuntu) Importance: Undecided Status: New ** Also affects: linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2170426 Title: Livelock between arc_prune and zfs_zget() hangs ZFS on large pools (fixed upstream, openzfs#19266) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2170426/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
