On Mon, 28 Jul 2025 11:52:23 +0200 Friedrich Weber <f.we...@proxmox.com> wrote:
> On 23/07/2025 20:15, Stoiko Ivanov wrote: > > the following patch seems applicable and might fix an issue observed > > in our enterprise support a while ago. containers run in their own > > cgroups, thus were probably not scanned by the kernel shrinker - this > > resulted in Dnode cache numbers of 300+% reported in arc_summary. > > > > FWICT the issue was introduced in ZFS 2.2.7 > > (commit 5f73630e9cbea5efa23d16809f06e0d08523b241 see: > > https://github.com/openzfs/zfs/issues/17052#issuecomment-3065907783) > > but I assume that the increase of zfs_arc_max by default makes it > > trigger OOMs far easier. > > > > The discussion of the PR was quite instructive: > > https://github.com/openzfs/zfs/pull/17542 > > > > minimally tested on a pair of trixie VMs (building + running > > replication of a couple of containers) > > > > Suggested-by: Thomas Lamprecht <t.lampre...@proxmox.com> > > Signed-off-by: Stoiko Ivanov <s.iva...@proxmox.com> > > --- > > FWIW, tested this by setting up ZFS on root with an additional RAID-0 > for guest disks, creating a Debian container and running a script [1] in it. > > ARC size targets as reported by arc_summary: > > Target size (adaptive): 100.0 % 794.0 MiB > Min size (hard limit): 31.3 % 248.2 MiB > Max size (high water): 3:1 794.0 MiB > > With 6.14.8-2-pve (ZFS 2.3.3 without this patch), the Dnode Cache and > ARC grow considerably while the script is running and both stay like > this after the script has exited: > > ARC size (current): 294.8 % 2.3 GiB > Dnode cache target: 10.0 % 79.4 MiB > Dnode cache size: 1181.9 % 938.5 MiB > > Same on > - 6.8.12-13-pve (ZFS 2.2.8) > - 6.8.12-8-pve (ZFS 2.2.7) > > With 6.8.12-6-pve (ZFS 2.2.6), the Dnode cache size still grows to >100% > and seems to stay there, but the ARC manages to stay below 100%: > > ARC size (current): 96.8 % 768.9 MiB > Dnode cache target: 10.0 % 79.4 MiB > Dnode cache size: 333.6 % 264.9 MiB > > With this patch on top of 6.8.12-9-pve, the Dnode Cache and ARC still > grow while the script is running, but both shrink again to <100% quickly > afterwards (within a minute or so): > > ARC size (current): 30.9 % 245.3 MiB > Dnode cache target: 10.0 % 79.4 MiB > Dnode cache size: 99.0 % 78.6 MiB > > We have an issue in enterprise support with a container-heavy workload > on ZFS 2.2.7 that is likely affected by this. However, they also saw > high Dnode cache size and >100% ARC on ZFS 2.2.6 -- the latter I > couldn't reproduce with ZFS 2.2.6, but perhaps I missed some factor. Thanks for the tests, short reproducer and feedback! > > Might be nice if we could make the fix available on PVE 8 as well, > though I'm not sure how easily this can be backported to ZFS 2.2. sent my attempt at a backport: https://lore.proxmox.com/pve-devel/20250728163041.1287899-1-s.iva...@proxmox.com/T/#u > > [1] > > #!/bin/bash > for h in $(seq 100); do > ( > mkdir "dir-$h" > cd "dir-$h" || exit 1 > for i in $(seq 100); > do > mkdir "$i" > for j in $(seq 100); > do > echo test > "$i/$j.txt" > done > done > )& > done > > _______________________________________________ pve-devel mailing list pve-devel@lists.proxmox.com https://lists.proxmox.com/cgi-bin/mailman/listinfo/pve-devel
