https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=288777
--- Comment #7 from Bren <[email protected]> --- I moved our remote backup process to rsync'ing the live filesystem vs. the snapdirs. Now I'm seeing similar but slightly different behavior: - After running the rsync process for some time, disk IO freezes completely on both the rpool and storage pools. - I can get logged in, but if I run even "dmesg" the command hangs forever (the OS drives are separate SSDs from the storage pool drives to that's odd). I can't ctrl-c out of it. - Running "top -mio" I see a zfs process at 100%. - This gets logged: swap_pager: indefinite wait buffer: bufobj: 0, blkno: 260988, size: 4096 - watchdogd doesn't always trigger a reboot. I'm running that with the these flags: -t 120 --pretimeout 60 --pretimeout-action log,printf,panic -e '/bin/ps' -w -T 15 - I capped arc_max to 50% physmem which didn't help: # sysctl -a | rg 'arc_(?:max|min)|physmem' vfs.zfs.arc_min: 2147483648 vfs.zfs.arc_max: 51339679744 hw.physmem: 102679359488 Last time this happened, I showed 50G free, 34G ARC total, 1954MB swap free. Before changing these settings, I did see memory use going above 90% but still several gig free. Most of this was ARC I imagine. As I understand it, ARC is evictable, so even if the system started to run out of available memory, it would free up memory from ARC before swapping or anything like that. I will revert these settings. - If I stop the rsync process the system recovers after a few minutes. Oddly enough (and thankfully) our production server is on 14.3-RELEASE and doesn't have this problem. It's mainly serving NFS and is fairly loaded, but probably not as much as the backup server running 2-4 x constant rsync processes, backups over NFS, plus regular ZFS snapshot sends from primary to backup. I think I'll reboot into 14.2 later today. Let me know if I can help with this in any other way. -- You are receiving this mail because: You are the assignee for the bug.
