On Mon, 2022-04-25 at 15:27 +0200, Alexander Leidinger wrote:Quoting Alexander Leidinger <alexan...@leidinger.net> (from Sun, 24 Apr 2022 19:58:17 +0200):> Quoting Alexander Leidinger <alexan...@leidinger.net> (from Fri, 22 > Apr 2022 09:04:39 +0200): > > > Quoting Doug Ambrisko <ambri...@ambrisko.com> (from Thu, 21 Apr > > 2022 09:38:35 -0700): > > > > I've attached mount.patch that when doing mount -v should > > > show the vnode usage per filesystem. Note that the problem I was > > > running into was after some operations arc_prune and arc_evict would > > > consume 100% of 2 cores and make ZFS really slow. If you are not > > > running into that issue then nocache etc. shouldn't be needed. > > > > I don't run into this issue, but I have a huge perf difference when > > using nocache in the nightly periodic runs. 4h instead of 12-24h > > (22 jails on this system). > > > > > On my laptop I set ARC to 1G since I don't use swap and in the past > > > ARC would consume to much memory and things would die. When the > > > nullfs holds a bunch of vnodes then ZFS couldn't release them. > > > > > > FYI, on my laptop with nocache and limited vnodes I haven't run > > > into this problem. I haven't tried the patch to let ZFS free > > > it's and nullfs vnodes on my laptop. I have only tried it via > > > > I have this patch and your mount patch installed now, without > > nocache and reduced arc reclaim settings (100, 1). I will check the > > runtime for the next 2 days. > > 9-10h runtime with the above settings (compared to 4h with nocache > and 12-24h without any patch and without nocache). > I changed the sysctls back to the defaults and will see in the next > run (in 7h) what the result is with just the patches. And again 9-10h runtime (I've seen a lot of the find processes in the periodic daily run of those 22 jails in the state "*vnode"). Seems nocache gives the best perf for me in this case.Sorry for jumping in here - I've got a couple of questions: - Will this also apply to nullfs read-only mounts? Or is it only in case of writing "through" a nullfs mount that these problems are seen? - Is it a problem also in 13, or is this "new" in -CURRENT? We're having weird and unexplained CPU spikes on several systems, even after tuning geli to not use gazillions of threads. So far our suspicion has been ZFS snapshot cleanups but this is an interesting contender - unless the whole "read only" part makes it moot.
For me this started after creating one more jail on this system and I dont't see CPU spikes (as the system is running permanently at 100% and the distribution of the CPU looks as I would expect it). The experience of Doug is a little bit different, as he experiences a high amount of CPU usage "for nothing" or even a dead-lock like situation. So I would say we see different things based on similar triggers.
The nocache option for nullfs is affecting the number of vnodes in use on the system no matter if ro or rw. As such you can give it a try. Note, depending on the usage pattern, the nocache option may increase lock contention. So it may or may not have a positive or negative performance impact.
Bye, Alexander. -- http://www.Leidinger.net alexan...@leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netch...@freebsd.org : PGP 0x8F31830F9F2772BF
pgp8REsu61FBW.pgp
Description: Digitale PGP-Signatur
