Re: zfs meta data slowness
On 7/22/2020 9:26 AM, Eugene Grosbein wrote: > > It's hard to read due to wrapping plus, it is truncated. > Maybe additional flag -d3 or similar will help, combined with dedirection < > /dev/null > top.out > so it won't use size of your terminal to wrap/truncate output. > > Also, make sure you invoke top while "zfs" command is running. > Also, procstat -kk for pid of "zfs" command would be useful (but may occur > pretty long). > > I suppose it blocks waiting for some kernel lock and procstat would show > details. .txt file is attached. I also ran procstat -kk in a loop for a bit while it was running. In this case the list too just under 5min to run. time /sbin/zfs list -Hp -oname,creation -S creation -t snapshot > /var/log/snapshots.out 2.311u 24.963s 4:55.13 9.2% 71+179k 2706576+26io 0pf+0w Ideally, I would like to bias slightly ARC to hold this meta data a little longer than evict it out of cache so fast. I put in an nvme disk hoping that would help, but it makes no difference or doesnt seem to anyways. ---Mike ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: zfs meta data slowness
22.07.2020 20:02, mike tancsa wrote:
>
> On 7/22/2020 1:29 AM, Eugene Grosbein wrote:
>> 22.07.2020 2:37, mike tancsa wrote:
>>
Something else special about the setup.
output of "top -b"
>>> ports are right now being built in a VM, but the problem (zrepl hanging)
>>> and zfs list -t snapshots taking forever happens regardless
>>>
>>> PID USERNAMETHR PRI NICE SIZERES STATEC TIMEWCPU
>>> COMMAND
>>> 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve
>>> 98783 root 2 21016M 5136K hdr->b 4 0:01 1.95% zfs
>>> 76489 root 21 230 738M54M uwait1 2:18 0.88% zrepl
>>> 98784 root 1 21013M 3832K piperd 3 0:01 0.59% zfs
>>> 99563 root 1 20013M 4136K zio->i 4 0:00 0.39% zfs
>> You need top -SHPI
>
> last pid: 66981; load averages: 3.44, 3.25, 3.34 up 1+22:49:34
> 08:58:06
> 2056 threads: 12 running, 1988 sleeping, 3 zombie, 53 waiting
> CPU 0: 7.7% user, 0.1% nice, 15.0% system, 0.7% interrupt, 76.5% idle
> CPU 1: 9.9% user, 0.1% nice, 16.6% system, 0.1% interrupt, 73.2% idle
> CPU 2: 10.0% user, 0.1% nice, 17.5% system, 0.4% interrupt, 71.9% idle
> CPU 3: 10.3% user, 0.1% nice, 21.2% system, 0.1% interrupt, 68.2% idle
> CPU 4: 9.7% user, 0.1% nice, 15.6% system, 0.4% interrupt, 74.3% idle
> CPU 5: 10.2% user, 0.1% nice, 21.3% system, 0.1% interrupt, 68.3% idle
> CPU 6: 9.7% user, 0.1% nice, 16.6% system, 0.5% interrupt, 73.1% idle
> CPU 7: 10.1% user, 0.1% nice, 21.3% system, 0.1% interrupt, 68.4% idle
> Mem: 4236M Active, 19G Inact, 283M Laundry, 37G Wired, 3248K Buf, 1667M Free
> ARC: 25G Total, 9939M MFU, 12G MRU, 397M Anon, 573M Header, 2143M Other
> 20G Compressed, 29G Uncompressed, 1.43:1 Ratio
> Swap: 20G Total, 20G Free
>
> PID USERNAMEPRI NICE SIZERES STATEC TIMEWCPU COMMAND
> 4439 root123 20 6167M 5856M CPU2 2 20.2H 100.00%
> bhyve{vcpu 0}
> 4439 root122 20 6167M 5856M CPU3 3 20.2H 99.17%
> bhyve{vcpu 1}
>11 root155 ki31 0B 128K CPU0 0 35.9H 65.38%
> idle{idle: cpu0}
>11 root155 ki31 0B 128K CPU4 4 34.8H 63.38%
> idle{idle: cpu4}
> 66956 root 90061M54M CPU5 5 0:09 62.70% zfs
>11 root155 ki31 0B 128K CPU6 6 34.3H 58.06%
> idle{idle: cpu6}
>11 root155 ki31 0B 128K RUN 2 33.7H 54.49%
> idle{idle: cpu2}
>11 root155 ki31 0B 128K RUN 1 34.3H 53.76%
> idle{idle: cpu1}
>11 root155 ki31 0B 128K RUN 3 32.0H 53.47%
> idle{idle: cpu3}
>11 root155 ki31 0B 128K CPU7 7 32.0H 50.68%
> idle{idle: cpu7}
>11 root155 ki31 0B 128K RUN 5 32.0H 48.29%
> idle{idle: cpu5}
>56 root-12- 0B 5168K -1 5:49 9.67%
> zpool-zbackup2{zio_write_issue_3}
>56 root-12- 0B 5168K -3 5:48 9.57%
> zpool-zbackup2{zio_write_issue_5}
>56 root-12- 0B 5168K -4 5:48 9.47%
> zpool-zbackup2{zio_write_issue_4}
>56 root-12- 0B 5168K -7 5:48 9.47%
> zpool-zbackup2{zio_write_issue_2}
>56 root-12- 0B 5168K -0 5:49 9.38%
> zpool-zbackup2{zio_write_issue_1}
>56 root-12- 0B 5168K -1 5:48 9.38%
> zpool-zbackup2{zio_write_issue_0}
> 63392 root 230 729M32M uwait0 0:01 4.20%
> zrepl{zrepl}
It's hard to read due to wrapping plus, it is truncated.
Maybe additional flag -d3 or similar will help, combined with dedirection <
/dev/null > top.out
so it won't use size of your terminal to wrap/truncate output.
Also, make sure you invoke top while "zfs" command is running.
Also, procstat -kk for pid of "zfs" command would be useful (but may occur
pretty long).
I suppose it blocks waiting for some kernel lock and procstat would show
details.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: zfs meta data slowness
On 7/22/2020 1:29 AM, Eugene Grosbein wrote:
> 22.07.2020 2:37, mike tancsa wrote:
>
>>> Something else special about the setup.
>>> output of "top -b"
>>>
>> ports are right now being built in a VM, but the problem (zrepl hanging)
>> and zfs list -t snapshots taking forever happens regardless
>>
>> PID USERNAMETHR PRI NICE SIZERES STATEC TIMEWCPU
>> COMMAND
>> 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve
>> 98783 root 2 21016M 5136K hdr->b 4 0:01 1.95% zfs
>> 76489 root 21 230 738M54M uwait1 2:18 0.88% zrepl
>> 98784 root 1 21013M 3832K piperd 3 0:01 0.59% zfs
>> 99563 root 1 20013M 4136K zio->i 4 0:00 0.39% zfs
> You need top -SHPI
last pid: 66981; load averages: 3.44, 3.25, 3.34 up 1+22:49:34
08:58:06
2056 threads: 12 running, 1988 sleeping, 3 zombie, 53 waiting
CPU 0: 7.7% user, 0.1% nice, 15.0% system, 0.7% interrupt, 76.5% idle
CPU 1: 9.9% user, 0.1% nice, 16.6% system, 0.1% interrupt, 73.2% idle
CPU 2: 10.0% user, 0.1% nice, 17.5% system, 0.4% interrupt, 71.9% idle
CPU 3: 10.3% user, 0.1% nice, 21.2% system, 0.1% interrupt, 68.2% idle
CPU 4: 9.7% user, 0.1% nice, 15.6% system, 0.4% interrupt, 74.3% idle
CPU 5: 10.2% user, 0.1% nice, 21.3% system, 0.1% interrupt, 68.3% idle
CPU 6: 9.7% user, 0.1% nice, 16.6% system, 0.5% interrupt, 73.1% idle
CPU 7: 10.1% user, 0.1% nice, 21.3% system, 0.1% interrupt, 68.4% idle
Mem: 4236M Active, 19G Inact, 283M Laundry, 37G Wired, 3248K Buf, 1667M Free
ARC: 25G Total, 9939M MFU, 12G MRU, 397M Anon, 573M Header, 2143M Other
20G Compressed, 29G Uncompressed, 1.43:1 Ratio
Swap: 20G Total, 20G Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
4439 root 123 20 6167M 5856M CPU2 2 20.2H 100.00%
bhyve{vcpu 0}
4439 root 122 20 6167M 5856M CPU3 3 20.2H 99.17%
bhyve{vcpu 1}
11 root 155 ki31 0B 128K CPU0 0 35.9H 65.38%
idle{idle: cpu0}
11 root 155 ki31 0B 128K CPU4 4 34.8H 63.38%
idle{idle: cpu4}
66956 root 90 0 61M 54M CPU5 5 0:09 62.70% zfs
11 root 155 ki31 0B 128K CPU6 6 34.3H 58.06%
idle{idle: cpu6}
11 root 155 ki31 0B 128K RUN 2 33.7H 54.49%
idle{idle: cpu2}
11 root 155 ki31 0B 128K RUN 1 34.3H 53.76%
idle{idle: cpu1}
11 root 155 ki31 0B 128K RUN 3 32.0H 53.47%
idle{idle: cpu3}
11 root 155 ki31 0B 128K CPU7 7 32.0H 50.68%
idle{idle: cpu7}
11 root 155 ki31 0B 128K RUN 5 32.0H 48.29%
idle{idle: cpu5}
56 root -12 - 0B 5168K - 1 5:49 9.67%
zpool-zbackup2{zio_write_issue_3}
56 root -12 - 0B 5168K - 3 5:48 9.57%
zpool-zbackup2{zio_write_issue_5}
56 root -12 - 0B 5168K - 4 5:48 9.47%
zpool-zbackup2{zio_write_issue_4}
56 root -12 - 0B 5168K - 7 5:48 9.47%
zpool-zbackup2{zio_write_issue_2}
56 root -12 - 0B 5168K - 0 5:49 9.38%
zpool-zbackup2{zio_write_issue_1}
56 root -12 - 0B 5168K - 1 5:48 9.38%
zpool-zbackup2{zio_write_issue_0}
63392 root 23 0 729M 32M uwait 0 0:01 4.20%
zrepl{zrepl}
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: zfs meta data slowness
On 7/22/2020 1:04 AM, Rick Macklem wrote:
> mike tancsa wrote:
>> Hi,
>>Thanks for the response. Reply in line
>>
>> On 7/20/2020 9:04 AM, Ronald Klop wrote:
>>> Hi,
>>>
>>> My first suggestion would be to remove a lot of snapshots. But that my
>>> not match your business case.
>> As its a backup server, its sort of the point to have all those snapshots.
> I'm the last guy who should be commenting on ZFS, since I never use it.
> However, it is my understanding that ZFS "pseudo automounts" each
> snapshot when you go there, so I think that might be what is taking
> so long (ie. not really meta data).
>
Thanks Rick, in this case, its just listing snapshots from the command line
zfs list -t snapshot
that is taking forever.
Best case scenario when the box boots up after running it once, it will
take about 25 seconds
but when the box is taking in zfs streams, it really slows down and can
be anywhere up to 30min
0{backup4}# time zfs list -t snapshot > /tmp/snap.out
1.839u 23.211s 3:11.69 13.0% 71+178k 2504801+38io 0pf+0w
0{backup4}# time zfs list -t snapshot > /tmp/snap.out
1.817u 23.612s 0:25.47 99.8% 71+178k 2472088+38io 0pf+0w
0{backup4}# time zfs list -t snapshot > /tmp/snap.out
2.040u 23.314s 0:25.40 99.8% 71+177k 2472105+38io 0pf+0w
0{backup4}#
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: zfs meta data slowness
Van: mike tancsa Datum: dinsdag, 21 juli 2020 21:37 Aan: Ronald Klop , FreeBSD-STABLE Mailing List Onderwerp: Re: zfs meta data slowness Hi, Thanks for the response. Reply in line On 7/20/2020 9:04 AM, Ronald Klop wrote: > Hi, > > My first suggestion would be to remove a lot of snapshots. But that my > not match your business case. As its a backup server, its sort of the point to have all those snapshots. > Maybe you can provide more information about your setup: > Amount of RAM, CPU? 64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz > output of "zpool status" # zpool status -x all pools are healthy That is nice to know. Instead of zpool status -x, the output of "zpool status" is very interesting. And "zpool list" also. That gives the reader information about your setup, which helps thinking along about the possible cause. But as somebody else mentioned. Profiling the kernel might be the best thing to do. Dtrace can be used for it. But I don't know these commands by heart. If I remember correctly there is an optimization for "zfs list -o name". This is much faster because it does not get extra information from the disks. See: https://svnweb.freebsd.org/base?view=revision&revision=230438 Regards, Ronald. > > output of "zfs list" if possible to share its a big list # zfs list | wc 8244120 107511 > Type of disks/ssds? old school Device Model: WDC WD80EFAX-68KNBN0 > What is the load of the system? I/O per second, etc. its not cpu bound, disks are sometimes running at 100% based on gstat, but not always > Do you use dedup, GELI? no and no > Something else special about the setup. > output of "top -b" > ports are right now being built in a VM, but the problem (zrepl hanging) and zfs list -t snapshots taking forever happens regardless PID USERNAMETHR PRI NICE SIZERES STATEC TIMEWCPU COMMAND 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve 98783 root 2 21016M 5136K hdr->b 4 0:01 1.95% zfs 76489 root 21 230 738M54M uwait1 2:18 0.88% zrepl 98784 root 1 21013M 3832K piperd 3 0:01 0.59% zfs 99563 root 1 20013M 4136K zio->i 4 0:00 0.39% zfs 16136 root 18 250 705M56M uwait3 29:58 0.00% zrepl-freebsd-amd64 1845 root 1 20012M 3772K nanslp 7 5:54 0.00% ossec-syscheckd 1567 root 1 20011M 2744K select 0 2:22 0.00% syslogd 1737 root 32 20011M 2844K rpcsvc 6 1:40 0.00% nfsd 1660 root 1 -52 r011M11M nanslp 5 1:18 0.00% watchdogd 1434 root 1 200 9988K 988K select 3 0:27 0.00% devd 2435 mdtancsa 1 20020M 8008K select 0 0:21 0.00% sshd 1754 root 3 20018M 3556K select 1 0:11 0.00% apcupsd 5917 root 1 20011M 2672K select 2 0:06 0.00% script 1449 _pflogd 1 20012M 3572K bpf 3 0:05 0.00% pflogd ---Mike > That kind of information. > > Regards, > Ronald. > > > Van: mike tancsa > Datum: zondag, 19 juli 2020 16:17 > Aan: FreeBSD-STABLE Mailing List > Onderwerp: zfs meta data slowness >> >> Are there any tweaks that can be done to speed up or improve zfs >> metadata performance ? I have a backup server with a lot of snapshots >> (40,000) and just doing a listing can take a great deal of time. Best >> case scenario is about 24 seconds, worst case, I have seen it up to 15 >> minutes. (FreeBSD 12.1-STABLE r363078) >> >> >> ARC Efficiency: 79.33b >> Cache Hit Ratio:92.81% 73.62b >> Cache Miss Ratio: 7.19% 5.71b >> Actual Hit Ratio: 92.78% 73.60b >> >> Data Demand Efficiency: 96.47% 461.91m >> Data Prefetch Efficiency: 1.00% 262.73m >> >> CACHE HITS BY CACHE LIST: >> Anonymously Used: 0.01% 3.86m >> Most Recently Used: 3.91% 2.88b >> Most Frequently Used: 96.06% 70.72b >> Most Recently Used Ghost: 0.01% 5.31m >> Most Frequently Used Ghost: 0.01% 10.47m >> >> CACHE HITS BY DATA TYPE: >> Demand Data: 0.61% 445.60m >> Prefetch Data:0.00% 2.63m >> Demand Metadata: 99.36% 73.15b >> Prefetch Metadata:0.03% 21.00m >> >> CACHE MISSES BY DATA TYPE: >> Demand Data: 0.29% 16.31m >>
Re: zfs meta data slowness
22.07.2020 2:37, mike tancsa wrote: >> Something else special about the setup. >> output of "top -b" >> > > ports are right now being built in a VM, but the problem (zrepl hanging) > and zfs list -t snapshots taking forever happens regardless > > PID USERNAMETHR PRI NICE SIZERES STATEC TIMEWCPU > COMMAND > 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve > 98783 root 2 21016M 5136K hdr->b 4 0:01 1.95% zfs > 76489 root 21 230 738M54M uwait1 2:18 0.88% zrepl > 98784 root 1 21013M 3832K piperd 3 0:01 0.59% zfs > 99563 root 1 20013M 4136K zio->i 4 0:00 0.39% zfs You need top -SHPI ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: zfs meta data slowness
mike tancsa wrote: >Hi, >Thanks for the response. Reply in line > >On 7/20/2020 9:04 AM, Ronald Klop wrote: >> Hi, >> >> My first suggestion would be to remove a lot of snapshots. But that my >> not match your business case. > >As its a backup server, its sort of the point to have all those snapshots. I'm the last guy who should be commenting on ZFS, since I never use it. However, it is my understanding that ZFS "pseudo automounts" each snapshot when you go there, so I think that might be what is taking so long (ie. not really meta data). Of course I have no idea what might speed that up. I would be tempted to look in ZFS for the "snapshot mounting code", in case I could find an obvious problem... rick > Maybe you can provide more information about your setup: > Amount of RAM, CPU? 64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz > output of "zpool status" # zpool status -x all pools are healthy > output of "zfs list" if possible to share its a big list # zfs list | wc 8244120 107511 > Type of disks/ssds? old school Device Model: WDC WD80EFAX-68KNBN0 > What is the load of the system? I/O per second, etc. its not cpu bound, disks are sometimes running at 100% based on gstat, but not always > Do you use dedup, GELI? no and no > Something else special about the setup. > output of "top -b" > ports are right now being built in a VM, but the problem (zrepl hanging) and zfs list -t snapshots taking forever happens regardless PID USERNAMETHR PRI NICE SIZERES STATEC TIMEWCPU COMMAND 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve 98783 root 2 21016M 5136K hdr->b 4 0:01 1.95% zfs 76489 root 21 230 738M54M uwait1 2:18 0.88% zrepl 98784 root 1 21013M 3832K piperd 3 0:01 0.59% zfs 99563 root 1 20013M 4136K zio->i 4 0:00 0.39% zfs 16136 root 18 250 705M56M uwait3 29:58 0.00% zrepl-freebsd-amd64 1845 root 1 20012M 3772K nanslp 7 5:54 0.00% ossec-syscheckd 1567 root 1 20011M 2744K select 0 2:22 0.00% syslogd 1737 root 32 20011M 2844K rpcsvc 6 1:40 0.00% nfsd 1660 root 1 -52 r011M11M nanslp 5 1:18 0.00% watchdogd 1434 root 1 200 9988K 988K select 3 0:27 0.00% devd 2435 mdtancsa 1 20020M 8008K select 0 0:21 0.00% sshd 1754 root 3 20018M 3556K select 1 0:11 0.00% apcupsd 5917 root 1 20011M 2672K select 2 0:06 0.00% script 1449 _pflogd 1 20012M 3572K bpf 3 0:05 0.00% pflogd ---Mike > That kind of information. > > Regards, > Ronald. > > > Van: mike tancsa > Datum: zondag, 19 juli 2020 16:17 > Aan: FreeBSD-STABLE Mailing List > Onderwerp: zfs meta data slowness >> >> Are there any tweaks that can be done to speed up or improve zfs >> metadata performance ? I have a backup server with a lot of snapshots >> (40,000) and just doing a listing can take a great deal of time. Best >> case scenario is about 24 seconds, worst case, I have seen it up to 15 >> minutes. (FreeBSD 12.1-STABLE r363078) >> >> >> ARC Efficiency: 79.33b >> Cache Hit Ratio:92.81% 73.62b >> Cache Miss Ratio: 7.19% 5.71b >> Actual Hit Ratio: 92.78% 73.60b >> >> Data Demand Efficiency: 96.47% 461.91m >> Data Prefetch Efficiency: 1.00% 262.73m >> >> CACHE HITS BY CACHE LIST: >> Anonymously Used: 0.01% 3.86m >> Most Recently Used: 3.91% 2.88b >> Most Frequently Used: 96.06% 70.72b >> Most Recently Used Ghost: 0.01% 5.31m >> Most Frequently Used Ghost: 0.01% 10.47m >> >> CACHE HITS BY DATA TYPE: >> Demand Data: 0.61% 445.60m >> Prefetch Data:0.00% 2.63m >> Demand Metadata: 99.36% 73.15b >> Prefetch Metadata:0.03% 21.00m >> >> CACHE MISSES BY DATA TYPE: >> Demand Data: 0.29% 16.31m >> Prefetch Data:4.56% 260.10m >> Demand Metadata: 95.02% 5.42b >> Prefetch Metadata:0.14% 7.75m >> >> >> Other than increase the metadata max, I havent really changed any >> tuneables >> >&g
Re: zfs meta data slowness
Hi, Thanks for the response. Reply in line On 7/20/2020 9:04 AM, Ronald Klop wrote: > Hi, > > My first suggestion would be to remove a lot of snapshots. But that my > not match your business case. As its a backup server, its sort of the point to have all those snapshots. > Maybe you can provide more information about your setup: > Amount of RAM, CPU? 64G, Xeon(R) CPU E3-1240 v6 @ 3.70GHz > output of "zpool status" # zpool status -x all pools are healthy > output of "zfs list" if possible to share its a big list # zfs list | wc 824 4120 107511 > Type of disks/ssds? old school Device Model: WDC WD80EFAX-68KNBN0 > What is the load of the system? I/O per second, etc. its not cpu bound, disks are sometimes running at 100% based on gstat, but not always > Do you use dedup, GELI? no and no > Something else special about the setup. > output of "top -b" > ports are right now being built in a VM, but the problem (zrepl hanging) and zfs list -t snapshots taking forever happens regardless PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 4439 root 12 40 20 6167M 5762M kqread 3 535:13 200.00% bhyve 98783 root 2 21 0 16M 5136K hdr->b 4 0:01 1.95% zfs 76489 root 21 23 0 738M 54M uwait 1 2:18 0.88% zrepl 98784 root 1 21 0 13M 3832K piperd 3 0:01 0.59% zfs 99563 root 1 20 0 13M 4136K zio->i 4 0:00 0.39% zfs 16136 root 18 25 0 705M 56M uwait 3 29:58 0.00% zrepl-freebsd-amd64 1845 root 1 20 0 12M 3772K nanslp 7 5:54 0.00% ossec-syscheckd 1567 root 1 20 0 11M 2744K select 0 2:22 0.00% syslogd 1737 root 32 20 0 11M 2844K rpcsvc 6 1:40 0.00% nfsd 1660 root 1 -52 r0 11M 11M nanslp 5 1:18 0.00% watchdogd 1434 root 1 20 0 9988K 988K select 3 0:27 0.00% devd 2435 mdtancsa 1 20 0 20M 8008K select 0 0:21 0.00% sshd 1754 root 3 20 0 18M 3556K select 1 0:11 0.00% apcupsd 5917 root 1 20 0 11M 2672K select 2 0:06 0.00% script 1449 _pflogd 1 20 0 12M 3572K bpf 3 0:05 0.00% pflogd ---Mike > That kind of information. > > Regards, > Ronald. > > > Van: mike tancsa > Datum: zondag, 19 juli 2020 16:17 > Aan: FreeBSD-STABLE Mailing List > Onderwerp: zfs meta data slowness >> >> Are there any tweaks that can be done to speed up or improve zfs >> metadata performance ? I have a backup server with a lot of snapshots >> (40,000) and just doing a listing can take a great deal of time. Best >> case scenario is about 24 seconds, worst case, I have seen it up to 15 >> minutes. (FreeBSD 12.1-STABLE r363078) >> >> >> ARC Efficiency: 79.33b >> Cache Hit Ratio: 92.81% 73.62b >> Cache Miss Ratio: 7.19% 5.71b >> Actual Hit Ratio: 92.78% 73.60b >> >> Data Demand Efficiency: 96.47% 461.91m >> Data Prefetch Efficiency: 1.00% 262.73m >> >> CACHE HITS BY CACHE LIST: >> Anonymously Used: 0.01% 3.86m >> Most Recently Used: 3.91% 2.88b >> Most Frequently Used: 96.06% 70.72b >> Most Recently Used Ghost: 0.01% 5.31m >> Most Frequently Used Ghost: 0.01% 10.47m >> >> CACHE HITS BY DATA TYPE: >> Demand Data: 0.61% 445.60m >> Prefetch Data: 0.00% 2.63m >> Demand Metadata: 99.36% 73.15b >> Prefetch Metadata: 0.03% 21.00m >> >> CACHE MISSES BY DATA TYPE: >> Demand Data: 0.29% 16.31m >> Prefetch Data: 4.56% 260.10m >> Demand Metadata: 95.02% 5.42b >> Prefetch Metadata: 0.14% 7.75m >> >> >> Other than increase the metadata max, I havent really changed any >> tuneables >> >> >> ZFS Tunables (sysctl): >> kern.maxusers 4416 >> vm.kmem_size 66691842048 >> vm.kmem_size_scale 1 >> vm.kmem_size_min 0 >> vm.kmem_size_max 1319413950874 >> vfs.zfs.trim.max_interval 1 >> vfs.zfs.trim.timeout 30 >> vfs.zfs.trim.tx
Re: zfs meta data slowness
19.07.2020 21:17, mike tancsa wrote: > Are there any tweaks that can be done to speed up or improve zfs > metadata performance ? I have a backup server with a lot of snapshots > (40,000) and just doing a listing can take a great deal of time. Best > case scenario is about 24 seconds, worst case, I have seen it up to 15 > minutes. (FreeBSD 12.1-STABLE r363078) I believe you have to perform kernel profiling for your specific use case. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: zfs meta data slowness
Hi, My first suggestion would be to remove a lot of snapshots. But that my not match your business case. Maybe you can provide more information about your setup: Amount of RAM, CPU? output of "zpool status" output of "zfs list" if possible to share Type of disks/ssds? What is the load of the system? I/O per second, etc. Do you use dedup, GELI? Something else special about the setup. output of "top -b" That kind of information. Regards, Ronald. Van: mike tancsa Datum: zondag, 19 juli 2020 16:17 Aan: FreeBSD-STABLE Mailing List Onderwerp: zfs meta data slowness Are there any tweaks that can be done to speed up or improve zfs metadata performance ? I have a backup server with a lot of snapshots (40,000) and just doing a listing can take a great deal of time. Best case scenario is about 24 seconds, worst case, I have seen it up to 15 minutes. (FreeBSD 12.1-STABLE r363078) ARC Efficiency: 79.33b Cache Hit Ratio:92.81% 73.62b Cache Miss Ratio: 7.19% 5.71b Actual Hit Ratio: 92.78% 73.60b Data Demand Efficiency: 96.47% 461.91m Data Prefetch Efficiency: 1.00% 262.73m CACHE HITS BY CACHE LIST: Anonymously Used: 0.01% 3.86m Most Recently Used: 3.91% 2.88b Most Frequently Used: 96.06% 70.72b Most Recently Used Ghost: 0.01% 5.31m Most Frequently Used Ghost: 0.01% 10.47m CACHE HITS BY DATA TYPE: Demand Data: 0.61% 445.60m Prefetch Data:0.00% 2.63m Demand Metadata: 99.36% 73.15b Prefetch Metadata:0.03% 21.00m CACHE MISSES BY DATA TYPE: Demand Data: 0.29% 16.31m Prefetch Data:4.56% 260.10m Demand Metadata: 95.02% 5.42b Prefetch Metadata:0.14% 7.75m Other than increase the metadata max, I havent really changed any tuneables ZFS Tunables (sysctl): kern.maxusers 4416 vm.kmem_size66691842048 vm.kmem_size_scale 1 vm.kmem_size_min0 vm.kmem_size_max1319413950874 vfs.zfs.trim.max_interval 1 vfs.zfs.trim.timeout30 vfs.zfs.trim.txg_delay 32 vfs.zfs.trim.enabled1 vfs.zfs.vol.immediate_write_sz 32768 vfs.zfs.vol.unmap_sync_enabled 0 vfs.zfs.vol.unmap_enabled 1 vfs.zfs.vol.recursive 0 vfs.zfs.vol.mode1 vfs.zfs.version.zpl 5 vfs.zfs.version.spa 5000 vfs.zfs.version.acl 1 vfs.zfs.version.ioctl 7 vfs.zfs.debug 0 vfs.zfs.super_owner 0 vfs.zfs.immediate_write_sz 32768 vfs.zfs.sync_pass_rewrite 2 vfs.zfs.sync_pass_dont_compress 5 vfs.zfs.sync_pass_deferred_free 2 vfs.zfs.zio.dva_throttle_enabled1 vfs.zfs.zio.exclude_metadata0 vfs.zfs.zio.use_uma 1 vfs.zfs.zio.taskq_batch_pct 75 vfs.zfs.zil_maxblocksize131072 vfs.zfs.zil_slog_bulk 786432 vfs.zfs.zil_nocacheflush0 vfs.zfs.zil_replay_disable 0 vfs.zfs.cache_flush_disable 0 vfs.zfs.standard_sm_blksz 131072 vfs.zfs.dtl_sm_blksz4096 vfs.zfs.min_auto_ashift 9 vfs.zfs.max_auto_ashift 13 vfs.zfs.vdev.trim_max_pending 1 vfs.zfs.vdev.bio_delete_disable 0 vfs.zfs.vdev.bio_flush_disable 0 vfs.zfs.vdev.def_queue_depth32 vfs.zfs.vdev.queue_depth_pct1000 vfs.zfs.vdev.write_gap_limit4096 vfs.zfs.vdev.read_gap_limit 32768 vfs.zfs.vdev.aggregation_limit_non_rotating131072 vfs.zfs.vdev.aggregation_limit 1048576 vfs.zfs.vdev.initializing_max_active1 vfs.zfs.vdev.initializing_min_active1 vfs.zfs.vdev.removal_max_active 2 vfs.zfs.vdev.removal_min_active 1 vfs.zfs.vdev.trim_max_active64 vfs.zfs.vdev.trim_min_active1 vfs.zfs.vdev.scrub_max_active 2 vfs.zfs.vdev.scrub_min_active 1 vfs.zfs.vdev
zfs meta data slowness
Are there any tweaks that can be done to speed up or improve zfs metadata performance ? I have a backup server with a lot of snapshots (40,000) and just doing a listing can take a great deal of time. Best case scenario is about 24 seconds, worst case, I have seen it up to 15 minutes. (FreeBSD 12.1-STABLE r363078) ARC Efficiency: 79.33b Cache Hit Ratio: 92.81% 73.62b Cache Miss Ratio: 7.19% 5.71b Actual Hit Ratio: 92.78% 73.60b Data Demand Efficiency: 96.47% 461.91m Data Prefetch Efficiency: 1.00% 262.73m CACHE HITS BY CACHE LIST: Anonymously Used: 0.01% 3.86m Most Recently Used: 3.91% 2.88b Most Frequently Used: 96.06% 70.72b Most Recently Used Ghost: 0.01% 5.31m Most Frequently Used Ghost: 0.01% 10.47m CACHE HITS BY DATA TYPE: Demand Data: 0.61% 445.60m Prefetch Data: 0.00% 2.63m Demand Metadata: 99.36% 73.15b Prefetch Metadata: 0.03% 21.00m CACHE MISSES BY DATA TYPE: Demand Data: 0.29% 16.31m Prefetch Data: 4.56% 260.10m Demand Metadata: 95.02% 5.42b Prefetch Metadata: 0.14% 7.75m Other than increase the metadata max, I havent really changed any tuneables ZFS Tunables (sysctl): kern.maxusers 4416 vm.kmem_size 66691842048 vm.kmem_size_scale 1 vm.kmem_size_min 0 vm.kmem_size_max 1319413950874 vfs.zfs.trim.max_interval 1 vfs.zfs.trim.timeout 30 vfs.zfs.trim.txg_delay 32 vfs.zfs.trim.enabled 1 vfs.zfs.vol.immediate_write_sz 32768 vfs.zfs.vol.unmap_sync_enabled 0 vfs.zfs.vol.unmap_enabled 1 vfs.zfs.vol.recursive 0 vfs.zfs.vol.mode 1 vfs.zfs.version.zpl 5 vfs.zfs.version.spa 5000 vfs.zfs.version.acl 1 vfs.zfs.version.ioctl 7 vfs.zfs.debug 0 vfs.zfs.super_owner 0 vfs.zfs.immediate_write_sz 32768 vfs.zfs.sync_pass_rewrite 2 vfs.zfs.sync_pass_dont_compress 5 vfs.zfs.sync_pass_deferred_free 2 vfs.zfs.zio.dva_throttle_enabled 1 vfs.zfs.zio.exclude_metadata 0 vfs.zfs.zio.use_uma 1 vfs.zfs.zio.taskq_batch_pct 75 vfs.zfs.zil_maxblocksize 131072 vfs.zfs.zil_slog_bulk 786432 vfs.zfs.zil_nocacheflush 0 vfs.zfs.zil_replay_disable 0 vfs.zfs.cache_flush_disable 0 vfs.zfs.standard_sm_blksz 131072 vfs.zfs.dtl_sm_blksz 4096 vfs.zfs.min_auto_ashift 9 vfs.zfs.max_auto_ashift 13 vfs.zfs.vdev.trim_max_pending 1 vfs.zfs.vdev.bio_delete_disable 0 vfs.zfs.vdev.bio_flush_disable 0 vfs.zfs.vdev.def_queue_depth 32 vfs.zfs.vdev.queue_depth_pct 1000 vfs.zfs.vdev.write_gap_limit 4096 vfs.zfs.vdev.read_gap_limit 32768 vfs.zfs.vdev.aggregation_limit_non_rotating131072 vfs.zfs.vdev.aggregation_limit 1048576 vfs.zfs.vdev.initializing_max_active 1 vfs.zfs.vdev.initializing_min_active 1 vfs.zfs.vdev.removal_max_active 2 vfs.zfs.vdev.removal_min_active 1 vfs.zfs.vdev.trim_max_active 64 vfs.zfs.vdev.trim_min_active 1 vfs.zfs.vdev.scrub_max_active 2 vfs.zfs.vdev.scrub_min_active 1 vfs.zfs.vdev.async_write_max_active 10 vfs.zfs.vdev.async_write_min_active 1 vfs.zfs.vdev.async_read_max_active 3 vfs.zfs.vdev.async_read_min_active 1 vfs.zfs.vdev.sync_write_max_active 10 vfs.zfs.vdev.sync_write_min_active 10 vfs.zfs.vdev.sync_read_max_active 10 vfs.zfs.vdev.sync_read_min_active 10 vfs.zfs.vdev.max_active 1000 vfs.zfs.vdev.async_write_active_max_dirty_percent60 vfs.zfs.vdev.async_write_active_min_dirty_percent30 vfs.zfs.vdev.mirror.non_rotating_seek_inc1 vfs
