Re: CURRENT: slow like crap! ZFS scrubbing and ports update > 25 min
Am Thu, 23 Mar 2017 15:38:05 +0300 Slawa Olhovchenkov schrieb: > On Wed, Mar 22, 2017 at 10:25:24PM +0100, O. Hartmann wrote: > > > Am Wed, 22 Mar 2017 21:10:51 +0100 > > Michael Gmelin schrieb: > > > > > > On 22 Mar 2017, at 21:02, O. Hartmann wrote: > > > > > > > > CURRENT (FreeBSD 12.0-CURRENT #82 r315720: Wed Mar 22 18:49:28 CET 2017 > > > > amd64) is > > > > annoyingly slow! While scrubbing is working on my 12 GB ZFS volume, > > > > updating /usr/ports takes >25 min(!). That is an absolute record now. > > > > > > > > I do an almost daily update of world and ports tree and have periodic > > > > scrubbing > > > > ZFS volumes every 35 days, as it is defined in /etc/defaults. Prts tree > > > > hasn't > > > > grown much, the content of the ZFS volume hasn't changed much (~ 100 > > > > GB, its fill > > > > is about 4 TB now) and this is now for ~ 2 years constant. > > > > > > > > I've experienced before that while scrubbing the ZFS volume, some > > > > operations, > > > > even the update of /usr/ports which resides on that ZFS RAIDZ volume, > > > > takes a bit > > > > longer than usual - but never that long like now! > > > > > > > > Another box is quite unusable while it is scrubbing and it has been > > > > usable times > > > > before. The change is dramatic ... > > > > > > > > > > What do "zpool list", "gstat" and "zpool status" show? > > > > > > > > > > > zpool list: > > > > NAME SIZE ALLOC FREE EXPANDSZ FRAGCAP DEDUP HEALTH > > ALTROOT > > TANK00 10.9T 5.45T 5.42T - 7%50% 1.58x ONLINE - > > > > Deduplication is off right now, I had one ZFS filesystem with dedup enabled > > > > gstat: not shown here, but the drives comprise the volume (4x 3 TB) show > > 100% busy > > each, but one drive is always a bit off (by 10% lower) and this drive is > > walking > > through all four drives ada2, ada3, ada4 and ada5. Nothing unusual in that > > situation. > > But the throughput is incredible low, for example ada4: > > > > L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name > > 2174174 1307 11.4 0 00.0 99.4| ada4 > > > > kBps (kilo Bits per second I presume) are peaking at ~ 4800 - 5000. On > > another bos, > > this is ~ 20x higher! Most time, kBps r and w stay at ~ 500 -600. > > kilo Bytes. > 174 rps is normal for general 7200 RPM disk. Transfer too low by every > request is about 1307/174 = ~8 KB. Don't know root cause of this. I am > see raid-z of 4 disk, 8*3 = ~24KB per record. May be compession enable > and zfs use 128KB record size? For this case this is expected > performance. Use 1MB and higher record size. > I've shutdown the box over night and rebooted this morning. After checking from remote the output of "zpool status", the throughput was at ~229 MBytes/s - a value I'd expected, peaking again at ~ 300 MBytes/s. I assume my crap home hardware is not providing more, but at this point, everything is as expected. The load, as observed via top, showed ~75 - 85% idle (top -S). But anyway, on the other home box with ZFS scrubbing active, the drives showed a throughput of ~ 110 MBystes/s and 129 MBytes/s - also value I'd expected. But the system was really jumpy and the load showed ~ 80% idle (two cores, 4 threads, 8 GB RAM, the first box mentioned with the larger array has 4 cores/8 threads and 16 GB). -- O. Hartmann Ich widerspreche der Nutzung oder Übermittlung meiner Daten für Werbezwecke oder für die Markt- oder Meinungsforschung (§ 28 Abs. 4 BDSG). pgpYqkPmFuvA2.pgp Description: OpenPGP digital signature
Re: CURRENT: slow like crap! ZFS scrubbing and ports update > 25 min
On Wed, Mar 22, 2017 at 10:25:24PM +0100, O. Hartmann wrote: > Am Wed, 22 Mar 2017 21:10:51 +0100 > Michael Gmelin schrieb: > > > > On 22 Mar 2017, at 21:02, O. Hartmann wrote: > > > > > > CURRENT (FreeBSD 12.0-CURRENT #82 r315720: Wed Mar 22 18:49:28 CET 2017 > > > amd64) is > > > annoyingly slow! While scrubbing is working on my 12 GB ZFS volume, > > > updating /usr/ports takes >25 min(!). That is an absolute record now. > > > > > > I do an almost daily update of world and ports tree and have periodic > > > scrubbing ZFS > > > volumes every 35 days, as it is defined in /etc/defaults. Prts tree > > > hasn't grown much, > > > the content of the ZFS volume hasn't changed much (~ 100 GB, its fill is > > > about 4 TB > > > now) and this is now for ~ 2 years constant. > > > > > > I've experienced before that while scrubbing the ZFS volume, some > > > operations, even the > > > update of /usr/ports which resides on that ZFS RAIDZ volume, takes a bit > > > longer than > > > usual - but never that long like now! > > > > > > Another box is quite unusable while it is scrubbing and it has been > > > usable times > > > before. The change is dramatic ... > > > > > > > What do "zpool list", "gstat" and "zpool status" show? > > > > > > > zpool list: > > NAME SIZE ALLOC FREE EXPANDSZ FRAGCAP DEDUP HEALTH ALTROOT > TANK00 10.9T 5.45T 5.42T - 7%50% 1.58x ONLINE - > > Deduplication is off right now, I had one ZFS filesystem with dedup enabled > > gstat: not shown here, but the drives comprise the volume (4x 3 TB) show 100% > busy each, > but one drive is always a bit off (by 10% lower) and this drive is walking > through all > four drives ada2, ada3, ada4 and ada5. Nothing unusual in that situation. But > the > throughput is incredible low, for example ada4: > > L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name > 2174174 1307 11.4 0 00.0 99.4| ada4 > > kBps (kilo Bits per second I presume) are peaking at ~ 4800 - 5000. On > another bos, this > is ~ 20x higher! Most time, kBps r and w stay at ~ 500 -600. kilo Bytes. 174 rps is normal for general 7200 RPM disk. Transfer too low by every request is about 1307/174 = ~8 KB. Don't know root cause of this. I am see raid-z of 4 disk, 8*3 = ~24KB per record. May be compession enable and zfs use 128KB record size? For this case this is expected performance. Use 1MB and higher record size. ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
Re: CURRENT: slow like crap! ZFS scrubbing and ports update > 25 min
Am Wed, 22 Mar 2017 21:10:51 +0100 Michael Gmelin schrieb: > > On 22 Mar 2017, at 21:02, O. Hartmann wrote: > > > > CURRENT (FreeBSD 12.0-CURRENT #82 r315720: Wed Mar 22 18:49:28 CET 2017 > > amd64) is > > annoyingly slow! While scrubbing is working on my 12 GB ZFS volume, > > updating /usr/ports takes >25 min(!). That is an absolute record now. > > > > I do an almost daily update of world and ports tree and have periodic > > scrubbing ZFS > > volumes every 35 days, as it is defined in /etc/defaults. Prts tree hasn't > > grown much, > > the content of the ZFS volume hasn't changed much (~ 100 GB, its fill is > > about 4 TB > > now) and this is now for ~ 2 years constant. > > > > I've experienced before that while scrubbing the ZFS volume, some > > operations, even the > > update of /usr/ports which resides on that ZFS RAIDZ volume, takes a bit > > longer than > > usual - but never that long like now! > > > > Another box is quite unusable while it is scrubbing and it has been usable > > times > > before. The change is dramatic ... > > > > What do "zpool list", "gstat" and "zpool status" show? > > > zpool list: NAME SIZE ALLOC FREE EXPANDSZ FRAGCAP DEDUP HEALTH ALTROOT TANK00 10.9T 5.45T 5.42T - 7%50% 1.58x ONLINE - Deduplication is off right now, I had one ZFS filesystem with dedup enabled gstat: not shown here, but the drives comprise the volume (4x 3 TB) show 100% busy each, but one drive is always a bit off (by 10% lower) and this drive is walking through all four drives ada2, ada3, ada4 and ada5. Nothing unusual in that situation. But the throughput is incredible low, for example ada4: L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 2174174 1307 11.4 0 00.0 99.4| ada4 kBps (kilo Bits per second I presume) are peaking at ~ 4800 - 5000. On another bos, this is ~ 20x higher! Most time, kBps r and w stay at ~ 500 -600. zpool status: pool: TANK00 state: ONLINE scan: scrub in progress since Wed Mar 22 19:12:41 2017 167G scanned out of 5.62T at 15.9M/s, 99h46m to go 0 repaired, 2.91% done config: NAME STATE READ WRITE CKSUM TANK00 ONLINE 0 0 0 raidz1-0ONLINE 0 0 0 gpt/tank00 ONLINE 0 0 0 gpt/tank01 ONLINE 0 0 0 gpt/tank02 ONLINE 0 0 0 gpt/tank03 ONLINE 0 0 0 logs gpt/zil00 ONLINE 0 0 0 cache gpt/l2arc00 ONLINE 0 0 0 errors: No known data errors Here are the incredible low numbers more clear. Usually, the hardware is capable of doing ~ 280 - 300 MBytes/s. It is stuck for hours at 15 - 16 MBytes/s. pgp8vB5BD9lum.pgp Description: OpenPGP digital signature
Re: CURRENT: slow like crap! ZFS scrubbing and ports update > 25 min
On 2017-03-22 16:02, O. Hartmann wrote: > CURRENT (FreeBSD 12.0-CURRENT #82 r315720: Wed Mar 22 18:49:28 CET 2017 > amd64) is > annoyingly slow! While scrubbing is working on my 12 GB ZFS volume, updating > /usr/ports > takes >25 min(!). That is an absolute record now. > > I do an almost daily update of world and ports tree and have periodic > scrubbing ZFS > volumes every 35 days, as it is defined in /etc/defaults. Prts tree hasn't > grown much, > the content of the ZFS volume hasn't changed much (~ 100 GB, its fill is > about 4 TB now) > and this is now for ~ 2 years constant. > > I've experienced before that while scrubbing the ZFS volume, some operations, > even the > update of /usr/ports which resides on that ZFS RAIDZ volume, takes a bit > longer than > usual - but never that long like now! > > Another box is quite unusable while it is scrubbing and it has been usable > times before. > The change is dramatic ... > > Regards, > > Oliver > Due to differences in the kern.hz setting between IllumOS and FreeBSD, the result is FreeBSD doesn't de-prioritize scrub I/O as much as IllumOS does. try: sysctl vfs.zfs.scrub_delay=40 This will speed for 40 ticks (40ms on FreeBSD), between each scrub I/O, allowing your ports update to proceed more quickly. 'zpool list' will show how fragmented your pool is, and how full it is, these may also provide insight. If you run 'top -S' while it is performing badly, what is the CPU load like? Is your /usr/ports dataset compressed? -- Allan Jude signature.asc Description: OpenPGP digital signature
Re: CURRENT: slow like crap! ZFS scrubbing and ports update > 25 min
> On 22 Mar 2017, at 21:02, O. Hartmann wrote: > > CURRENT (FreeBSD 12.0-CURRENT #82 r315720: Wed Mar 22 18:49:28 CET 2017 > amd64) is > annoyingly slow! While scrubbing is working on my 12 GB ZFS volume, updating > /usr/ports > takes >25 min(!). That is an absolute record now. > > I do an almost daily update of world and ports tree and have periodic > scrubbing ZFS > volumes every 35 days, as it is defined in /etc/defaults. Prts tree hasn't > grown much, > the content of the ZFS volume hasn't changed much (~ 100 GB, its fill is > about 4 TB now) > and this is now for ~ 2 years constant. > > I've experienced before that while scrubbing the ZFS volume, some operations, > even the > update of /usr/ports which resides on that ZFS RAIDZ volume, takes a bit > longer than > usual - but never that long like now! > > Another box is quite unusable while it is scrubbing and it has been usable > times before. > The change is dramatic ... > What do "zpool list", "gstat" and "zpool status" show? ___ freebsd-current@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
