Re: high disk %busy, while almost nothing happens
On Thu, Nov 26, 2015 at 10:46 PM, Eugene M. Zheganin wrote: > Hi. > > On 26.11.2015 14:19, Eugene M. Zheganin wrote: > > Hi. > > > > I'm using FreeBSD 10.1-STABLE as an application server, last week I've > > noticed that disks are always busy while gstat shows that the activity > > measured in iops/reads/writes is low, form my point of view: > > > > > > L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name > > 8 56 50 520 160.6 6286 157.4 100.2 > gpt/zfsroot0 > > 8 56 51 1474 162.8 5228 174.4 99.9 > gpt/zfsroot1 > > > > These %busy numbers arent't changing much, and from my point of view > > both disks do very little. > > > The thing is, it was the compression. As soon as I cleared the gzip > compression from busy datasets, %busy went down, almost to zero. > Affected datasets were filled with poorly compressionable files, mostly > archives or zlib-compressed data. > Data which isn't very compressible isn't a very great on a transparently compressed filesystem. Gzip is particularly bad at this. LZ4 may have had only a slight impact. Setting gzip-1 would have also been less overhead than the default gzip which I believe is gzip-6. > And this is kind of counter-intuitive: one could think that worse-case > scenario would be redundant CPU load, with constand disk i/o. In > practice, otherwise, high disk %busy happens. > Well that's basically what you had. And %busy is not really meaningful. L(q) and ops are the ones to keep an eye on. -- Adam ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: high disk %busy, while almost nothing happens
Hi. On 27.11.2015 01:37, Ivan Klymenko wrote: > On Thu, 26 Nov 2015 14:19:18 +0500 > "Eugene M. Zheganin" wrote: > >> Hi. >> >> I'm using FreeBSD 10.1-STABLE as an application server, last week I've >> noticed that disks are always busy while gstat shows that the activity >> measured in iops/reads/writes is low, form my point of view: >> > Hi. > > You have processes with STATE zio->i ? hard to tell now, since I've managed to solve the problem. If I encounter this again, with the state you mentioned, what could it mean ? Eugene. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: high disk %busy, while almost nothing happens
Hi. On 26.11.2015 14:19, Eugene M. Zheganin wrote: > Hi. > > I'm using FreeBSD 10.1-STABLE as an application server, last week I've > noticed that disks are always busy while gstat shows that the activity > measured in iops/reads/writes is low, form my point of view: > > > L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name > 8 56 50 520 160.6 6286 157.4 100.2 gpt/zfsroot0 > 8 56 51 1474 162.8 5228 174.4 99.9 gpt/zfsroot1 > > These %busy numbers arent't changing much, and from my point of view > both disks do very little. > The thing is, it was the compression. As soon as I cleared the gzip compression from busy datasets, %busy went down, almost to zero. Affected datasets were filled with poorly compressionable files, mostly archives or zlib-compressed data. And this is kind of counter-intuitive: one could think that worse-case scenario would be redundant CPU load, with constand disk i/o. In practice, otherwise, high disk %busy happens. Could someone explain that ? I only found this because of the flow-capture was starting like for years, and I started to suspect the compression setting. Eugene. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: high disk %busy, while almost nothing happens
On Thu, 26 Nov 2015 14:19:18 +0500 "Eugene M. Zheganin" wrote: > Hi. > > I'm using FreeBSD 10.1-STABLE as an application server, last week I've > noticed that disks are always busy while gstat shows that the activity > measured in iops/reads/writes is low, form my point of view: > Hi. You have processes with STATE zio->i ? ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: A recent 10.2-STABLE no longer builds on a no-exec /usr/src file system
Mark Martinec wrote on 11/26/2015 19:31: Up to about a week ago building world on FreeBSD 10.2-STABLE went just fine. Today after svn update the build fails: # make buildworld [...] CC='cc ' mkdep -f .depend.getprotoent_test -a -I/usr/src/lib/libc/tests/net -I/usr/src/lib/libnetbsd -I/usr/src/contrib/netbsd-tests -std=gnu99 /usr/src/contrib/netbsd-tests/lib/libc/net/t_getprotoent.c echo getprotoent_test: /usr/obj/usr/src/tmp/usr/lib/libc.a /usr/obj/usr/src/tmp/usr/lib/private/libatf-c.a >> .depend.getprotoent_test (cd /usr/src/lib/libc/tests/net && make -f /usr/src/lib/libc/tests/net/Makefile _RECURSING_PROGS= SUBDIR= PROG=ether_aton_test DEPENDFILE=.depend.ether_aton_test .MAKE.DEPENDFILE=.depend.ether_aton_test depend) /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr /usr/src/sys/net/if_ethersubr.c aton_ether_subr.c make[7]: exec(/usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr) failed (Permission denied) *** Error code 1 Stop. make[7]: stopped in /usr/src/lib/libc/tests/net *** Error code 1 It turns out that our file system /usr/src had an "exec" flag turned off, so now running a command: /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr fails with "Permission denied". It would be valuable if building a system on an exec-protected src file system would continue to be possible. Not sure if the /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr is the only such new command breaking the build. Anyway, a simple workaround is to run shell from a command line instead of as a shebang, i.e.: # /bin/sh /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr instead of: # /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr I was puzzled by similar thing years ago. I was using /var/db and /tmp mounted with noexec. And then there was some changes. Ports need /var/db with exec because of some script in /var/db/pkg and /tmp must have exec too for buildworld or installworld (I don't remember it well, now I always do mount -u -o current,exec /tmp before build + install world and kernel) Anyway - it would be better to not have these partitions mounted with exec. Miroslav Lachman ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: ZFS - poor performance with "large" directories
On Wed, 25 Nov 2015 16:24:48 +0100 Albert Cervin wrote: > Just to close this off, when using Samba with ZFS it seems to be very > important (if you have many files in a directory) to make it case > sensitive as per: > https://wiki.samba.org/index.php/Performance_tuning#Handling_Large_Directories. > > Everything is now roses and works as expected. > > Sorry ZFS that I accused you! ;) Thanks for telling us what the problem (and fix) was. -- Torfinn Ingolfsen ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
A recent 10.2-STABLE no longer builds on a no-exec /usr/src file system
Up to about a week ago building world on FreeBSD 10.2-STABLE went just fine. Today after svn update the build fails: # make buildworld [...] CC='cc ' mkdep -f .depend.getprotoent_test -a -I/usr/src/lib/libc/tests/net -I/usr/src/lib/libnetbsd -I/usr/src/contrib/netbsd-tests -std=gnu99 /usr/src/contrib/netbsd-tests/lib/libc/net/t_getprotoent.c echo getprotoent_test: /usr/obj/usr/src/tmp/usr/lib/libc.a /usr/obj/usr/src/tmp/usr/lib/private/libatf-c.a >> .depend.getprotoent_test (cd /usr/src/lib/libc/tests/net && make -f /usr/src/lib/libc/tests/net/Makefile _RECURSING_PROGS= SUBDIR= PROG=ether_aton_test DEPENDFILE=.depend.ether_aton_test .MAKE.DEPENDFILE=.depend.ether_aton_test depend) /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr /usr/src/sys/net/if_ethersubr.c aton_ether_subr.c make[7]: exec(/usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr) failed (Permission denied) *** Error code 1 Stop. make[7]: stopped in /usr/src/lib/libc/tests/net *** Error code 1 It turns out that our file system /usr/src had an "exec" flag turned off, so now running a command: /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr fails with "Permission denied". It would be valuable if building a system on an exec-protected src file system would continue to be possible. Not sure if the /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr is the only such new command breaking the build. Anyway, a simple workaround is to run shell from a command line instead of as a shebang, i.e.: # /bin/sh /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr instead of: # /usr/src/contrib/netbsd-tests/lib/libc/net/gen_ether_subr Mark ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
hyperv fails to build, when "nodevice ata" is set in kernel-config
All my drives here are using ahci and the currently-used 9.2-PRERELEASE kernel sees them thus: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 ada3 at ahcich4 bus 0 scbus4 target 0 lun 0 I'm trying to upgrade to 10.2 getting the following error from buildkernel working with the same kernel-config file as before: /usr/src/sys/dev/hyperv/stordisengage/hv_ata_pci_disengage.c:76:10: fatal error: 'ata_if.h' file not found #include ^ 1 error generated. mkdep: compile failed Please, advise. Thanks! Yours, -mi ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: ZFS - poor performance with "large" directories
On Thu, Nov 26, 2015 at 2:19 AM, krad wrote: > true, but in my experience usb pen drives are variable in terms of > performance across different sticks and different areas of the same stick. > This can complicate things a little, and is often not worth the effort. You > obviously run the ssd over usb though, and I still do on one server I run > as I haven't been able to sort the down time yet. > Nowadays, USB 3.x-based sticks in USB 3.x ports should be fast enough that they'll be helpful. You won't get the full 5 Gbps from one (unless you spend as much or more than an SSD), but it will be much better than the measly 0.5 Gbps of a USB 2.x stick/port. Don't bother trying with a USB 2.x stick, or with anything plugged into a USB 2.x port. Invariably, it will just slow things down. I used to use 8 GB USB2 sticks in USB2 ports for L2ARC (with a separate one for the root filesystem). When I had 4x IDE disks in a raidz1 vdev, they helped. When I migrated to 4x SATA1 disks in a raidz1 vdev, they helped. When I migrated to 4x SATA3 disks in dual-mirror vdevs (with root-on-ZFS), suddenly the USB stick became the bottleneck. Removing it actually made the whole system faster (better throughput, more IOps, lower latency, smoother system overall). As always, YMMV, and test it with your own setup. :) -- Freddie Cash fjwc...@gmail.com ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: ZFS - poor performance with "large" directories
true, but in my experience usb pen drives are variable in terms of performance across different sticks and different areas of the same stick. This can complicate things a little, and is often not worth the effort. You obviously run the ssd over usb though, and I still do on one server I run as I haven't been able to sort the down time yet. On 25 November 2015 at 12:16, Gerrit Kühn wrote: > On Wed, 25 Nov 2015 12:06:38 + krad wrote about Re: > ZFS - poor performance with "large" directories: > > K> consumer SSDs are cheap enough now not to bother with usb drives I would > K> imagine. > > Sure. I was just suggesting a USB drive as a quick way to check if this > might help at all. Most people have USB drives lying around, and they can > simply be plugged into any computer. A SSD (cheap or not) more likely > needs to be bought first, and the server box might need to be opened to > have it installed. > > > cu > Gerrit > ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
high disk %busy, while almost nothing happens
Hi. I'm using FreeBSD 10.1-STABLE as an application server, last week I've noticed that disks are always busy while gstat shows that the activity measured in iops/reads/writes is low, form my point of view: L(q) ops/sr/s kBps ms/rw/s kBps ms/w %busy Name 8 56 50 520 160.6 6286 157.4 100.2 gpt/zfsroot0 8 56 51 1474 162.8 5228 174.4 99.9 gpt/zfsroot1 These %busy numbers arent't changing much, and from my point of view both disks do very little. zpool iostat: [root@gw0:~]# zpool iostat 1 capacity operationsbandwidth poolalloc free read write read write -- - - - - - - zfsroot 270G 186G 90131 1,17M 1,38M -- - - - - - - zfsroot 270G 186G113 93 988K 418K -- - - - - - - zfsroot 270G 186G112 0 795K 93,8K -- - - - - - - zfsroot 270G 186G109 55 1,28M 226K -- - - - - - - zfsroot 270G 186G112116 1,36M 852K -- - - - - - - zfsroot 270G 186G105 47 1,44M 1,61M -- - - - - - - What can cause this ? Pool is fragmented indeed, but I have others server with comparable amount of fragmentation, and no signs of busyness while reads/writes are that low. # zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAGCAP DEDUP HEALTH ALTROOT zfsroot 456G 270G 186G -51%59% 1.00x ONLINE - Loader settings: vfs.root.mountfrom="zfs:zfsroot" vfs.zfs.arc_max="2048M" vfs.zfs.zio.use_uma=1 I've tried to play with vfs.zfs.zio.use_uma, but without any noticeable effect. I've also tried to add separate log devices - this didn't help either. Thanks. Eugene. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
