[zfs-discuss] Slow zfs writes
Hi, My OmniOS host is expreiencing slow zfs writes ( around 30 times slower ). iostat reports below error though pool is healthy. This is happening in past 4 days though no change was done to system. Is the hard disks faulty ? Please help. # zpool status -v root@host:~# zpool status -v pool: test state: ONLINE status: The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable. action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags. config: NAME STATE READ WRITE CKSUM test ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 c2t0d0 ONLINE 0 0 0 c2t1d0 ONLINE 0 0 0 c2t2d0 ONLINE 0 0 0 c2t3d0 ONLINE 0 0 0 c2t4d0 ONLINE 0 0 0 raidz1-1 ONLINE 0 0 0 c2t5d0 ONLINE 0 0 0 c2t6d0 ONLINE 0 0 0 c2t7d0 ONLINE 0 0 0 c2t8d0 ONLINE 0 0 0 c2t9d0 ONLINE 0 0 0 raidz1-3 ONLINE 0 0 0 c2t12d0 ONLINE 0 0 0 c2t13d0 ONLINE 0 0 0 c2t14d0 ONLINE 0 0 0 c2t15d0 ONLINE 0 0 0 c2t16d0 ONLINE 0 0 0 c2t17d0 ONLINE 0 0 0 c2t18d0 ONLINE 0 0 0 c2t19d0 ONLINE 0 0 0 c2t20d0 ONLINE 0 0 0 c2t21d0 ONLINE 0 0 0 c2t22d0 ONLINE 0 0 0 c2t23d0 ONLINE 0 0 0 raidz1-4 ONLINE 0 0 0 c2t24d0 ONLINE 0 0 0 c2t25d0 ONLINE 0 0 0 c2t26d0 ONLINE 0 0 0 c2t27d0 ONLINE 0 0 0 c2t28d0 ONLINE 0 0 0 c2t29d0 ONLINE 0 0 0 c2t30d0 ONLINE 0 0 0 raidz1-5 ONLINE 0 0 0 c2t31d0 ONLINE 0 0 0 c2t32d0 ONLINE 0 0 0 c2t33d0 ONLINE 0 0 0 c2t34d0 ONLINE 0 0 0 c2t35d0 ONLINE 0 0 0 c2t36d0 ONLINE 0 0 0 c2t37d0 ONLINE 0 0 0 raidz1-6 ONLINE 0 0 0 c2t38d0 ONLINE 0 0 0 c2t39d0 ONLINE 0 0 0 c2t40d0 ONLINE 0 0 0 c2t41d0 ONLINE 0 0 0 c2t42d0 ONLINE 0 0 0 c2t43d0 ONLINE 0 0 0 c2t44d0 ONLINE 0 0 0 spares c5t10d0AVAIL c5t11d0AVAIL c2t45d0AVAIL c2t46d0AVAIL c2t47d0AVAIL # iostat -En c4t0d0 Soft Errors: 0 Hard Errors: 5 Transport Errors: 0 Vendor: iDRACProduct: Virtual CD Revision: 0323 Serial No: Size: 0.00GB 0 bytes Media Error: 0 Device Not Ready: 5 No Device: 0 Recoverable: 0 Illegal Request: 1 Predictive Failure Analysis: 0 c3t0d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: iDRACProduct: LCDRIVE Revision: 0323 Serial No: Size: 0.00GB 0 bytes Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 Illegal Request: 0 Predictive Failure Analysis: 0 c4t0d1 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 Vendor: iDRACProduct: Virtual Floppy Revision: 0323 Serial No: Size: 0.00GB 0 bytes Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 root@host:~# fmadm faulty --- -- - TIMEEVENT-ID MSG-ID SEVERITY --- -- - Jan 05 08:21:09 7af1ab3c-83c2-602d-d4b9-f9040db6944a ZFS-8000-HC Major Host: host Platform: PowerEdge-R810 Product_sn : Fault class : fault.fs.zfs.io_failure_wait Affects : zfs://pool=test faulted but still in service Problem in : zfs://pool=test faulted but still in service Description : The ZFS pool has experienced currently unrecoverable I/O failures. Refer to http://illumos.org/msg/ZFS-8000-HCfor more information. Response: No automated response will be taken. Impact : Read and write I/Os cannot be serviced. Action : Make sure the affected devices are connected, then run 'zpool clear'. Regards, Ram
[zfs-discuss] ZFS monitoring
Hello, I'n updating Devilator, the performance data collector for Orca and FreeBSD to include ZFS monitoring. So far I am graphing the ARC and L2ARC size, L2ARC writes and reads, and several hit/misses data pairs. Any suggestions to improve it? What other variables can be interesting? An example of the current state of the program is here: http://devilator.frobula.com Thanks, Borja. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS monitoring
On Mon, Feb 11, 2013 at 9:53 AM, Borja Marcos bor...@sarenet.es wrote: Hello, I'n updating Devilator, the performance data collector for Orca and FreeBSD to include ZFS monitoring. So far I am graphing the ARC and L2ARC size, L2ARC writes and reads, and several hit/misses data pairs. Any suggestions to improve it? What other variables can be interesting? An example of the current state of the program is here: http://devilator.frobula.com Thanks, Borja. The zpool iostat output has all sorts of statistics I think would be useful/interesting to record over time. --Tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS monitoring
On Feb 11, 2013, at 4:56 PM, Tim Cook wrote: The zpool iostat output has all sorts of statistics I think would be useful/interesting to record over time. Yes, thanks :) I think I will add them, I just started with the esoteric ones. Anyway, still there's no better way to read it than running zpool iostat and parsing the output, right? Borja. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS monitoring
On 02/11/2013 04:53 PM, Borja Marcos wrote: Hello, I'n updating Devilator, the performance data collector for Orca and FreeBSD to include ZFS monitoring. So far I am graphing the ARC and L2ARC size, L2ARC writes and reads, and several hit/misses data pairs. Any suggestions to improve it? What other variables can be interesting? An example of the current state of the program is here: http://devilator.frobula.com Hi Borja, I've got one thing up for review in Illumos for upstreaming: #3137 L2ARC Compression. This adds another kstat called l2_asize, which tells you how big the L2ARC actually is, taking into account compression, e.g. # kstat -n arcstats 5 | egrep '(\l2_size\|\l2_asize\)' l2_asize25708032 l2_size 29117952 You can use this to track L2ARC compression efficiency, etc. If the kstat is not present, then L2ARC compression isn't available. Anyway, just a quick thought. -- Saso ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS monitoring
On 2013-02-11 17:14, Borja Marcos wrote: On Feb 11, 2013, at 4:56 PM, Tim Cook wrote: The zpool iostat output has all sorts of statistics I think would be useful/interesting to record over time. Yes, thanks :) I think I will add them, I just started with the esoteric ones. Anyway, still there's no better way to read it than running zpool iostat and parsing the output, right? I believe, in this case you'd have to run it as a continuous process and parse the outputs after the first one (overall uptime stat, IIRC). Also note that on problems with ZFS engine itself, zpool may lock up and thus halt your program - so have it ready to abort an outstanding statistics read after a timeout and perhaps log an error. And if pools are imported-exported during work, the zpool iostat output changes dynamically, so you basically need to parse its text structure every time. The zpool iostat -v might be even more interesting though, as it lets you see per-vdev statistics and perhaps notice imbalances, etc... All that said, I don't know if this data isn't also available as some set of kstats - that would probably be a lot better for your cause. Inspect the zpool source to see where it gets its numbers from... and perhaps make and RTI relevant kstats, if they aren't yet there ;) On the other hand, I am not certain how Solaris-based kstats interact or correspond to structures in FreeBSD (or Linux for that matter)?.. HTH, //Jim Klimov ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Slow zfs writes
root@host:~# fmadm faulty --- -- - TIME EVENT-ID MSG-ID SEVERITY --- -- - Jan 05 08:21:09 7af1ab3c-83c2-602d-d4b9-f9040db6944a ZFS-8000-HC Major Host : host Platform : PowerEdge-R810 Product_sn : Fault class : fault.fs.zfs.io_failure_wait Affects : zfs://pool=test faulted but still in service Problem in : zfs://pool=test faulted but still in service Description : The ZFS pool has experienced currently unrecoverable I/O failures. Refer to http://illumos.org/msg/ZFS-8000-HC for more information. Response : No automated response will be taken. Impact : Read and write I/Os cannot be serviced. Action : Make sure the affected devices are connected, then run 'zpool clear'. -- The pool looks healthy to me, but it it isn't very well balanced. Have you been adding new VDEVs on your way to grow it? Check if of the VDEVs are fuller than others. I don't have an OI/IllumOS system available ATM, but IIRC this can be done with iostat -v. Older versions of ZFS striped to all VDEVs regardless to fill, which slowed down the write speeds rather horribly if some VDEVs were full (90%). This shouldn't be the case with OmniOS, but it *may* be the case with an old zpool version. I don't know. I'd check fill rate of the VDEVs first, then perhaps try to upgrade the zpool unless you have to be able to mount it on an older version of zpool (on S10 or similar). Vennlige hilsener / Best regards roy -- Roy Sigurd Karlsbakk (+47) 98013356 r...@karlsbakk.net http://blogg.karlsbakk.net/ GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt -- I all pedagogikk er det essensielt at pensum presenteres intelligibelt. Det er et elementært imperativ for alle pedagoger å unngå eksessiv anvendelse av idiomer med xenotyp etymologi. I de fleste tilfeller eksisterer adekvate og relevante synonymer på norsk.___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Slow zfs writes
Hi Roy, You are right. So it looks like re-distribution issue. Initially there were two Vdev with 24 disks ( disk 0-23 ) for close to year. After which which we added 24 more disks and created additional vdevs. The initial vdevs are filled up and so write speed declined. Now how to find files that are present in a Vdev or a disk. That way I can remove and re-copy back to distribute data. Any other way to solve this ? Total capacity of pool - 98Tb Used - 44Tb Free - 54 Tb root@host:# zpool iostat -v capacity operationsbandwidth pool alloc free read write read write --- - - - - - - test 54.0T 62.7T 52 1.12K 2.16M 5.78M raidz1 11.2T 2.41T 13 30 176K 146K c2t0d0 - - 5 18 42.1K 39.0K c2t1d0 - - 5 18 42.2K 39.0K c2t2d0 - - 5 18 42.5K 39.0K c2t3d0 - - 5 18 42.9K 39.0K c2t4d0 - - 5 18 42.6K 39.0K raidz1 13.3T 308G 13100 213K 521K c2t5d0 - - 5 94 50.8K 135K c2t6d0 - - 5 94 51.0K 135K c2t7d0 - - 5 94 50.8K 135K c2t8d0 - - 5 94 51.1K 135K c2t9d0 - - 5 94 51.1K 135K raidz1 13.4T 19.1T 9455 743K 2.31M c2t12d0 - - 3137 69.6K 235K c2t13d0 - - 3129 69.4K 227K c2t14d0 - - 3139 69.6K 235K c2t15d0 - - 3131 69.6K 227K c2t16d0 - - 3141 69.6K 235K c2t17d0 - - 3132 69.5K 227K c2t18d0 - - 3142 69.6K 235K c2t19d0 - - 3133 69.6K 227K c2t20d0 - - 3143 69.6K 235K c2t21d0 - - 3133 69.5K 227K c2t22d0 - - 3143 69.6K 235K c2t23d0 - - 3133 69.5K 227K raidz1 2.44T 16.6T 5103 327K 485K c2t24d0 - - 2 48 50.8K 87.4K c2t25d0 - - 2 49 50.7K 87.4K c2t26d0 - - 2 49 50.8K 87.3K c2t27d0 - - 2 49 50.8K 87.3K c2t28d0 - - 2 49 50.8K 87.3K c2t29d0 - - 2 49 50.8K 87.3K c2t30d0 - - 2 49 50.8K 87.3K raidz1 8.18T 10.8T 5295 374K 1.54M c2t31d0 - - 2131 58.2K 279K c2t32d0 - - 2131 58.1K 279K c2t33d0 - - 2131 58.2K 279K c2t34d0 - - 2132 58.2K 279K c2t35d0 - - 2132 58.1K 279K c2t36d0 - - 2133 58.3K 279K c2t37d0 - - 2133 58.2K 279K raidz1 5.42T 13.6T 5163 383K 823K c2t38d0 - - 2 61 59.4K 146K c2t39d0 - - 2 61 59.3K 146K c2t40d0 - - 2 61 59.4K 146K c2t41d0 - - 2 61 59.4K 146K c2t42d0 - - 2 61 59.3K 146K c2t43d0 - - 2 62 59.2K 146K c2t44d0 - - 2 62 59.3K 146K On Mon, Feb 11, 2013 at 10:23 PM, Roy Sigurd Karlsbakk r...@karlsbakk.netwrote: root@host:~# fmadm faulty --- -- - TIMEEVENT-ID MSG-ID SEVERITY --- -- - Jan 05 08:21:09 7af1ab3c-83c2-602d-d4b9-f9040db6944a ZFS-8000-HC Major Host: host Platform: PowerEdge-R810 Product_sn : Fault class : fault.fs.zfs.io_failure_wait Affects : zfs://pool=test faulted but still in service Problem in : zfs://pool=test faulted but still in service Description : The ZFS pool has experienced currently unrecoverable I/O failures. Refer to http://illumos.org/msg/ZFS-8000-HCfor more information. Response: No automated response will be taken. Impact : Read and write I/Os cannot be serviced. Action : Make sure the affected devices are connected, then run 'zpool clear'. -- The pool looks healthy to me, but it it isn't very well balanced. Have you been adding new VDEVs on your way to grow it? Check if of the VDEVs are fuller than others. I don't have an OI/IllumOS system available ATM, but IIRC this can be done with iostat -v. Older versions of ZFS striped to all VDEVs regardless to fill, which slowed down the write speeds rather horribly if some VDEVs were full (90%). This shouldn't be the case with OmniOS, but it *may* be the case with an old zpool version. I don't know. I'd check fill
[zfs-discuss] Bp rewrite
Hi, Anyone knows if there is any progress on bp_rewrite ? Its much awaited to solve re-distribution issue, and moving vdevs. Regards, Ram ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
