Re: [zfs-discuss] [storage-discuss] Disk Issues
Thanks everyone who has tried to help. this has gotten a bit crazier, I removed the 'faulty' drive and let the pool run in degraded mode. It would appear that now another drive has decided to play up; de-bash-4.0# zpool status pool: data state: DEGRADED status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scrub: resilver completed after 2h35m with 0 errors on Wed Feb 17 13:48:16 2010 config: NAMESTATE READ WRITE CKSUM dataDEGRADED 0 0 0 raidz2-0 DEGRADED 0 0 0 c6t0d0 ONLINE 0 0 0 c6t1d0 ONLINE 0 0 0 c6t2d0 ONLINE 0 0 0 c6t3d0 ONLINE 0 0 0 c6t4d0 OFFLINE 0 0 0 366G resilvered c6t5d0 ONLINE 0 0 0 errors: No known data errors Now I'm transferring some data to the pool vicer/sw/s Mr/s Mw/s wait actv svc_t %w %b s/w h/w trn tot sd0 0.00.00.00.0 0.0 0.00.0 0 0 0 0 0 0 sd2 18.50.02.10.0 0.2 0.0 13.5 3 4 0 0 0 0 sd3 18.50.02.10.0 0.2 0.0 14.8 4 5 0 0 0 0 sd4 1.00.00.00.0 9.0 1.0 .9 100 100 0 0 0 0 sd5 19.50.02.10.0 0.2 0.0 11.9 3 4 0 0 0 0 sd6 0.00.00.00.0 0.0 0.00.0 0 0 0 0 0 0 sd7 18.50.02.10.0 0.3 0.1 22.7 8 8 0 0 0 0 sd8 0.00.00.00.0 0.0 0.00.0 0 0 0 24 24 48 sd9 0.00.00.00.0 0.0 0.00.0 0 0 0 24 24 48 extended device statistics errors --- devicer/sw/s Mr/s Mw/s wait actv svc_t %w %b s/w h/w trn tot sd0 0.00.00.00.0 0.0 0.00.0 0 0 0 0 0 0 sd2 0.50.00.00.0 0.0 0.0 15.6 0 1 0 0 0 0 sd3 0.50.00.00.0 0.0 0.0 33.5 0 2 0 0 0 0 sd4 0.50.00.00.0 9.0 1.0 1.9 100 100 0 0 0 0 sd5 0.50.00.00.0 0.0 0.0 21.4 0 1 0 0 0 0 sd6 0.00.00.00.0 0.0 0.00.0 0 0 0 0 0 0 sd7 0.00.00.00.0 0.0 0.00.0 0 0 0 0 0 0 sd8 0.00.00.00.0 0.0 0.00.0 0 0 0 24 24 48 sd9 0.00.00.00.0 0.0 0.00.0 0 0 0 24 24 48 extended device statistics errors --- devicer/sw/s Mr/s Mw/s wait actv svc_t %w %b s/w h/w trn tot sd0 0.00.00.00.0 0.0 0.00.0 0 0 0 0 0 0 sd2 0.50.50.00.0 0.0 0.05.9 0 1 0 0 0 0 sd3 0.50.50.00.0 0.0 0.0 10.3 0 1 0 0 0 0 sd4 0.50.00.00.0 9.0 1.0 1.8 100 100 0 0 0 0 sd5 0.50.50.00.0 0.0 0.0 11.1 0 1 0 0 0 0 sd6 0.00.00.00.0 0.0 0.00.0 0 0 0 0 0 0 sd7 0.50.50.00.0 0.0 0.08.2 0 1 0 0 0 0 sd8 0.00.00.00.0 0.0 0.00.0 0 0 0 24 24 48 sd9 0.00.00.00.0 0.0 0.00.0 0 0 0 24 24 48 extended device statistics errors --- devicer/sw/s Mr/s Mw/s wait actv svc_t %w %b s/w h/w trn tot sd0 0.50.00.00.0 0.0 0.01.7 0 0 0 0 0 0 sd2 6.5 16.00.00.7 0.3 0.1 15.5 5 6 0 0 0 0 sd3 6.07.50.00.7 0.4 0.1 33.8 8 8 0 0 0 0 sd4 0.50.00.00.0 9.0 1.0 1.9 100 100 0 0 0 0 sd5 5.5 17.50.00.7 0.2 0.09.9 4 5 0 0 0 0 sd6 0.00.00.00.0 0.0 0.00.0 0 0 0 0 0 0 sd7 6.5 17.50.00.7 0.4 0.1 18.0 6 6 0 0 0 0 sd8 0.00.00.00.0 0.0 0.00.0 0 0 0 24 24 48 sd9 0.00.00.00.0 0.0 0.00.0 0 0 0 24 24 48 extended device statistics errors --- devicer/sw/s Mr/s Mw/s wait actv svc_t %w %b s/w h/w trn tot sd0 0.00.00.00.0 0.0 0.00.0 0 0 0 0 0 0 sd2 2.00.00.10.0 0.0 0.0 16.4 1 2 0 0 0 0 sd3 2.00.00.10.0 0.0 0.0 29.4 1 3 0 0 0 0 sd4 1.00.00.00.0 9.0 1.0 .9 100 100 0 0 0 0 sd5 2.00.00.10.0 0.0 0.0 28.4 1 4 0 0 0 0 sd6
Re: [zfs-discuss] Disk Issues
Ok, I changed the cable and also tried swapping the port on the motherboard. The drive continued to have huge asvc_t and also started to have huge wsvc_t. I unplugged it and the 'pool' is now operating as per expected performance wise. See the 'storage' forum for any further updates as I am now convinced this has nothing to do with ZFS or my attempt to disable the ZIL. 8-) -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Disk Issues
While not strictly a ZFS issue as such I thought I'd post here as this and the storage forums are my best bet in terms of getting some help. I have a machine that I recently set up with b130, b131 and b132. With each build I have been playing around with ZFS raidz2 and mirroring to do a little performance testing. This is a 6 SATA port ICH10 Intel motherboard, running in AHCI mode. OS is on a USB flash. Suffice to say I have noticed that 1 particular drive out of 6 seems to have very high asvc_t practically all the time. This is an excerpt from 'iostat -xnM c6t2d0 2'; [i] r/sw/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device 70.5 502.00.04.1 0.0 1.30.02.2 0 54 c6t2d0 50.5 137.50.03.0 0.0 0.70.03.9 0 47 c6t2d0 71.0 163.50.04.8 0.0 0.80.03.4 0 61 c6t2d0 13.5 29.50.01.0 0.0 2.60.0 61.4 0 88 c6t2d0 1.00.50.00.0 0.0 3.60.0 2406.2 0 100 c6t2d0 1.01.00.00.0 0.0 4.00.0 1993.4 0 100 c6t2d0 1.01.50.00.0 0.0 4.00.0 1593.8 0 100 c6t2d0 2.03.00.00.1 0.0 4.00.0 791.6 0 100 c6t2d0 1.02.00.00.1 0.0 4.00.0 1320.3 0 100 c6t2d0 1.05.00.00.3 0.0 3.60.0 595.1 0 100 c6t2d0 [/i] and here is the drive shown with the other in the raidz2 pool extended device statistics r/sw/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device 0.01.50.00.0 0.0 0.00.00.5 0 0 c6t0d0 0.01.50.00.0 0.0 0.00.00.3 0 0 c6t1d0 1.01.00.00.0 0.0 4.00.0 1994.8 0 100 c6t2d0 1.01.50.00.0 0.0 0.00.05.2 0 1 c6t3d0 1.01.50.00.0 0.0 0.00.06.9 0 1 c6t4d0 1.01.50.00.0 0.0 0.00.0 10.1 0 2 c6t5d0 extended device statistics r/sw/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device 1.05.50.00.2 0.0 0.00.01.6 0 1 c6t0d0 1.05.50.00.2 0.0 0.00.01.5 0 1 c6t1d0 2.03.50.00.1 0.0 4.00.0 721.8 0 100 c6t2d0 1.05.50.00.2 0.0 0.00.01.9 0 1 c6t3d0 1.05.50.00.2 0.0 0.00.01.6 0 1 c6t4d0 2.05.50.00.2 0.0 0.00.03.1 0 2 c6t5d0 extended device statistics r/sw/s Mr/s Mw/s wait actv wsvc_t asvc_t %w %b device 0.03.50.00.1 0.0 0.00.00.4 0 0 c6t0d0 0.03.50.00.1 0.0 0.00.01.8 0 0 c6t1d0 1.02.00.00.1 0.0 4.00.0 1327.1 0 100 c6t2d0 1.03.50.00.1 0.0 0.00.04.9 0 1 c6t3d0 1.03.50.00.1 0.0 0.00.03.9 0 1 c6t4d0 1.03.50.00.1 0.0 0.00.02.0 0 1 c6t5d0 I have seen asvc_t as high as 2. There do not appear to be any errors hardware wise as 'iostat -e' shows [i] errors --- device s/w h/w trn tot sd0 0 0 0 0 sd2 0 0 0 0 sd3 0 0 0 0 sd4 0 0 0 0 sd5 0 0 0 0 sd6 0 0 0 0 sd7 0 0 0 0 [/i] 'zpool iostat -v 2' pauses for anywhere between 3 and 10 seconds before it prints the stats for that particular drive in the pool; capacity operationsbandwidth poolalloc free read write read write -- - - - - - - data 185G 5.26T 3115 8.96K 2.48M raidz2 185G 5.26T 3115 8.96K 2.48M c6t0d0 - - 2 26 2.70K 643K c6t1d0 - - 2 26 2.49K 643K * INSERT ~10 SECOND PAUSE* c6t2d0 - - 2 24 2.81K 643K c6t3d0 - - 2 26 2.75K 643K c6t4d0 - - 2 26 2.45K 643K c6t5d0 - - 2 26 2.71K 643K -- - - - - - - rpool 3.50G 3.94G 0 0 9.99K 1010 c5t0d0s0 3.50G 3.94G 0 0 9.99K 1010 -- - - - - - - swpool 102K 3.69G 0 0 19 0 /dev/rdsk/c7t0d0 102K 3.69G 0 0 19 0 -- - - - - - - I have booted up a linux rescue CD that has S.M.A.R.T support (system rescue CD) and performed the 'long' test on each drive. All drives pass the 'test'. There also appears to be no system errors with the drives under linux either. Can anyone shed any light on this issue, or suggest what I could try next ? I am sort of discounting hardware problems given that I do not see errors from the live linux CD. Maybe I
Re: [zfs-discuss] Disk Issues
I'd say your easiest two options are swap ports and see if the problem follows the drive. If it does, swap the drive out. --Tim ___ Yep, that sounds like a plan. Thanks for your suggestion. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Separate Zil on HDD ?
Hi all, I have a home server based on SNV_127 with 8 disks; 2 x 500GB mirrored root pool 6 x 1TB raidz2 data pool This server performs a few functions; NFS : for several 'lab' ESX virtual machines NFS : mythtv storage (videos, music, recordings etc) Samba : for home directories for all networked PCs I backup the important data to external USB hdd each day. I previously had a linux NFS server that I had mounted 'ASYNC' and, as one would expect, NFS performance was pretty good getting close to 900gb/s. Now that I have moved to opensolaris, NFS performance is not very good, I'm guessing mainly due to the 'SYNC' nature of NFS. I've seen various threads and most point at 2 options; 1. Disable the ZIL 2. Add independent log device/s I happen to have 2 x 250GB Western Digital RE3 7200rpm (Raid edition, rated for 24x7 usage etc) hard drives sitting doing nothing and was wondering whether it might speed up NFS, and possibly general filesystem usage, by adding these devices as log devices to the data pool. I understand that an SSD is considered ideal for log devices but I'm thinking that these 2 drives should at least be better than having the ZIL 'inside' the zpool. If adding these devices, should I add them as mirrored or individual to get some sort of load balancing (according to zpool manpage) and perhaps a little bit more performance ? I'm running ZFS version 19 which 'zpool upgrade -v' shows me as having 'log device removal' support. Can I easily remove these devices if I find that they have resulted in little/no performance improvements ? Any help/tips greatly appreciated. Cheers. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Backing up ZVOLs
Hello all, Are there any best practices / recommendations for ways of doing this ? In this case the ZVOLs would be iSCSI LUNS containing ESX VMs .I am aware of the of the need for the VMs to be quiesced for the backups to be useful. Cheers. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Backing up ZVOLs
Thanks for the help. I was curious whether the zfs send|receive was considered suitable given a few things I've read which said somethings along the lines of don't count on being able to restore this stuff. Ideally that is what I would use with the 'incremental' option so as to only backup changed blocks on subsequent backups. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
