On Fri, Jul 26, 2013 at 01:00:30PM +1000, Russell Coker wrote: > > also the numbers in the READ WRITE and CKSUM columns will show you > > the number of errors detected for each drive. > > However those numbers are all 0 for me.
as i said, i interpret that as indicating that there's no real problem with the drive - unless the kernel is retrying successfully before zfs notices the drive is having problems? is that the case? > I'm now replacing the defective disk. I've attached a sample of iostat > output, it seems to be reading from all disks and then reconstructing > the parity for the new disk which is surprising, I had expected it to > just read the old disk and write to the new disk there's (at least) two reasons for that. first is that raidz is only similar to raid5/6 but not exactly the same. the redundant copies of a data block can exist anywhere on any of the drives in the vdev, so it's not just a straight dd-style copy from the old drive to the new. the second is that when you're replacing a drive, the old one may not be reliable or trustworthy, or may even be absent from the system. also note that with a raidz vdev you get the IOPS of a single drive - this is why for large pools it is better to have multiple smaller raidz vdevs than one large vdev (e.g. 3 x 5-drive raidz vdevs rather than 1 x 15-drive raidz vdev). > but instead I get a scrub as well as the "resilver". that's odd. what makes you say that? > So the rate of rebuild is considerably less than half what I had hoped > for, I had expected something like 130MB/s for contiguous reads and > writes and instead each of the 5 disks is doing about 45MB/s. which is about 180MB/s total read. btw, from what I understand of the way it works, ZFS avoids the performance penalty on raid writes by always writing the entire (variable-width) stripe. this can result in writes being faster than reads in some cases, and is the main reason for the recommendation to use 2, 4, 8 (a power of two) number of DATA disks plus however many "parity" disks (1 for raidz1, 2 for raidz2 etc) in a raidz pool. and, of course, with raidz (or raid5) writes are always going to be limited to the speed of, at best, a single drive. the SATA controller is also a factor, many (most?) aren't capable of running four or more drives at full speed simultaneously. even a cheap-but-midrange SAS card like my LSI cards couldn't run all 8 ports at full speed with 6Gbps SSDs flat out (since i'm only running hard disks and not SSDs on them, i will never be limited by that so don't care) > avg-cpu: %user %nice %system %iowait %steal %idle > 1.69 0.00 20.23 3.79 0.00 74.29 > > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz > avgqu-sz await svctm %util > sda 373.90 0.40 298.30 5.80 92344.00 75.20 303.91 > 1.36 4.48 2.23 67.96 > sdb 195.90 0.40 502.70 5.80 89902.40 75.20 176.95 > 1.66 3.27 1.25 63.72 > sdc 374.20 0.60 300.30 6.00 92286.40 76.80 301.54 > 1.41 4.59 2.38 72.84 > sdd 175.10 0.60 539.30 6.00 89230.40 76.80 163.78 > 1.78 3.27 1.24 67.76 > sdl 0.00 174.30 0.00 681.10 0.00 88107.10 129.36 > 6.40 9.39 1.32 89.72 hmm. does iostat know about 4K sectors yet? maybe try that with -m for megabytes/sec rather than rsec/s. also, what does 'zpool iostat' (or 'zpool iostat -v') and 'zpool status' say? craig -- craig sanders <[email protected]> _______________________________________________ luv-main mailing list [email protected] http://lists.luv.asn.au/listinfo/luv-main
