On Fri, 26 Jul 2013 14:18:44 +1000
Craig Sanders <[email protected]> wrote:
> On Fri, Jul 26, 2013 at 01:00:30PM +1000, Russell Coker wrote:
> > > also the numbers in the READ WRITE and CKSUM columns will show you
> > > the number of errors detected for each drive.
> >
> > However those numbers are all 0 for me.
>
> as i said, i interpret that as indicating that there's no real problem
> with the drive - unless the kernel is retrying successfully before zfs
> notices the drive is having problems? is that the case?
No, the very first message in this thread included the zpool status
output which stated that 1.4M of data had been regenerated.
> > I'm now replacing the defective disk. I've attached a sample of
> > iostat output, it seems to be reading from all disks and then
> > reconstructing the parity for the new disk which is surprising, I
> > had expected it to just read the old disk and write to the new disk
>
> there's (at least) two reasons for that.
>
> first is that raidz is only similar to raid5/6 but not exactly the
> same. the redundant copies of a data block can exist anywhere on any
> of the drives in the vdev, so it's not just a straight dd-style copy
> from the old drive to the new.
>
> the second is that when you're replacing a drive, the old one may not
> be reliable or trustworthy, or may even be absent from the system.
zpool replace tank \
sdd /dev/disk/by-id/ata-ST4000DM000-1F2168_Z300MHWF-part2
In this case the old disk was online, I ran the above replace command
so ZFS should know that the new disk needs to be an exact copy of the
old.
> > but instead I get a scrub as well as the "resilver".
>
> that's odd. what makes you say that?
I've attached the zpool status output. It shows the disk as being
replaced but is accessing all disks according to the iostat output I
attached previously.
> and, of course, with raidz (or raid5) writes are always going to be
> limited to the speed of, at best, a single drive.
Actually for contiguous writes a RAID-5 array can be expected to
exceed the performance of a single disk. It's not difficult to
demonstrate this in real life.
> the SATA controller is also a factor, many (most?) aren't capable
> of running four or more drives at full speed simultaneously. even a
> cheap-but-midrange SAS card like my LSI cards couldn't run all 8 ports
> at full speed with 6Gbps SSDs flat out (since i'm only running hard
> disks and not SSDs on them, i will never be limited by that so don't
> care)
Yes, that's always been an issue, dating back to IDE days.
> > avg-cpu: %user %nice %system %iowait %steal %idle
> > 1.69 0.00 20.23 3.79 0.00 74.29
> >
> > Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s
> > avgrq-sz avgqu-sz await svctm %util sda 373.90
> > 0.40 298.30 5.80 92344.00 75.20 303.91 1.36 4.48
> > 2.23 67.96 sdb 195.90 0.40 502.70 5.80
> > 89902.40 75.20 176.95 1.66 3.27 1.25 63.72
> > sdc 374.20 0.60 300.30 6.00 92286.40 76.80
> > 301.54 1.41 4.59 2.38 72.84 sdd 175.10
> > 0.60 539.30 6.00 89230.40 76.80 163.78 1.78 3.27
> > 1.24 67.76 sdl 0.00 174.30 0.00 681.10
> > 0.00 88107.10 129.36 6.40 9.39 1.32 89.72
>
> hmm. does iostat know about 4K sectors yet? maybe try that with -m for
> megabytes/sec rather than rsec/s.
>
> also, what does 'zpool iostat' (or 'zpool iostat -v') and 'zpool
> status' say?
I've attached the zpool iostat output and it claims that the only real
activity is reading from the old disk at 38MB/s and writing to the new
disk at the same speed. I've attached the iostat -m output which shows
that all disks are being accessed at a speed just over 45MB/s. I
guess that the difference between 38 and 45 would be due to some random
variation, zpool gives an instant response based on past data while
iostat runs in real time and gives more current data.
--
My Main Blog http://etbe.coker.com.au/
My Documents Blog http://doc.coker.com.au/
# zpool status
pool: tank
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jul 26 12:47:30 2013
905G scanned out of 10.3T at 160M/s, 17h4m to go
215G resilvered, 8.59% done
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sda ONLINE 0 0 0
sdb ONLINE 0 0 0
sdc ONLINE 0 0 0
replacing-3 ONLINE 0 0 17
sdd ONLINE 0 0 0
(resilvering)
ata-ST4000DM000-1F2168_Z300MHWF-part2 ONLINE 0 0 0
(resilvering)
# zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------- ----- ----- ----- ----- ----- -----
tank 10.3T 605G 54 8 5.69M 419K
raidz1 10.3T 605G 54 8 5.69M 419K
sda - - 28 2 1.96M 150K
sdb - - 27 2 1.91M 149K
sdc - - 27 3 1.96M 150K
replacing - - 1.05K 95 38.4M 386K
sdd - - 25 2 1.92M 149K
ata-ST4000DM000-1F2168_Z300MHWF-part2 - - 0 861 22
38.7M
avg-cpu: %user %nice %system %iowait %steal %idle
2.10 0.00 23.63 6.39 0.00 67.88
Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz
avgqu-sz await svctm %util
sda 271.70 0.50 383.00 6.00 46.49 0.04 244.94
3.19 8.20 2.02 78.52
sdb 146.80 0.60 514.20 5.80 45.14 0.04 177.95
2.81 5.40 1.38 71.64
sdc 300.50 0.50 369.10 5.90 46.31 0.04 253.09
2.73 7.27 2.09 78.44
sdd 153.60 0.50 505.60 5.90 45.17 0.04 180.99
2.82 5.52 1.46 74.44
sdl 0.00 199.70 0.00 733.80 0.00 44.39 123.90
3.76 5.13 1.10 80.44
_______________________________________________
luv-main mailing list
[email protected]
http://lists.luv.asn.au/listinfo/luv-main
