Comment at end...
Mattias Pantzare wrote:
On Wed, Sep 8, 2010 at 15:27, Edward Ned Harvey <sh...@nedharvey.com> wrote:
From: pantz...@gmail.com [mailto:pantz...@gmail.com] On Behalf Of
Mattias Pantzare
It
is about 1 vdev with 12 disk or 2 vdev with 6 disks. If you have 2
vdev you have to read half the data compared to 1 vdev to resilver a
disk.
Let's suppose you have 1T of data. You have 12-disk raidz2. So you have
approx 100G on each disk, and you replace one disk. Then 11 disks will each
read 100G, and the new disk will write 100G.
Let's suppose you have 1T of data. You have 2 vdev's that are each 6-disk
raidz1. Then we'll estimate 500G is on each vdev, so each disk has approx
100G. You replace a disk. Then 5 disks will each read 100G, and 1 disk
will write 100G.
Both of the above situations resilver in equal time, unless there is a bus
bottleneck. 21 disks in a single raidz3 will resilver just as fast as 7
disks in a raidz1, as long as you are avoiding the bus bottleneck. But 21
disks in a single raidz3 provides better redundancy than 3 vdev's each
containing a 7 disk raidz1.
In my personal experience, approx 5 disks can max out approx 1 bus. (It
actually ranges from 2 to 7 disks, if you have an imbalance of cheap disks
on a good bus, or good disks on a crap bus, but generally speaking people
don't do that. Generally people get a good bus for good disks, and cheap
disks for crap bus, so approx 5 disks max out approx 1 bus.)
In my personal experience, servers are generally built with a separate bus
for approx every 5-7 disk slots. So what it really comes down to is ...
Instead of the Best Practices Guide saying "Don't put more than ___ disks
into a single vdev," the BPG should say "Avoid the bus bandwidth bottleneck
by constructing your vdev's using physical disks which are distributed
across multiple buses, as necessary per the speed of your disks and buses."
This is assuming that you have no other IO besides the scrub.
You should of course keep the number of disks in a vdev low for
general performance reasons unless you only have linear reads (as your
IOPS will be close to what only one disk can give for the whole vdev).
There is another optimization in the Best Practices Guide that says the
number of devices in a vdev should be (N+P) with P = 1 (raidz), 2
(raidz2), or 3 (raidz3) and N equals 2, 4, or 8.
I.e. 2^N + P where N is 1, 2, or 3 and P is the RAIDZ level.
I.e. Optimal sizes
RAIDZ1 vdevs should have 3, 5, or 9 devices in each vdev
RAIDZ2 vdevs should have 4, 6, or 10 devices in each vdev
RAIDZ3 vdevs should have 5, 7, or 11 devices in each vdev
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
