Hi,
Thanks for the quick reply. Now that you have mentioned , we have a
different issue. What is the advantage of using spare disks instead of
including them in the raid-z array? If the system pool is on mirrored disks,
I think that this would be enough (hopefully). When one disk fails, isn't
it better to have a spare disk on hold, instead of one more disk in the
raid-z and no spares(or just a few)? or, rephrased, is it safer and faster
to replace a disk in a raid-z3 and restore the data from the other disks, or
to have a raid-z2 with a spare disk?
Thank you,
On Mon, Nov 29, 2010 at 6:03 AM, Erik Trimble erik.trim...@oracle.comwrote:
On 11/28/2010 1:51 PM, Paul Piscuc wrote:
Hi,
We are a company that want to replace our current storage layout with one
that uses ZFS. We have been testing it for a month now, and everything looks
promising. One element that we cannot determine is the optimum number of
disks in a raid-z pool. In the ZFS best practice guide, 7,9 and 11 disks are
recommended to be used in a single raid-z2. On the other hand, another user
specifies that the most important part is the distribution of the defaul
128k record size to all the disks. So, the recommended layout would be:
4-disk RAID-Z2 = 128KiB / 2 = 64KiB = good
5-disk RAID-Z2 = 128KiB / 3 = ~43KiB = not good
6-disk RAID-Z2 = 128KiB / 4 = 32KiB = good
10-disk RAID-Z2 = 128KiB / 8 = 16KiB = good
What is your recommendations regarding the number of disks? We are
planning to use 2 raid-z2 pools with 8+2 disks, 2 spare, 2 SSDs for L2ARC, 2
SSDs for ZIL, 2 for syspool, and a similar machine for replication.
Thanks in advance,
You've hit on one of the hardest parts of using ZFS - optimization. Truth
of the matter is that there is NO one-size-fits-all best solution. It
heavily depends on your workload type - access patterns, write patterns,
type of I/O, and size of average I/O request.
A couple of things here:
(1) Unless you are using Zvols for raw disk partitions (for use with
something like a database), the recordsize value is a MAXIMUM value, NOT an
absolute value. Thus, if you have a ZFS filesystem with a record size of
128k, it will break up I/O into 128k chunks for writing, but it will also
write smaller chunks. I forget what the minimum size is (512b or 1k, IIRC),
but what ZFS does is use a Variable block size, up to the maximum size
specified in the recordsize property. So, if recordsize=128k and you
have a 190k write I/O op, it will write a 128k chunk, and a 64k chunk (64
being the smallest multiple of 2 greater than the remaining 62 bits of
info). It WON'T write two 128k chunks.
(2) #1 comes up a bit when you have a mix of file sizes - for instance,
home directories, where you have lots of small files (initialization files,
source code, etc.) combined with some much larger files (images, mp3s,
executable binaries, etc.). Thus, such a filesystem will have a wide
variety of chunk sizes, which makes optimization difficult, to say the
least.
(3) For *random* I/O, a raidZ of any number of disks performs roughly like
a *single* disk in terms of IOPs and a little better than a single disk in
terms of throughput. So, if you have considerable amounts of random I/O,
you should really either use small raidz configs (no more than 4 data
disks), or switch to mirrors instead.
(4) For *sequential* or large-size I/O, a raidZ performs roughly equivalent
to a stripe of the same number of data disks. That is, a N-disk raidz2 will
perform about the same as a (N-2) disk stripe in terms of throughput and
IOPS.
(5) As I mentioned in #1, *all* ZFS I/O is broken up into
powers-of-two-sized chunks, even if the last chunk must have some padding in
it to get to a power-of-two. This has implications as to the best number
of disks in a raidZ(n).
I'd have to re-look at the ZFS Best Practices Guide, but I'm pretty sure
the recommendation of 7, 9, or 11 disks was for a raidz1, NOT a raidz2. Due
to #5 above, best performance comes with an EVEN number of data disks in any
raidZ, so a write to any disks is always a full portion of the chunk, rather
than a partial one (that sounds funny, but trust me). The best balance of
size, IOPs, and throughput is found in the mid-size raidZ(n) configs, where
there are 4, 6 or 8 data disks.
Honestly, even with you describing a workload, it will be hard for us to
give you a real exact answer. My best suggestion is to do some testing with
raidZ(n) of different sizes, to see the tradeoffs between size and
performance.
Also, in your sample config, unless you plan to use the spare disks for
redundancy on the boot mirror, it would be better to configure 2 x 11-disk
raidZ3 than 2 x 10-disk raidZ2 + 2 spares. Better reliability.
--
Erik Trimble
Java System Support
Mailstop: usca22-123
Phone: x17195
Santa Clara, CA
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http
