Wow. Thanks for the data. This is somewhat consistent with what I
predict in RAIDoptimizer.
Robert Milkowski wrote:
Hello zfs-discuss,
Server: x4500, 2x Opetron 285 (dual-core), 16GB RAM, 48x500GB
filebench/randomread script, filesize=256GB
Your performance numbers are better than I predict, and I believe it
is because of filesize=256GB. With that small of a filesize, you
won't be getting any long seeks. In other words, your performance
will go down as you use more of the space and need long seeks.
I've added my RAIDoptimizer predictions below...
RAIDoptimizer predicts that a single Hitachi E7K500 SATA drive is
good for 79 short, random read iops (7,200 rpm, 8.5 ms avg seek)
These drives should be using the SATA framework which includes
NCQ, so that will also have a positive affect on performance for
many threads.
2 disks for system, 2 disks as hot-spares, atime set to off for a
pool, cache_bshift set to 8K (2^13), recordsize untouched (default).
pool: 4x raid-z (5 disks) + 4x raid-z (6 disks) means that one pool
was created wit 4 raid-z1 groups each with 5 disks and another 4
raid-z1 groups each with 6 disks.
1. pool: 4x raid-z (5 disks) + 4x raid-z (6 disks)
(36 disks of usable space)
a. nthreads = 1 ~60 ops
b. nthreads = 4 ~250 ops
c. nthreads = 8 ~520 ops
d. nthreads = 128 ~1340 ops
1340/8 = 167 ops
RAIDoptimizer predicts 632 iops for lots of threads
measured/predicted = 2.12
2. pool: 2x raid-z2 (10 disks) + 2x raid-z2 (12 disks)
(36 disks of usable space)
a. nthreads = 1 ~50 ops
b. nthreads = 4 ~190 ops
c. nthreads = 8 ~360 ops
d. nthreads = 128 ~720 ops
720/4 = 180 ops
RAIDoptimizer predicts 316 iops for lots of threads
measured/predicted = 2.28
3. pool: 2x raid-z2 (22 disks)
(40 disks of usable space)
a. nthreads = 1 ~40 ops
b. nthreads = 4 ~120 ops
c. nthreads = 8 ~160 ops
d. nthreads = 128 ~345 ops
345/2 = 172 ops
RAIDoptimizer predicts 158 iops for lots of threads
measured/predicted = 2.18
4. pool: 4x raid-z (11 disks)
(40 disks of usable space)
a. nthreads = 1 ~50 ops
b. nthreads = 4 ~190 ops
c. nthreads = 8 ~340 ops
d. nthreads = 128 ~710 ops
710/4 = 177 ops
RAIDoptimizer predicts 316 iops for lots of threads
measured/predicted = 2.25
5. pool: 4x raid-z2 (11 disks)
(36 disks of usable space)
a. nthreads = 1 ~55 ops
b. nthreads = 4 ~200 ops
c. nthreads = 8 ~350 ops
d. nthreads = 128 ~760 ops
760/4 = 190 ops
RAIDoptimizer predicts 316 iops for lots of threads
measured/predicted = 2.41
6. pool:22x mirror (2 disks)
(22 disks of usable space)
a. nthreads = 1 ~75 ops
b. nthreads = 4 ~320 ops
c. nthreads = 8 ~670 ops
d. nthreads = 128 ~3900 ops
3900/22 = 177 ops
3900/44 = 88 ops (it's a read test after all)
RAIDoptimizer predicts 3474 iops for lots of threads
measured/predicted = 1.12
Well, random reads really tends to give about 1-2x # of IOs of a
single disk in a raid-z[12] group. For some workloads it's really bad.
For some workloads I would definitely prefer much better random read
performance in terms of IO/s and trade in write performance.
Maybe something like raid-y[12] which are more like classical
raid-[56]? That way user would have a choice - good read performance
however excellent write performance, or just the opposite.
I don't see how you can get both end-to-end data integrity and
read avoidance. Given the large number of horror stories from
RAID-5 users, I'd rather not give in on end-to-end data integrity.
Ideas?
Right now RAID-Z1 and RAID-Z2 is just plainly horrible in terms of #
IOs for random-read environments when caching ratio is marginal.
This is especially painful with x4500.
Yes.
I might need to adjust RAIDoptimizer's predictions for RAID-Z/Z2.
I'll need more data to prove a new model, though [yes, its on my list :-)]
-- richard
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
