Many thanks to all who responded. I learned a lot from this thread! For now
I have decided to make a 3 way mirror because of the read performance. I
don't want to take a risk on an unmirrored drive.
Instead of replying to everyone separately I am following the Sun Managers
system since I read that newsgroup occasionalliy also. Here's a summary of
the responses.
Jim Klimov wrote:
Well, you can use this drive as a separate scratch area, as a separate
single-disk pool, without redundancy. You'd have a separate spindle for
some dedicated tasks with data you're okay with losing.
I thought about that and I really don't like losing data. I also don't
generate much temporary data so I love ZFS because it makes mirroring
easy. On my other systems where I don't have ZFS I run hourly backups from
drive to drive. Consumer drives are pretty good these days but you never
know when one will fail. I had a failure recently on a Linux box and
although I didn't lose data because I back up hourly it's still annoying to
deal with. If I hadn't had another good drive with that data on it I would
have lost critical data.
You can also make the rpool a three-way mirror which may increase read
speeds if you have enough concurrentcy. And when one drive breaks, your
rpool is still mirrored.
I think that's the best suggestion. I didn't realize a 3 way mirror would
help performance but you and several others said it does, so that's what I
will do. Thanks for the suggestions, Jim.
Roy pointed out a theoretical 50% read increase when adding the third drive.
Thanks Roy!
Edward Ned Harvey wrote:
In my benchmarking, I found 2-way mirror reads 1.97x the speed of a single
disk, and a 3-way mirror reads 2.91x a single disk.
Always great having hard data to base a decision on! That helped me make my
decision! Thanks Edward!
Jim Klimov answered a question that came up based on comments that read
performance was improved in a three way mirror:
Writes in a mirror are deemed to be not faster than the slowest disk - all
two or three drives must commit a block before it is considered written
(in sync write mode), likewise for TXG sync but with some optimization by
caching and write-coalescing.
Thanks Jim! Good to know.
Edward Ned Harvey pointed out If you make it a 3-way mirror, your write
performance will be unaffected, but your read performance will increase 50%
over a 2-way mirror. All 3 drives can read different data simultaneously
for the net effect of 3x a single disk read performance.
Bob clarified the theoretical benefit of adding a third drive to a mirror by
saying I think that a read performance increase of (at most) 33.3% is more
correct. You might obtain (at most) 50% over one disk by mirroring it. Zfs
makes a random selection of which disk to read from in a mirror set so the
improvement is not truely linear.
Thanks guys, that makes sense.
Daniel Carosone suggested keeping the extra drive around in case of a
failure and in the meantime using an SSD in the 3rd SATA slot. He pointed
out a few other options that could help with performance besides creating a
3 way mirror when he wrote:
Namely, leave the third drive on the shelf as a cold spare, and use the
third sata connector for an ssd, as L2ARC, ZIL or even possibly both
(which will affect selection of which device to use).
That's not an option for me right now but I am planning to revisit SSD again
when the consumer drives are reliable enough and don't have wear issues.
Right now overall integrity and long service life are more important
than absolute performance on this box, although since I have the integrity
with the ZFS mirror I could add an SSD but I really don't want to deal with
another failure as long as I don't have to. I do want additional performance
if I can afford it, but not at the expense of possible data loss.
Daniel also wrote:
L2ARC is likely to improve read latency (on average) even more than a
third submirror. ZIL will be unmirrored, but may improve writes at an
acceptable risk for development system. If this risk is acceptable, you
may wish to consider whether setting sync=disabled is also acceptable at
least for certain datasets.
I don't know what L2ARC is, but I'll take a look on the net. I did hear
about ZIL but don't understand it fully, but I figured spending 500G on ZIL
would be unwise. By that I mean I understand ZIL doesn't require much
storage but if I don't have an identical drive I can't add a drive or slice
with less storage than the other drives in a mirror to that mirror, so I
would be forced to waste a lot of storage to implement ZIL.
Finally, if you're considering spending money, can you increase the RAM
instead? If so, do that first.
This mobo is maxed out at 4G, it's a socket 775 I bought a couple of years
ago. I have always seen the benefits to more RAM and I agree with you it
helps more than people generally believe. Next time I buy a new box I am
hoping to go with 8 to 16G although on
