Thank you all for the information, I believe it is much clearer to me.
"Sequential Reads" should scale with the number of disks in the entire
zpool (regardless of amount of vdevs), and "Random Reads" will scale with
just the number of vdevs (aka idea I had before only applies to "Random
Reads"), which I am much more happy with. Everything on my system should be
mostly sequential as editing should not occur much (aka no virtual machine
type things), when things get changed it usually means deleteing the old
file and adding the "updated" file.
I read though that ZFS does not have a "defragmentation" tool, is this
still the case? It would seem with such a performance difference between
"sequential reads" and "random reads" for raidzN's, a defragmentation tool
would be very high on ZFS's TODO list ;).
> Is the enclosure just a JBOD? If it is not, can it present drives
>I assume your other hardware won't be a bottleneck?
>(PCI buses, disk controllers, RAM access, etc.)
A little more extra information about my system, it is a JBODs. The disks
go to a SAS-2 Expander (RES2SV240), and that has a single connection to a
tyan motherboard which has a LSI SAS 2008 chip controller built-in. The CPU
is a i3 with a DMI of 5 GT/s (DMI is new to me vs FSB). RAM is server grade
unbuffered ECC DDR3 1333 8GB sticks. It is a dedicated machine which will
do nothing but serve files over the network.
To my understanding the Network or disks themselves should be my bottlenet;
SAS-2 connection between SAS Expander and mobo should be 24Gbit/s or
3GB/s(1.5GB/s if SAS-1), and 5 GT/s should provide ~20 GB/s max bandwidth
for the 64bit machine from what I read online. I don't think this affects
me, but, I was also curious, does anyone know if the mobo <> sas expander
will still establish a SAS-2 connection (if they both support SAS-2) if the
backpanes only support SAS-1 / SATA 3Gb/s? I never looked up the backpane
partnumbers in my Norco, but, think they support SATA 6Gb/s, so assume they
support SAS-2. But in essance SAS Expander <> HDD's wont be over 3Gb/s per
port, so as long as SAS Expander <> mobo establish's it's own SAS-2
connection regardless of what SAS Expander <> HDD's do, then I don't even
have to think about it. 1.5GB/s (SAS-1) is still above my optimal max
In essance, if the drives can provide it (and network interface ignored) I
think the theoredical limitation is 3GB/s. I mentioned 1.25GB/s for the
10GigE is max I am looking at, but, I'd be happy with anywhere between
500MB/s->1GB/s for sequential reads of large files (don't really care about
any type of writes, and hopefully random reads do not happen so often *will
test with iopattern*).
> What is the uncorrectable
> error rate on these 3TB drives? What is the real random I/Ops
> capability of these 3TB drives?
I'm unsure of these myself, all the other parts have arrived, or are on
route, but I have not actually bought the HDD's yet so can still choose
almost anything. It will probably be cheapest consumer drives I can get
though (probably "Seagate Barracude Green ST3000DM001"'s or "Hitachi 5K3000
3TB"'s). The 1.5TB's I have in my old system are pretty much the same thing.
> How much space do you _need_, including reasonable growth?
My old system is 9.55TB and almost full, and I have about 3TB spread out
elsewear. This was setup about 5years ago. With the 20 disk enclosure I'm
thinking about 30TB usuable space (but maybe only usign 15 disks at first),
and hoping it'll last for another 5 years.
> How did you measure this?
ATTO Benchmark is what I used on the local machine for the 500MB/s number.
For small files (1kB>16kB) it is small (50MB>150MB), for the larger 256kB+
it reads ~550MB/s). This is hardware RAID5 though. Over the 1Gbit network
Windows7 always gets up to ~100MB/s when writing/reading from the RAID5
> What OS? I have a 16 CPU Solaris 10 SPARC server with 16 GB of RAM
The new ZFS system OS will probably be OpenIndiana with the v28 zpool. I
have been looking at FreeNAS (FreeBSP) and a little up in the air on which
Thank you all for the information. I will very likley create two zpools
(one for 1.5TB drives, and one for 3TB drives), initially I thought down
the road if the pool ever fills up (probably like 5+ years) I would start
swaping the 1.5TB drives with 3TB drives to let the small vdev "expand"
after all were replaced, but, I didn't realize there could potentially be
performance problems via block size differences of 1.5TB (~5 year old
drives) drives and 3TB+ drives (~5 years in the future).
zfs-discuss mailing list