>>Ray said: >>> Using a Dell R720 head unit, plus a bunch of Dell MD1200 JBODs dual pathed >>> to a couple of LSI SAS switches. >> >Marion said: >> How many HBA's in the R720? > Ray said: > We have qty 2 LSI SAS 9201-16e HBA's (Dell resold[1]).
Sounds similar in approach to the Aberdeen product another sender referred to, with SAS switch layout: http://www.aberdeeninc.com/images/1-up-petarack2.jpg One concern I had is that I compared our SuperMicro JBOD with 40x 4TB drives in it, connected via a dual-port LSI SAS 9200-8e HBA, to the same pool layout on a 40-slot server with 40x SATA drives in it. But the server uses no SAS expanders, instead using SAS-to-SATA octopus cables to connect the drives directly to three internal SAS HBA's (2x 9201-16i's, 1x 9211-8i). What I found was that the internal pool was significantly faster for both sequential and random I/O than the pool on the external JBOD. My conclusion was that I would not want to exceed ~48 drives on a single 8-port SAS HBA. So I thought that running the I/O of all your hundreds of drives through only two HBA's would be a bottleneck. LSI's specs say 4800MBytes/sec for an 8-port SAS HBA, but 4000MBytes/sec for that card in an x8 PCIe-2.0 slot. Sure, the newer 9207-8e is rated at 8000MBytes/sec in an x8 PCIe-3.0 slot, but it still has only the same 8 SAS ports going at 4800MBytes/sec. Yes, I know the disks probably can't go that fast. But in my tests above, the internal 40-disk pool measures 2000MBytes/sec sequential reads and writes, while the external 40-disk JBOD measures at 1500 to 1700 MBytes/sec. Not a lot slower, but significantly slower, so I do think the number of HBA's makes a difference. At the moment, I'm leaning toward piling six, eight, or ten HBA's into a server, preferably one with dual IOH's (thus two PCIe busses), and connecting dual-path JBOD's in that manner. I hadn't looked into SAS switches much, but they do look more reliable than daisy-chaining a bunch of JBOD's together. I just haven't seen how to get more bandwidth through them to a single host. Regards, Marion _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
