Thanks! This is great information. It sounds like you’re advocating one giant zpool consisting of a bunch of raidz2 vdevs. What would be the ideal number of hard drives in a given raidz2 vdev? Would read/write performance with raidz2 + mirrored ZIL be comparable or better than just two mirrored drives without ZIL? If we end up with with a failed disk and need to resilver a replacement, what would performance look like during that time frame?
Are there any downsides to storing everything in one giant zpool rather than breaking it up into smaller zpools (i.e. are any of my concerns from my previous email about higher risk of failure valid)? Assuming I go with one zpool, would it be better to create smaller filesystems and distribute our data amongst them, or are there no downsides to storing all data in a single filesystem? Alex On Mar 6, 2014, at 9:20 AM, Jon <[email protected]> wrote: > Hi Alex, > > I build similar systems for polished/productized "Enterprise Private > Cloud" solutions and more recently I've been focusing on storage. I > would recommend you have individual RAIDz2 arrays and hot-add RAIDz2 > arrays (ZFS handles this well) to your pool to online-expand your > storage as needed. > > You can hot-add JBOD chassis for more arrays as well. Make sure to use > an LSI card like the 9207-8e > > Place the ZIL on two Mach16 SSDs (mirrored) to speed up writes ; Joyent > shares information about what hardware they use and the SSDs are listed > here http://eng.joyent.com/manufacturing/bom.html#270-0022 > > Also hot-add less expensive 2.5" SSDs as needed for the L2ARC to speed > up reads. > > See https://blogs.oracle.com/brendan/entry/test written by Brendan Gregg > the Lead Performance Engineer at Joyent. > > - Jon > > On 03/06/2014 06:33 AM, Alex Adriaanse wrote: >> We’re needing to store many TBs of data (somewhere between 10TB and >> 50TB) on a SmartOS server (or cluster of SmartOS servers). Most of this >> data will be fairly static, and the files that will take up this space >> will typically average around 1MB in size. Performance isn’t going to be >> a huge concern, except that we’ll also house a database server where >> performance is a bit more important (although two 7200RPM drives in a >> mirrored zpool seem to be holding up currently just fine). We’ll be >> using zones for most virtual machines on this server. >> >> We’re looking at getting a dedicated server with a 36 hard drive bay >> chassis to give us lots of room for growth. I’m envisioning we have two >> options: >> >> 1. Create one giant zpool that will eventually span up to ~32 hard >> drives and dump all the files in there. When using 4TB drives in >> RAID10, this would give us 32TB of usable storage. This would be the >> easiest solution because we could dump most of our data in a single >> filesystem. However, I’m concerned about scalability and >> reliability, as the wrong two hard drives going out at the same time >> would kill the entire zpool, and also dealing with a ~32TB >> filesystem may be challenging (we use zfs send/receive to replicate >> our data to an offsite mirror server, so at some point the diffs >> between snapshots will become huge as we collect more and more data; >> the initial full zfs send/receive would take forever, too). >> 2. We create one zpool for each group of 4 hard drives in RAID10 (with >> 4TB hard drives this would give us 8TB of usable space per zpool). >> This would require some forethought in our application as we’d have >> to split our data carefully to make sure it doesn’t exceed 8TB per >> filesystem, but this shouldn’t be a big deal. >> >> >> I’m leaning towards the second option. How would you recommend we setup >> the zpool(s) on this server? Are there other options I haven’t thought >> of that I should explore? >> >> Thanks, >> >> Alex >> *smartos-discuss* | Archives >> <https://www.listbox.com/member/archive/184463/=now> >> <https://www.listbox.com/member/archive/rss/184463/25552528-d52a4d93> | >> Modify >> <https://www.listbox.com/member/?&;> >> Your Subscription [Powered by Listbox] <http://www.listbox.com> >> ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
