First I'd like to note that contrary to the nomenclature there isn't any one "SAN" product that all operates the same. There are a number of different vendor provided solutions that use a FC SAN to deliver luns to hosts, and they each have their own limitations. Forgive my pedanticism please.
>On Sun, Oct 28, 2012 at 04:43:34PM +0700, Fajar A. Nugraha wrote: > On Sat, Oct 27, 2012 at 9:16 PM, Edward Ned Harvey > (opensolarisisdeadlongliveopensolaris) > <opensolarisisdeadlongliveopensola...@nedharvey.com> wrote: > >> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-() > >> boun...@opensolaris.org] On Behalf Of Fajar A. Nugraha > >> > >> So my > >> suggestion is actually just present one huge 25TB LUN to zfs and let > >> the SAN handle redundancy. > > > create a bunch of 1-disk volumes and let ZFS handle them as if they're JBOD. > > Last time I use IBM's enterprise storage (which was, admittedly, a > long time ago) you can't even do that. And looking at Morris' mail > address, it should be revelant :) > > ... or probably it's just me who haven't found how to do that. Which > why I suggested just use whatever the SAN can present :) Many vendors don't allow the hosts to control the disk - or only do it on their lower end storage. Usually because the higher end stuff has 'intelligent' caching, extra processors, etc etc, to handle the storage workload in their own way. > >You are entering the uncharted waters of ``multi-level disk >>management'' here. Both ZFS and the SAN use redundancy and error- >checking to ensure data integrity. Both of them also do automatic >replacement of failing disks. A good SAN will present LUNs that >>behave as perfectly reliable virtual disks, guaranteed to be error >free. Almost all of the time, ZFS will find no errors. If ZFS does >find an error, there's no nice way to recover. Most commonly, this >happens when the SAN is powered down or rebooted while the ZFS host >is still running. That's been my experience - ZFS plays nice, we do scrubs monthly to check for any data problems, which we have to get fixed from backups. The biggest messes happen when all the paths go offline for whatever reason while the host is still running. We have redundant networks of SAN switches and our storage arrays don't get powered down if we can help it, so that doesn't come up too often, usually only when someone is playing with the cables in an unfriendly way. All that being said, the real limitations probably end up being on your hardware. For example, one vendor's tier one solution only handles synchronous mirroring on 4TB luns or smaller - so if you're using that vendor's software and doing synchronous mirroring, you'd want to do a bunch of 4TB or less luns. Some vendors do 'wide-striping' across pools of disk on the back end, but some don't. If yours doesn't, and you can't present the disks directly, you'll want to find out what layout on the back end is optimized for the best performance for your particular workload via ZFS (database, digital media streaming, etc), and set it up that way. That will probably tell you what LUN sizes to use. On your host side, there's also the consideration of ssd/scsi queuing. If you're running on only one LUN, you're limiting your IOPS to only one IO queue over your FC paths, and if you have that throttled (per many storage vendors recommendations about ssd:ssd_max_throttle and zfs:zfs_vdev_max_pending), then one LUN will throttle your IOPS back on your host. That might also motivate you to split into multiple LUNS so your OS doesn't end up bottle-necking your IO before it even gets to your SAN HBA. Summary - my experience on FC SANs (previous and ongoing) is that ZFS is great in that it doesn't tell me what LUN sizes are the best to use. It's a combination of what my storage array limitations and strengths are, as well as my OS configuration and application workload that usually end up telling me if I should use one big lun, or twenty small ones. And when I get it wrong (because what percentage of times does an app admin/DBA accurately guess their IOPS and storage bandwidth needs), ZFS usually lets me fix it, sometimes even non-disruptive to uptime. Cheers, Brian -- -Gary Mills- -refurb- -Winnipeg, Manitoba, Canada- _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > _______________________________________________ > zfs-discuss mailing list > zfs-discuss@opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
