First I'd like to note that contrary to the nomenclature there isn't any one
"SAN" product that all operates the same. There are a number of different
vendor provided solutions that use a FC SAN to deliver luns to hosts, and they
each have their own limitations. Forgive my pedanticism please.
>On Sun, Oct 28, 2012 at 04:43:34PM +0700, Fajar A. Nugraha wrote:
> On Sat, Oct 27, 2012 at 9:16 PM, Edward Ned Harvey
> <opensolarisisdeadlongliveopensola...@nedharvey.com> wrote:
> >> From: zfs-discuss-boun...@opensolaris.org [mailto:zfs-discuss-()
> >> boun...@opensolaris.org] On Behalf Of Fajar A. Nugraha
> >> So my
> >> suggestion is actually just present one huge 25TB LUN to zfs and let
> >> the SAN handle redundancy.
> > create a bunch of 1-disk volumes and let ZFS handle them as if they're JBOD.
> Last time I use IBM's enterprise storage (which was, admittedly, a
> long time ago) you can't even do that. And looking at Morris' mail
> address, it should be revelant :)
> ... or probably it's just me who haven't found how to do that. Which
> why I suggested just use whatever the SAN can present :)
Many vendors don't allow the hosts to control the disk - or only do it on their
lower end storage. Usually because the higher end stuff has 'intelligent'
caching, extra processors, etc etc, to handle the storage workload in their own
>You are entering the uncharted waters of ``multi-level disk
>>management'' here. Both ZFS and the SAN use redundancy and error-
>checking to ensure data integrity. Both of them also do automatic
>replacement of failing disks. A good SAN will present LUNs that
>>behave as perfectly reliable virtual disks, guaranteed to be error
>free. Almost all of the time, ZFS will find no errors. If ZFS does
>find an error, there's no nice way to recover. Most commonly, this
>happens when the SAN is powered down or rebooted while the ZFS host
>is still running.
That's been my experience - ZFS plays nice, we do scrubs monthly to check for
any data problems, which we have to get fixed from backups. The biggest messes
happen when all the paths go offline for whatever reason while the host is
still running. We have redundant networks of SAN switches and our storage
arrays don't get powered down if we can help it, so that doesn't come up too
often, usually only when someone is playing with the cables in an unfriendly
All that being said, the real limitations probably end up being on your
hardware. For example, one vendor's tier one solution only handles synchronous
mirroring on 4TB luns or smaller - so if you're using that vendor's software
and doing synchronous mirroring, you'd want to do a bunch of 4TB or less luns.
Some vendors do 'wide-striping' across pools of disk on the back end, but some
don't. If yours doesn't, and you can't present the disks directly, you'll want
to find out what layout on the back end is optimized for the best performance
for your particular workload via ZFS (database, digital media streaming, etc),
and set it up that way. That will probably tell you what LUN sizes to use.
On your host side, there's also the consideration of ssd/scsi queuing. If
you're running on only one LUN, you're limiting your IOPS to only one IO queue
over your FC paths, and if you have that throttled (per many storage vendors
recommendations about ssd:ssd_max_throttle and zfs:zfs_vdev_max_pending), then
one LUN will throttle your IOPS back on your host. That might also motivate you
to split into multiple LUNS so your OS doesn't end up bottle-necking your IO
before it even gets to your SAN HBA.
Summary - my experience on FC SANs (previous and ongoing) is that ZFS is great
in that it doesn't tell me what LUN sizes are the best to use. It's a
combination of what my storage array limitations and strengths are, as well as
my OS configuration and application workload that usually end up telling me if
I should use one big lun, or twenty small ones. And when I get it wrong
(because what percentage of times does an app admin/DBA accurately guess their
IOPS and storage bandwidth needs), ZFS usually lets me fix it, sometimes even
non-disruptive to uptime.
-Gary Mills- -refurb- -Winnipeg, Manitoba, Canada-
zfs-discuss mailing list
> zfs-discuss mailing list
zfs-discuss mailing list