>>>>> "bergman" == bergman <[email protected]> writes:
bergman> I'm adding a new storage device to our SAN, and this is a
bergman> good opportunity to reexamine our config and make
bergman> changes. I'm looking for recommendation about storage
bergman> configuration decisions.
bergman> RHCS-clustered Linux NFS servers with SAN attached
bergman> storage, serving ~45 Linux clients (scientific
bergman> computing cluster and stand-alone computational
bergman> servers)--mainly CentOS 5.4, some CentOS 4.8.
I assume the RHCS cluster exports NFS partitions to the clients? And
are your able to change filesystems by any chance? I think XFS allows
you to re-size filesystems on the fly, while ext3 only allows you to
grow filesystems.
Personally, I like Netapps (though I hate the 16Tb RAW limit on
Aggregates/volumes, though that goes away (slightly) with version 8.0
of OnTap.
I can grow/shrink volumes, put quotas in place, put in reporting only
quotas (so I can find out space hogs more quickly), etc.
bergman> Capacity and managment are much greater priorities
bergman> than performance
Where does cost fall? *grin*
bergman> Minimize future downtime due to regular maintenace
bergman> (ie., fsck may take ~1.5TB/hr)
How often do you FSCK your filesystems? I would hope that with a
cluster you don't crash at all...
bergman> Within our environment, allocation has been a greater
bergman> problem than performance--individual volumes are be
bergman> full, while others have lots of space. A configuration
bergman> where storage can be flexibility allocated to
bergman> individuals or projects is desireable.
Yeah, been there, done that. Still fighting it in larger quantities
now.
How are you doing your backups? How they are done will impact your
design.
bergman> A unified presentation to end-users is
bergman> desirable. We're using automount extensively now, and
bergman> this produces a confusion among end-users (typically
bergman> resulting in a desperate question about why "my
bergman> project directory is missing").
bergman> I've had very bad experience with automount in a mixed
bergman> environment (Linux automount clients mounting from
bergman> Linux, Solaris and Irix NFS servers) and would prefer
bergman> to avoid automount...but the non-Linux NFS servers are
bergman> going away soon, so this isn't an absolute
bergman> requirement.
Umm... above you state that it's just for a cluster of compute nodes,
and now you imply you need access from other OS flavors. Which is it?
bergman> As filesystems grow, I have concerns about
bergman> management--fsck time becomes excessive, and I've
bergman> heard annecdotes about filesystem stability over
bergman> certain sizes (8TB, etc.). For this reason, a solution
bergman> that virtually consolidates separate back-end
bergman> filesystems of moderate sizes may be preferable
bergman> (sounds like automount!).
Sorta. We did the Acopia NFS aggregator thing for a while (not cheap
mind you) but kept running into issues and the elephant in the room
was backups and how to get a unified restore was a killer.
Basically, the Acopia sits in front of a bunch of NFS back ends and
virtualizes them. You could transparently migrated data from
expesive/fast disks to slower/cheaper disks and have various policies
to move data around. Nice idea, very nice idea.
When you have data spread across three different backends and one goes
down.... life gets interesting. And backups are also problematic,
because you need to make sure data migrations don't happen during the
backups, etc.
bergman> * union mounts Virtually combine separate filesystems
bergman> by overlaying directory trees. Data sources are
bergman> ranked by priority, to determine which directory
bergman> takes precendece. There are kernel and user-land
bergman> (FUSE) implementations.
This is a path to insanity. Your priority is not another user's
priority. And then who wins?
bergman> * LVM
bergman> Logically group unformatted storage volumes (LUNS)
bergman> into virtual LUNs, formatting those as
bergman> filesystems. Logical volumes can grow in size
bergman> (depending on available storage and the capability
bergman> of the filesystem on the logical volume). Using
bergman> LVM would likely mean having 4 large virtual
bergman> volumes (home, project, scratch, and archive
bergman> directories), each made of from LUNs from
bergman> different storage devices.
bergman> - difficult to address performance issues when a
filesystem is
bergman> made up of LUNs from different storage devices
bergman> potentially using different RAID levels, etc.
Why do you care? You said performance was secondary.
bergman> - complexity on NFS server
So?
bergman> - potential filesystem issues at large sizes
bergman> (fsck time, ext3fs stability over 8TB,
bergman> etc.)
ext3fs has been *very* stable. Just make sure you have a journal
enabled with enough size and you should be all set.
bergman> + existing familiarity with LVM
bergman> + simple presentation to user
bergman> + simple management on NFS clients (ie., a single
bergman> mount point per data type)
Simplicity is a virtue...
bergman> * automount
bergman> Similar to unionfs, individual filesystems would provide
separate
bergman> directories.
bergman> - historical poor experience with automount
bergman> - managment of automount maps
bergman> - end-user confusion
bergman> - smaller filesystems = allocation problems
bergman> + simplicity on NFS server
bergman> + widespread use
bergman> + flexible use of back-end space
bergman> + virtual consolidation of small back-end filesystems
I'd rather consolidate all the smaller filesystems into larger, single
filesystems and then apply quotas to directories or sub-sets instead.
bergman> Any suggested alternatives?
Buy a netapp, get rid of the SAN completely and just use NFS to
manange your data. I feel that SANs are over-rated, esp for simple
file sharing in an engineering environment.
As another user said, put Solaris 10uX on an x86_64 box with ZFS
talking to your existing and new SAN and export the data that way.
The Sun 7000 series boxes are sexy. We almost went that route about a
year ago, but the lack of reporting for per-user disk quotas then was
a hindrance. But being able to carve things up easily and
increase/reduce volumes as will might be just as good, except you have
to manually scan volume(s) to determine usage.
John
_______________________________________________
Discuss mailing list
[email protected]
http://lopsa.org/cgi-bin/mailman/listinfo/discuss
This list provided by the League of Professional System Administrators
http://lopsa.org/
