On May 22, 2008, at 8:36 PM, John Summerfield wrote:
Terry Davis wrote:
Hello,
I am going to be implementing an NFS cluster on RHEL5 x86_64 that I
would
like to serve up a single very large volume (one volume for
application
reasons). Right now my SAN max volume capability is 15 TB. So,
I'd like to
use LVM and tie several physical volumes together into a sigle
logical
volume. As stated, the cluster will be serving this volume up
via NFS and
the data is random files but I estimate the number of files to be
around 70
million per 10 TB of data.
1) What file system should I use? Ext3 appears to have a 4k block
size
limitation which puts my max size at 8 TB. Reiserfs doesnt appear
to be
supported. XFS?? GFS??
2) Is LVM safe for this type of deployment?
3) Any other thoughts?
Really, does it have to be a single filesystem? Can you export a
hierarchy of mounted smaller filesystems?
What is the largest filesystem you have now, and how long does it
take to fsck? (I've not tried this either, but I'd not try it on an
important system).
I think John has really raised an excellent point. I'm in a similar
position, where users are requesting "huge" (by my standards)
filesystems, and I find myself having to explain to them why they
aren't a good idea, but that is based on my admittedly limited
knowledge and experience.
I only want to use filesystems that are supported by Red Hat. (This
shows that my experience is from the AIX and Solaris world. :-)
When we first started using RHEL, we wound up creating some 4, 6, and
8 TB filesystems (ext3, over LVM, from an FC SAN with SATABeast
storage devices.) Due to our lack of experience in this environment,
we wound up doing some things "wrong", and started experiencing semi-
regular (once every 3-5 weeks) corruption on those filesystems. First
we tried fsck'ing, and after waiting many hours for it to finish, only
to find that it didn't fix all the problems, we wound up restoring
from backup, which took days in some cases. Not exactly the
experience I want to provide to my users.
Aside from that, there is the issue of backup. We have a well
established and fairly robust backup system based on Tivoli Storage
Manager. It works great. It also does its backups in parallel, but
only on the filesystem level. When you've got an 8 TB filesystem,
with 2-3 TB of changes daily, do you know what that does to your daily
backups? I do: it hoses them, and give you heartburn.
it is certainly possible that I just don't know what I'm doing, but I
don't think so. What I do think is that, while disk capacity has
grown exponentially in the past few years, making multi-terabyte
filesystems technically possible, in reality, all the tools the
sysadmin needs to keep those filesystems healthy, and to fix them when
their broken, just haven't kept pace.
(Not to hijack this subject, but I wish there was some resource or
community that had experience in this area. I'm liberal in politics,
but uber-conservative at work. I don't want to be the "first" to do
something weird, like 8 TB filesystems. I want to learn from the pain
and experience of others. :-)
So, to repeat John's point: do you really need to do it that way? Do
you understand the issues you're going to face with backup and
recovery? Do you users, or whomever is asking you do to it this way
understand them?
-s-
_______________________________________________
rhelv5-list mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/rhelv5-list
