We have been running UW-IMAP on top of GFS for almost 2 years now at
UVM, running both mbox and mbx formatted mailboxes. It is a 6-node
cluster, with 3 lock managers running in failover (not distributed)
mode. The 6 mail nodes are attached to a switched SAN, and the storage
is provided by an IBM DS4800.
This is not a recommended config, obviously (and MRC has reminded us of
this :-) However, when we still had a single-system running UW-IMAP,
we decided to give the cluster a try, which would hopefully provide
better scalability while still providing a single-system image -
preventing us from having to touch 15,000+ desktop systems spread across
the state.
To make this work, we had to make a few small changes to the UW-IMAP
code base. The INBOX directory (/var/spool/mail) and c-client lockfile
location (/tmp) had to be moved to a GFS-shared filesystem, and into a
directory hash. Spool files were hashed by username (mine is
/var/spool/mail/j/t/jtl), c-client lockfiles were hashed by the LSB of
the inode number (so lockfile .fe06.7bcdf3 gets put in
/gfs/tmp/f/3/.fe06.7bcdf3). The hashing reduces directory lock
contention, which can make GFS perform worse than a single system with
local filesystems.
We also had to write a simple daemon to listen for the "kiss of death"
used to break locks on older connected imapds. Basically, the c-client
lockfile also includes the node ID, and if c-client wants to KOD someone
on another node, a UDP packet gets sent to that node, received on that
node by a daemon, who sends the KOD locally. The daemon is
affectionately named "dkod".
We're still running it, so we've had some level of success, but it is
not problem-free. Occasionally, we see imapds that have deadlocked, for
reasons that we can't explain. The problem exhibits itself to the user
as a mailbox that can't be opened read-write. However, we've *never*
seen corruption of mailboxes under GFS (and we DID see that occasionally
on the old single-system.)
In the beginning, there were bugs in GFS that we tickled with our
particular config, but Sistina (now RedHat) has fixed the problems that
we've run into. A good piece of advice is: don't create very large
(1TB) filesystems, because they can take days to fsck. We haven't had
to fsck in over a year, but in the first months of running it when we
had some bad hardware and a filesystem with minor corruption, we had to
take some filesystems down for hours while it was fsck'd. Sometimes we
were better off moving all the data we could off the filesystem and
creating a new filesystem in its place. Again, that hasn't happened in
over a year.
Overall, we're mostly happy with it at this point, although it would
have been nice to avoid the first few months of pain that we had due to
bugs in the filesystem.
We'd be happy to share code if anyone's interested.
Jim
Erik Kangas wrote:
Hello?
As a next step to the discussion of UW Imap on NFS, has anyone played
with UW IMAP on GFS (RedHat Global File System -
http://www.redhat.com/en_us/USA/home/solutions/gfs/) in an environment
where multiple IMAP machines are sharing the files.
From what I have read, GFS should take care of all the locking and
synchronization across a large number of attached machines without any
problem. Specifically, the file system appears and behaves like a local
file system. However, GFS is relatively new and not specifically
supported by most SAN vendors.
If anyone has any experience setting up a SAN with GFS with load
balanced UW IMAP servers using a folder format that depends on the
NFS-averse file system operations (i.e. mbx) please let us all know what
you have seen.
Thank you,
-Erik Kangas
http://LuxSci.com
_______________________________________________
Imap-uw mailing list
[email protected]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
_______________________________________________
Imap-uw mailing list
[email protected]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
