On Tue, 7 Nov 2006, Brian Thompson wrote:
Mark, after reading your last email, I guess it's just luck that we're using
traditional unix mailboxes.
Luck, or necessity, take your pick... ;-)
I wasn't previously aware that there are
additional negative issues related to using the other various folder formats
with respect to NFS that make traditional unix mailboxes the proper
choice (if it's necessary to use NFS at all).
Well, I've tried to get the word out, but sometimes a burnt hand is the
best way to take advice about fire to heart...
My question is, in our situation where there's only one NFS server
and one NFS client dealing with the imap folders, are any of the
negative effects of NFS alleviated?
No.
There is no locking, and that precludes mbx and mix. flock() locks, which
are preferred, don't work over NFS.
On SVR4 systems such as Solaris, we have to use POSIX fcntl() locks.
fcntl() locks have extremely unfortunate semantics which require a very
inefficient workaround (ever wonder why imapd is always spawning inferior
processes on SVR4?) to prevent lock loss.
Now, fcntl() locks *do* work over NFS. They do that using the lockd and
statd daemons. That's a problem. Even the modest level of locking done
by UW imapd for traditional UNIX mailboxes overwhelms the ability of the
lockd and statd daemons to handle the load. The result is a cluster-wide
collapse and deadlock.
As a result, UW imapd goes to considerable effort to detect if a file
about to be fcntl() locked is on an NFS filesystem, and if so the lock is
disabled, just as it would have been with flock()).
There are three different ways to check for NFS on SVR4:
. The oldest systems use way (1); and a binary built for it can't have
way (2) or (3) because the calls did not exist.
. Somewhat newer systems have way (2); way (1) does not work; and a
binary built for it can't have way (3) because the call does not exist.
. The newest systems use way (3); way (2) *may* also work; way (1) does
not work.
A binary built for way (2) systems *may* work with way (3) systems; but
otherwise there isn't binary portability.
Way (3) is a recent (within the past month) discovery for me. I learned
about it because a site running the latest version of Solaris reported the
familiar statd/lockd collapse/deadlock problem that I thought that I had
abolished when I discovered way (2) nearly a decade ago.
So, no, the latest version of Solaris didn't fix locking over NFS. The
problem with statd/lock is still there. This is just with the modest
level of locking that is done in traditional UNIX format for the "kiss of
death" feature to prevent inadvertant simultaneous access. Locking in mbx
and mix formats is far more complex and frequent.
Or, is it NFS itself that's the
problem (unrelated to concurrent access by multiple NFS clients).
I've seen the inode and the data caches get out of sync with each other on
an NFS client even when that NFS client was the only thing accessing the
file. Use of random access read/write I/O is one of the triggers.
It was no fun having to cat megabyte of cruft to multiple files repeatedly
in order to get the caches purged and be able to work with the file again.
This also begs the question: if there is only a single NFS client, with no
access to the data by other NFS clients (or by processes on the NFS
server), then what purpose is served by having NFS in the loop at all?
The IMAP server is going over a network to have another computer read from
its disk and deliver the bits to the IMAP server; why not have the IMAP
server read the bits from a directly-connected disk?
The purported advantages of NFS monoliths are the various redundancy
features that the monolith vendors have added to the product; those
features are not part of NFS itself. There is no technical reason why an
IMAP server, with local disk, can not be built to have the same level of
data redundancy.
If you worry about the IMAP server crashing; why do you believe that an
IMAP server is more likely to crash than an NFS server?
Also, if we were to switch over to "mix" format imap folders, will that
cause us additional problems in our scenario compared to traditional
unix format folders, considering that we only have one NFS server
and one NFS client touching the imap folders in the users' home
directories and the inbox (/var/mail) is on local disk?
INBOX in mix format is a directory called INBOX on the user's home
directory. There is no use of the spool directory (/var/mail) in mix
other than to snarf messages from a traditional UNIX mailbox in the spool
directory to the mix format INBOX.
Thus mix is unsuitable for you as long as the home directory is NFS
mounted.
Obviously any particular user might have a home computer, work
computer, and notebook all trying to access the same imap folders
at the same time. Anyway, I guess I'm still not clear on whether the
NFS issues that have been warned against are related to multiple
imap clients, multiple NFS clients, or simply the NFS protocol itself.
Yes to all three, especially if you use a mailbox format other than
traditional UNIX format.
-- Mark --
http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.
_______________________________________________
Imap-uw mailing list
[email protected]
https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
