----- Original Message ----- From: "Mark Crispin" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Thursday, July 08, 2004 10:54 AM Subject: Re: File Locking
> > Again - from what I understood, this issue with flock() over NFS has been > > fixed - besides isn't that what the nfslock daemon is for? > > I hope not! Those lock-over-NFS daemons are for fcntl() locking only, and > THEY DO NOT WORK WORTH A DAMN! It is a *feature* that flock() does not > use those broken daemons. That would then also apply for exclusive writes as well So if we were to insist on remote storage we would have to modify IMAP server source to do fcntrl instead of flocks where necessary and it still wouldn't work because lock files can't be counted on. > > > To me this means that the documentation on file locking is relevant to > > us???? Although I don't feel that I fully understand the implications. > > The jist that you should get is that although there is basic compatibility > in locking between all this software, you should use companion software > whenever possible. > > For example: if you use UW imapd for your IMAP server, you should use UW > ipop3d for your POP3 server. If you use Cyrus imapd for your IMAP server, > you should use Cyrus pop3d for your POP3 server. > > If you mix non-companion softare, such as UW imapd and cucipop, you will > have basic compatibility on locking but less compatibility than if you use > companion software. To what extent this may cause problems is uncertain. > It is not a recommended configuration. That is a recommendation I will make as well. Thank You > >> Has your management considered that they have a single point of failure > >> (the NFS server) in their configuration; and that when that point fails > >> your entire facility is dead? > > The NFS Server is mirrored and has a seamless failover device. > > OK, so I cut the wire that goes to your NFS server. Or I run a virus on > it that piddles over the filesystem. It's still a single point of > failure. > It is mirrored on a separate server and is not on the same switch either. The problems you are describing are no different than any redundancy plans that any architecture would have to implement and while of course not bullet-proof, I really feel like we're covered. Perhaps you could explain better what you mean. > > Do you mind if I ask how many users you serve? > > 6 digits. > Earlier you said: > When you say each user gets a separate server do you mean each individual > account? Yes. > and by server do you mean server application or complete server > hardware? Server system. Hardware. I want to understand this better. When you say each user account is on a separate piece of hardware I would take that to mean that you would then need a full system times 6 digits. Surely this isn't what you mean. Is it only the currently active accounts, or is there a way you divide the user space among thousands of users per machine? If you had 999,999 users and each server only cost you 10 dollars (fat chance), are you really saying you spent nearly 10 million dollars on server hardware so each user could have one!? > > Lustre is able to mirror data over multiple target nodes and rebuild the > > data seamlessly if lost. It promises to create paralell connections to > > different disks on different servers over separate network ports with > > multiple "middleware" (MetaData) servers constantly redirecting traffic. > > We're hoping that this would sufficiently divide the labor for I/O. > > The more that I hear about Lustre, the more convinced I am that it is > unsuitable for IMAP. It's optimizing the wrong things. You may be very right there. It is intended for high-computational applications such as scientific research number crunching. Don't Take my word for it, I'm not an expert in Lustre by a long shot I may be completely off. This much I can tell you. We need to get our IMAP server away from NFS ASAP and the current candidates are 1) a screaming machine with Super I/O capability with every service running on it, 2) our current architecture with Lustre as our remote file system in place of NFS, or 3) a complete rebuild of our email architecture. The Last of which is going to take quite some time to migrate. Right now we're just investigating our options. To summarize what you have told me though: our problem is three-fold. 1) Our system architecture needs a complete redesign 2) The current system needs a serious increase in I/O capability 3) Locking isn't working with our system. We haven't seen corrupted data, so I guess we've been lucky. If file locking isn't contributing to the I/O problem (because locks aren't occurring) then from my perspective so far Lustre is the only candidate with improvement over I/O from what NFS currently offers -- Do you know of another??? What do I need to look for to make sure it would be suitable for IMAP? What optimizations that is? Unless you mean to tell me that IMAP is never to be used in any type of file system besides a local file system and that is an inherent requirement / limitation of IMAP. If file locking is occurring, and has an incompatibility because of the non c-client applications (or some other hidden blocking problem with file locking) Lustre shows promise to remedy this because of it's "improved locking techniques" which I'll have to look in to (as skeptical as I may be). In the latter case performance is not likely to improve but in the short-term at least we could avoid data corruption. (Because the Lustre OST provides flock support as well as exclusive reads/writes). The bottom line is until we find something better Lustre shows the promise of replacing NFS and removing related limitations without requiring us to completely change everything overnight, and without requiring us to purchase more hardware..
