----- Original Message ----- From: "Mark Crispin" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Wednesday, July 07, 2004 6:05 PM Subject: Re: File Locking
> It means that flock() will succeed, having done nothing. Thus, the > application will think that there is an flock() lock, but in fact there > isn't. Consequently, any assumptions that depend upon there being an > flock() lock are invalid. That would then make me assume that though our IMAP server appears to be working it must be silently working incorrectly, because we are accessing IMAP's data store from NFS. What symptoms would I experience when a flock() lock erroneously succeeds? Again - from what I understood, this issue with flock() over NFS has been fixed - besides isn't that what the nfslock daemon is for? > > are cutsiepop and sendmail two of programs that you refer to as "non c-type > > applications" I apologize for my ignorance on at least two counts. By cutsiepop I meant cucipop; and I meant to quote "non c-client type applications" which I'm finding (or have been told) that cucipop is among them. To me this means that the documentation on file locking is relevant to us???? Although I don't feel that I fully understand the implications. The jist of what I'm getting is "non c-client type applications may not get along well when accessing files concurrently with c-client type applications" The part that says "c-client based applications have a reasonable chance of winning as long as you don't use NFS for remote access to mail files." may explain the lag we experience when using IMAP simultaneously with POP and SMTP -- UNTIL we put Imap directly on the data system and it seems to steal priority from everyone else. (IMAP becomes screaming fast, and POP and SMTP get blocked altogether) > > The documentation also says that c-client uses flock on SystemV systems. > > No it does not. SystemV systems do not have flock(). They have fcntl() > locking. Again guilty, I understood this after further investigation - but from what I gathered, they are similar: "System V added new functions to the fnctl() system call, and a simple interface through the lockf() subroutine.." "Functionally, fcntl() locking is a superset of flock(); it is possible to implement a flock() emulator using fcntl(),...[but]" > I have not heard of Lustre before, but I remain highly skeptical. As am I, but NFS will not work and I have yet to discover another tool which will allow us to do what we want. Somehow we need to infinitely expand the data store and reduce (eliminate) the bottleneck with I/O. I understand now that there is an inherent flaw because all servers accessing the same data store clogs up the turnpike no matter how you look at it. Lustre promises to solve this problem by letting the clients access many data stores in paralell as if they were one volume. In my book, where there is a will, there is a way. I'll bet if we had 30,000 dollars to burn we could get a small SANS that would work for us as wasteful as that may be as far as a focus of resources. Since we need to solve this problem on a budget, and have been given a specific vision, Lustre really is the only candidate in the running for a tool that will work for us. Until we can change something management will remain convinced that it is a misconfiguration with IMAP and Lustre may end up being our next step if it is promising at all. > Has your management considered that they have a single point of failure > (the NFS server) in their configuration; and that when that point fails > your entire facility is dead? The NFS Server is mirrored and has a seamless failover device. > > When you say each user gets a separate server do you mean each individual > > account? > > Yes. > > > and by server do you mean server application or complete server > > hardware? > > Server system. Hardware. Do you mind if I ask how many users you serve? With 10,000+ Customers with an average of 5+ email accounts each it would be impossible for us to implement such an architecture. > > I do know that when IMAP is run on a > > separate server, and accesses the mail box over NFS it is slow, when it is > > put directly on the server that has the data it runs much faster > > This is exactly what I expect. NFS is evil for IMAP. Don't use it. > > > but seems > > to block POP and SMTP almost entirely. > > What do you mean by this? > When we put IMAP *on* the NFS server (POP and SMTP access the data over NFS and IMAP accesses the data directly from the drive) IMAP begins to perform a degree of magnitude better than we ever expected, however SMTP processes are waiting for IO, and we are unable to get more than a minimum amount of POP and SMTP connections made to those servers, of which only a handful complete and the rest time out. I imagine (I'm guessing) that IMAP gets all the locks on the files before either of the other two services are able just because IMAP gets requests in first. (It may also be because we are not yet using maildir and the entire /var/spool/mail folder has to be locked temporarily each time a file is locked. Lustre promises to fix this problem) > > What I am looking for is what would be required to allow all the data to > > reside in one [system] that would have the capacity to appear as one mount > > point > > Bad idea. Single point of failure. Single bottleneck for I/O. > Distributed services are good. Lustre is able to mirror data over multiple target nodes and rebuild the data seamlessly if lost. It promises to create paralell connections to different disks on different servers over separate network ports with multiple "middleware" (MetaData) servers constantly redirecting traffic. We're hoping that this would sufficiently divide the labor for I/O. And that is the goal we want to distribute services and --hearing it echo in my head I know it is contradictory but-- we want them to work as a whole. > > -- Mark -- > > http://staff.washington.edu/mrc > Science does not emerge from voting, party politics, or public debate. > Si vis pacem, para bellum. >
