GFS is POSIX compliant. Flock, lock files, and fcntl locks work across the cluster. Caches are kept synchronized, so a change to a buffer on one node will show up in another. The biggest problem we have had is when a user receives several hundred messages simultaneously, the 3 mail servers will get hundreds of procmail processes backed up waiting for the lock files.
There are down sides, GFS is slower then EXT3, all locks have to be cleared through GULM (Grand Unified Lock Manager). GULM is a service that runs on another host that keeps track of what is locked where. Essentially, any lock that would have been managed in a local kernel is move off to GULM.
There is a temptation to compare GFS to NFS or CIFS. GFS is more accurately compared to EXT3 or JFS. Each host that mounts a file system from a SAN has its own journal and must join the lock space for the file system. If the host crashes, one of the other hosts will replay the journal and the downed host will be fenced from the shared storage. After the downed host is rebooted, it can be unfenced and return to the cluster. NFS and CIFS require a shared file server that manages the storage. GFS has a shared SAN and all of the cluster members have read/write access to the same volume on the SAN and work together to manage the file system. Because each node has direct access to the SAN, many operations (reads/writes) can be faster then NFS or CIFS and with direct I/O on some operations are faster then an EXT3.
GFS does add a lot of complexity. I would suggest you look at other options first. We were looking at GFS for web use. Unfortunately, we out grew the mail server and needed to replace it. After some tinkering, we decided to move everything from the old RS/6000 to a cluster running Linux and GFS. It has been up and running for 10 months now. Linux and GFS appears to the users to be 100% compatible with the old mail server. Users can log in and run pine, use imap, pop, procmail, exactly like they did on the old server. A load balancer distributes imap, pop, and smtp traffic between the 3 mail servers. A fourth server is also used as the "shell" server where some users log in to run pine and/or the old unix mail program.
Currently there are around 5000 students, faculty and staff that use this system. Lately, there have been as many as 800 logged in simultaneously, most via imap. We have had some problems with the SAN getting flooded, email generates a lot of disk I/O.
The engineers at RedHat have done a great job with GFS. If you do your home work and with some tinkering, you can get a highly available cluster of servers that will process email.
Recently, Mark Crispin posted a set of issues with NFS and why it should be avoided. At the end there was a comment about how to process email efficiently. Mark, could you post that?
Matt
On Thu, 2006-11-02 at 19:08 -0500, Erik Kangas wrote:
I wonder if anyone has played with GFS SAN as an alternateive to NFS that may/is supposed to have locking-like-local? http://www.redhat.com/docs/manuals/csgfs/admin-guide/ Or does this have similar deficiencies to NFS? -Erik Kangas LuxSci Mark Crispin wrote: > On Thu, 2 Nov 2006, Josko Plazonic wrote: >> It also has read and write delegation which should take care of some >> of your other complaints about NFS. For example, if you are granted >> read delegation no other client can write to the file. With write >> delegation no other client can write to or read the file. > > That isn't good enough. IMAP has shared read/write. The NFSv4 RFC > itself says that it expects that lock manipulation is less frequent > than read()/write() operations. To do what IMAP needs with primitive > as read/write delegation, lock manipulation has to be twice as frequent. > >> Just saying it might warrant another look - if nothing else to >> discover from what shortcomings it suffers... > > I fail to see the benefit in investing more time chasing a > non-solution that has never worked right in the past; and which has > known unfixable design flaws even if it can be made to work now. > > A vastly superior, proven, and cheaper solution is known and available > today. > > It isn't as if I spend my days thinking "gee, I haven't beaten my head > against that wall in a while, time to go and do it again." > > -- Mark -- > > http://staff.washington.edu/mrc > Science does not emerge from voting, party politics, or public debate. > Si vis pacem, para bellum. > _______________________________________________ > Imap-uw mailing list > [email protected] > https://mailman1.u.washington.edu/mailman/listinfo/imap-uw > _______________________________________________ Imap-uw mailing list [email protected] https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
_______________________________________________ Imap-uw mailing list [email protected] https://mailman1.u.washington.edu/mailman/listinfo/imap-uw
