---- Original Message ----- From: "Peer Heinlein" <[EMAIL PROTECTED]> To: <[email protected]> Sent: Saturday, March 24, 2007 11:09 AM Subject: [Courier-imap] Building IMAP-cluster with NFS/SAN > >Hi! > >[First: I read the discussion "Courier-Imap over NFS or OCFS" some weeks >ago, but it didn't answered all my questions.] > >For several weeks now I'm investigating how to build a real >high-performance-cluster with redundant IMAP-Nodes with Courier-IMAP.
As long as NFS storage can do required Ops/s with acceptable latency, all you need to worry is a "sticky" IMAP connection to Courier-IMAP servers from outside. Because of the nature of Maildir, the usual NFS locking problem does not surface. Lots of way to do "sticky" connections, but if you want failover as weel as load ballancing, something like F5 of Cisco CSS is a best way to do it... > >At the moment we have about 30.000 Accounts with round about 150 GByte >mailspace, located on a NFS-storage with 7.200 RPM RAID-1-SATA-discs. We >have two Postfix- and two IMAP-Nodes. We have 3200 accounts with about 150GB mailspace, so i think i get give you pretty good number for that - 5 min 24hour average of 1600 Ops/s, max 5 min average of 4k Ops/s. From our MRTG. > >The performance on the discs are bad -- iostat shows up to 100% IO-usage. Using ext3 or ext2? Won't do - ext is not good with thousand of small files in a directory. I set up test server before MBOX to Maildir migration with just 20GB of data, and my test users complained it was slow - and it was. I would say XFS is your best chioce - about the same (or better) performance (according to postmark benchmark i ran) as ReiserFS (which is designed for small file performance) without any data loss problems ReiserFS had historically. > >For sure -- it's possibe to build a better NFS-storage with 15.000 RPM >SCSI -- but we have to scale up to 60.000 or 90.000 Accounts with 400 / >500 GByte Maildir-Data. But will that help? I can't believe that tuning >the discs gives enough performance boost. Otherwise: I can not believe >that it is necessary to buy a very expensive NetApp-Storage for just some >hundred gigs of Maildir-storage. Think again - i was trying to find something acceptable to replace our NetApp and couldn't. Nothing comes close to damn thing in term of Ops/s. Nothing less expensive is even in a ballpark. So, if you want to to use commodity storage, you either use direct attached and something like Cyrus+Murder to route IMAP connections to right IMAP server, or you split your users, use multiple commodity NFS, mount multiple NFS shares on each IMAP server, use LDAP alias lookup to route mail to right server. > >Looks like a frequently used Maildir-storage > 250 Gbyte is to hard for >less then 10 harddiscs, because it's to much I/O. Cache would give a >performance boost -- should I give my NFS-server > 10 GByte RAM just for >caching? > >Yes, it is possible to use an IMAP-proxy to spread the storage over >several IMAP-Nodes with local storage. But that includes to many single >point of failures und is very unflexible in the handling of performance >peaks. See http://cyrusimap.web.cmu.edu//ag.html > >Even if I would buy an expensive SAN (with fibre and all that expensive >stuff) it would be necessary to use a Cluster-FS and I'm not sure, wether >they will provide enough performance on many small files and I'm note >sure wether they will be able to handle the caching good enough. Is it >really necessary to buy a >50.000 $-SAN with >20 harddiscs just to handle >some hundred Gigs of Maildir-Data?!?! Well, i asked Redhat Engineers a year ago during Linux Expo about GFS performance on small files and it looks like they never considered that type of use for GSF. But as soon as GFS matures and tunable enough for Maildir scenario - that's what i'm doing. > >Otherwise NFS is not good in caching, too... > >I don't know what to do any more -- *every* possibility looks bad to me. >What's better? NFS? Any kind of Cluster-FS? A real SAN? RAID-5, RAID-1 >oder RAID-10? Is it necessary to switch back to several non-redundant >IMAP-nodes with local storage to have a better cache-performance? Or do I >have to run one 4-way server with 16 GByte memory to handle *all* that >connections on one single server?! > >I can't believe I'm the only one having this problem -- but I haven't >found real answers yet. Sure, there's a lot of documentation about what >is *possible*, but looks like nobody really *knows* how to handle that. > >Brian runs hundred of thousand accounts, he can buy a NetApp for that. But >what should I do with 60.000 - 90.000 frequently used IMAP-accounts? Well, our college just bought a new FAS250 with 4TB (2TB usable after RAID and snapshot allocation). Cost us less than $40k (educational pricing). Well worth to us compared with administration overhead of dealing with multiple (in our case i believe we would need 10) commodity servers running NFS. This gives our user atleast 350MB quota. > >Hope some people give me some hints about their configuration and >solutions... Details about size of storage and number of clients are >welcome... :-) I think my usage pattern should be similar to yours - about same ammount of data, just 10% of users compared to you, but very heavy ones (college). I gave you Ops/s stats before. At any time during peak hours we have 250 x 4 (4 servers) IMAP connections. Discounting multiple connections (some mail clients open up to 5 cons) it's about 700 users connected at any time (most use webmail). Each server has just 4GB of ram (peak RAM usage is 1GB if you don't count cache). And they run apache for personal web pages as well. So, RAM requirements for IMAP server is very low. Each server have 2 Xeon 2.8Ghz CPUs and ran Fedora core 3. Peak cpu utilization is 25%, 1 min 24 nour load average - under 2, 15min 24h - peaks at 1.4. > >Best regards, > >Peer Let me know if you need more info. cheers Vadim > >-- >Heinlein Professional Linux Support GmbH >Linux: Akademie - Support - Hosting > >http://www.heinlein-support.de > >Besuchen Sie uns: CeBIT 2007: Stand G64/3 im LinuxPark! > >Zwangsangaben lt. §35a HGB: >HRB 93818 B / Amtsgericht Berlin-Charlottenburg, >Geschäftsführer: Peer Heinlein -- Sitz: Berlin > >------------------------------------------------------------------------- >Take Surveys. Earn Cash. Influence the Future of IT >Join SourceForge.net's Techsay panel and you'll get the chance to share >your >opinions on IT & business topics through brief surveys-and earn cash >http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV >_______________________________________________ >Courier-imap mailing list >[email protected] >Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-imap ------------------------------------------------------------------------- Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT & business topics through brief surveys-and earn cash http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV _______________________________________________ Courier-imap mailing list [email protected] Unsubscribe: https://lists.sourceforge.net/lists/listinfo/courier-imap
