Re: imap server performance benchmarks
On Fri, 09 Mar 2012 19:17:15 -0600, Da Rock wrote: Yes, thats true. That was tested in the paper: a cyrus? using sql database backend performed faster in searches and lookups. But writing and deleting was a drag, and you lose the shell; but I'm not sure that thats such a problem as one could find tools in the sql commands (provided you know databases well enough). Since Archiveopteryx is so tightly integrated with Postgres, this seems to be less of a problem. From their FAQ[1]: Some question about capacity.This question crops up in different shapes — “how many users?”, “how big?” Archiveopteryx's bottleneck is the number of deliveries per minute, everything else is irrelevant. How many messages do you need to inject into the database in the busiest five-minute period of theday? In a business, that's usually in the morning and immediately after lunch. On fast PC hardware,Archiveopteryx currently handles in the neighbourhood of 4000 deliveries per minute. Wayback Machine has this FAQ entry going back to 2007. I'm pretty sure that on current hardware we can do more than 4000 messages per minute. On the topic of deletes: They're pretty fast in AOX. Deletion is only a flag and a nightly cron does the real purging. You set a retention policy -- you choose how long the email stays in the DB before it's actually purged. It's pretty slick, and I like setting things like forced deletion of all emails in my SPAM folder if they're older than 30 days, and my other mailboxes I can undelete up to 14 days after. It's saved my butt once or twice. I'd love to have this for our customer's email. The real problem when you start dipping into this type of an environment is figuring out how to support it. You're no longer running a mail server; you're now a DBA. If I implemented this at work I have three hurdles: 1) Not pissing anyone off when they find out their GPG is broken (low likelihood, but it's naughty to do this. FYI, they're working on a fix but it has significant hurdles.) 2) We're now admins of a 120GB Postgres database. This is a daunting task, and the hardware requirements are more than if you were just running Dovecot/Cyrus. (AOX does dedup and my estimate brings this down to ~100GB, but I don't know how the big indexes will be) 3) Well now we probably want a slave so backups don't lock the tables at night I absolutely love the idea, but outside of my own email or hosting for a friend I don't think it's a feasible solution, which saddens me... a few more devs and the project could really shine. [1] http://archiveopteryx.org/faq/mailstore#capacity ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: imap server performance benchmarks
On 03/10/12 08:21, Mark Felder wrote: On Fri, 09 Mar 2012 14:33:53 -0600, Weldon Godfrey wrote: I would highly recommend looking at Dovecot. I have used courier for several ISPs then switched to Dovecot on my last install. Its ability for caching the index per user is of great performance advantage if you choose to leverage an IMAP based webmail solution (like Roundcube, or even Squirrelmail, but I would recommend Roundcube over Squrrel)...especially if you are going to deal with mailboxes of tens of thousands of messages. I concur. We use Dovecot at work and even put the indexes on Intel SSDs. My boss's mailbox is actually 1.2 million messages. That's pretty insane. For my personal email I prefer Archiveopteryx which sanitizes, normalizes, and deduplicates your email. There are caveats (breaks gpg), but I find it to be wicked fast -- faster than Dovecot on my server. Here's an annoying problem: You have a filesystem with 9 million messages in Maildirs. Backing this up or even rsyncing this sucks. Now imagine being able to use database replication and use database dumps for your backups. Pretty slick. You lose the beautiful power of shell utilities, though Yes, thats true. That was tested in the paper: a cyrus? using sql database backend performed faster in searches and lookups. But writing and deleting was a drag, and you lose the shell; but I'm not sure that thats such a problem as one could find tools in the sql commands (provided you know databases well enough). My idea was to run some tests based on extremes, but by the sounds of some of these replies I'm a small fry it seems, not as extreme as I thought... :) ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: imap server performance benchmarks
On Fri, 09 Mar 2012 14:33:53 -0600, Weldon Godfrey wrote: I would highly recommend looking at Dovecot. I have used courier for several ISPs then switched to Dovecot on my last install. Its ability for caching the index per user is of great performance advantage if you choose to leverage an IMAP based webmail solution (like Roundcube, or even Squirrelmail, but I would recommend Roundcube over Squrrel)...especially if you are going to deal with mailboxes of tens of thousands of messages. I concur. We use Dovecot at work and even put the indexes on Intel SSDs. My boss's mailbox is actually 1.2 million messages. That's pretty insane. For my personal email I prefer Archiveopteryx which sanitizes, normalizes, and deduplicates your email. There are caveats (breaks gpg), but I find it to be wicked fast -- faster than Dovecot on my server. Here's an annoying problem: You have a filesystem with 9 million messages in Maildirs. Backing this up or even rsyncing this sucks. Now imagine being able to use database replication and use database dumps for your backups. Pretty slick. You lose the beautiful power of shell utilities, though ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
RE: imap server performance benchmarks
-Original Message- From: owner-freebsd-questi...@freebsd.org [mailto:owner-freebsd-questi...@freebsd.org] On Behalf Of Bill Campbell Sent: Friday, March 09, 2012 1:46 PM To: freebsd-questions@freebsd.org Subject: Re: imap server performance benchmarks On Fri, Mar 09, 2012, Da Rock wrote: > I'm reconsidering my current setup (postfix/courier) for imap and I was > doing some research on performance comparisons between imap server > setups. I stumbled on this article just just about fell of my chair > laughing when I read the last article on future benchmarking tests to > perform: > > research.microsoft.com/pubs/138302/lisa.pdf > > Considering I have close to a hundred folders or more, and an average of > 50,000 emails in each (yes, not good, and I am working on archiving but > it won't help _that_ much) with nearly 200,000 in just one! I got a real > kick out of the comment that "no sane email user would have more than > 21,000 emails in a folder" - that would make me certifiable :D Oh, and > that most email wouldn't be more than a GB or so... mine's edging 6GB > already... I would highly recommend looking at Dovecot. I have used courier for several ISPs then switched to Dovecot on my last install. Its ability for caching the index per user is of great performance advantage if you choose to leverage an IMAP based webmail solution (like Roundcube, or even Squirrelmail, but I would recommend Roundcube over Squrrel)...especially if you are going to deal with mailboxes of tens of thousands of messages. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: imap server performance benchmarks
On Fri, Mar 09, 2012, Da Rock wrote: > I'm reconsidering my current setup (postfix/courier) for imap and I was > doing some research on performance comparisons between imap server > setups. I stumbled on this article just just about fell of my chair > laughing when I read the last article on future benchmarking tests to > perform: > > research.microsoft.com/pubs/138302/lisa.pdf > > Considering I have close to a hundred folders or more, and an average of > 50,000 emails in each (yes, not good, and I am working on archiving but > it won't help _that_ much) with nearly 200,000 in just one! I got a real > kick out of the comment that "no sane email user would have more than > 21,000 emails in a folder" - that would make me certifiable :D Oh, and > that most email wouldn't be more than a GB or so... mine's edging 6GB > already... My security folder averages about 19,000 messages, containing all security reports for the last 30 days (it's a bit over 22,000 today). > So, all jokes aside, I contemplated that I would make an ideal test case > to the extreme for benchmarking imap servers. Anyone have any > suggestions on what to test/how? Anyone have some tools they have > created for a similar challenge? I have my own ideas, but if anyone > wants me to try something I'd be willing to give it a shot. We have been using courier-imap for at least a decade including at several regional ISPs with 10,000 accounts or so. This has worked very well with minimal problems. During this time we have had to move all e-mail accounts to new servers, and this has been pretty simple. Set up the accounts and $HOME directories on the new machine. Change the DNS to point to it. Then copy all the data from the old box to the new using rsync. This would take a couple of hours with 10,000 accounts. During this time all new mail was processed and available immediately while older messages weren't until the rsync jobs were complete. When we first started building systems for ISPs in 1994, we were using the University of Washington IMAP server, and had a variety of problems as systems grew. Its monolithic file storage required special handling, and I learned about inetd limits on handling large numbers of incoming connections the hard way. I much prefer the Maildir stores as they allow use of standard *nix utilities, find, grep, etc., are hard to corrupt, and have no locking problems with NFS mounts. The courier method of having multiple authentication processes has held up very well over the years under heavy usage. Bill -- INTERNET: b...@celestial.com Bill Campbell; Celestial Software LLC URL: http://www.celestial.com/ PO Box 820; 6641 E. Mercer Way Voice: (206) 236-1676 Mercer Island, WA 98040-0820 Fax:(206) 232-9186 Skype: jwccsllc (206) 855-5792 Now if there is one thing that we do worse than any other nation, it is try and manage somebody else's affairs. Will Rogers ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: imap server performance benchmarks
--As of March 9, 2012 12:44:55 PM +1000, Da Rock is alleged to have said: I'm reconsidering my current setup (postfix/courier) for imap and I was doing some research on performance comparisons between imap server setups. I stumbled on this article just just about fell of my chair laughing when I read the last article on future benchmarking tests to perform: research.microsoft.com/pubs/138302/lisa.pdf Considering I have close to a hundred folders or more, and an average of 50,000 emails in each (yes, not good, and I am working on archiving but it won't help _that_ much) with nearly 200,000 in just one! I got a real kick out of the comment that "no sane email user would have more than 21,000 emails in a folder" - that would make me certifiable :D Oh, and that most email wouldn't be more than a GB or so... mine's edging 6GB already... So, all jokes aside, I contemplated that I would make an ideal test case to the extreme for benchmarking imap servers. Anyone have any suggestions on what to test/how? Anyone have some tools they have created for a similar challenge? I have my own ideas, but if anyone wants me to try something I'd be willing to give it a shot. --As for the rest, it is mine. No idea, but as someone who recently had to trim one of his mail folders (same setup) as it was having trouble with over 210,000 messages, I'd be interested in your results. ;) Daniel T. STaal --- This email copyright the author. Unless otherwise noted, you are expressly allowed to retransmit, quote, or otherwise use the contents for non-commercial purposes. This copyright will expire 5 years after the author's death, or in 30 years, whichever is longer, unless such a period is in excess of local copyright law. --- ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
Re: imap server performance benchmarks
On 09/03/2012 03:44, Da Rock wrote: I'm reconsidering my current setup (postfix/courier) for imap and I was doing some research on performance comparisons between imap server setups. I stumbled on this article just just about fell of my chair laughing when I read the last article on future benchmarking tests to perform: research.microsoft.com/pubs/138302/lisa.pdf Considering I have close to a hundred folders or more, and an average of 50,000 emails in each (yes, not good, and I am working on archiving but it won't help _that_ much) with nearly 200,000 in just one! I got a real kick out of the comment that "no sane email user would have more than 21,000 emails in a folder" - that would make me certifiable :D Oh, and that most email wouldn't be more than a GB or so... mine's edging 6GB already... So, all jokes aside, I contemplated that I would make an ideal test case to the extreme for benchmarking imap servers. Anyone have any suggestions on what to test/how? Anyone have some tools they have created for a similar challenge? I have my own ideas, but if anyone wants me to try something I'd be willing to give it a shot. If anyone has a better idea on which list this should be posted to as well - I considered the lists available (I'm hooked up to most) and couldn't see any better. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org" No IMAP test is as vicious or as thorough as a real life company deciding to change its mail client from one day to the next and counting on IMAP to automagically restore local archives. If the company more or less uses IMAP folder as a share drives it is even better. It happened to me once. Postfix/Dovecot did handle the change quite well, yet some mailboxes took days before the local copy was in sync with IMAP folders. There was about 200GB of mail to download (35 users company) the load average was under 0.25 all the time on an i5 dual core with 8GB of ram. Duplicating a mailbox X times and having X clients doing a local copy of the entire mailbox sounds like a good first test, with mailbox size and number X on par with what you expect to find on your network. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
imap server performance benchmarks
I'm reconsidering my current setup (postfix/courier) for imap and I was doing some research on performance comparisons between imap server setups. I stumbled on this article just just about fell of my chair laughing when I read the last article on future benchmarking tests to perform: research.microsoft.com/pubs/138302/lisa.pdf Considering I have close to a hundred folders or more, and an average of 50,000 emails in each (yes, not good, and I am working on archiving but it won't help _that_ much) with nearly 200,000 in just one! I got a real kick out of the comment that "no sane email user would have more than 21,000 emails in a folder" - that would make me certifiable :D Oh, and that most email wouldn't be more than a GB or so... mine's edging 6GB already... So, all jokes aside, I contemplated that I would make an ideal test case to the extreme for benchmarking imap servers. Anyone have any suggestions on what to test/how? Anyone have some tools they have created for a similar challenge? I have my own ideas, but if anyone wants me to try something I'd be willing to give it a shot. If anyone has a better idea on which list this should be posted to as well - I considered the lists available (I'm hooked up to most) and couldn't see any better. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"
