Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Am Donnerstag, den 24.06.2010, 22:30 -0500 schrieb Stan Hoeppner: Does running imapproxy in this scenario yield any benefit, given there is no network traffic involved? it does. IMAPPRoxy caches for example all directories. If the user open a folder, then you have a new connection to the IMAP server, that can take a while (auth ...). After a few seconds the connection is closed (that's quite normal for webapplications like webmail etc.) and than the user requests a new folder ... . So the imapproxy can cache all these things and make the Webmail faster. The only problem is, if there are new folders, so maybe it can take a few minutes, before the folder is visible. cu denny signature.asc Description: Dies ist ein digital signierter Nachrichtenteil
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Fri, 25 Jun 2010, Denny Schierz wrote: it does. IMAPPRoxy caches for example all directories. If the user open a folder, then you have a new connection to the IMAP server, that can take a while (auth ...). After a few seconds the connection is closed (that's quite normal for webapplications like webmail etc.) and than the user requests a new folder ... . So the imapproxy can cache all these things and make the Webmail faster. The only problem is, if there are Hmm, IMHO Dovecot is to cache most of the stuff, isn't it? Esp. LOGIN user pass is cheap, because the password is cached. new folders, so maybe it can take a few minutes, before the folder is visible. imapproxy caches the output of LIST? On the website the FAQ only talks about connections, for me it reads that imapproxy saves to open another TCP connection, the possible creation of other Dovecot processes and the LOGIN command. Regards, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBTCRcy7+Vh58GPL/cAQLkxgf/cohz4dHwX6gFAVQxEmWIg/rNXHDe+Oll iHxJ0DFXTSvkLRICoIyKyYFeHx6iQgRfQ+Nu8tLw+OEZrL/QZT4hgxYqIXm+8HID azsf3mIQeLEMd7X7er2F+xWJwmhPEWj9klxDXkAXjdhIVOdXQ9Nc28dfPbqQUgD0 UpyRwDANTjLz5YU+Itgc3dPMtokkkW7Fl6xu9DdnyXkwi3GOGpNjrpL+dSO3vaZR JTT/mwA+1K3WKUiT8m7M7ytbFMLXJNALPZxAUCoTTXvwdInrED/5gcsLiXmXOQqx rFM3Q+sfGdNiwAQ3ileVpG3d9OR/uvgUbyfzIuPcVObw0eQ8qU4E8Q== =gZIq -END PGP SIGNATURE-
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 25.6.2010, at 8.04, Denny Schierz wrote: Am Donnerstag, den 24.06.2010, 22:30 -0500 schrieb Stan Hoeppner: Does running imapproxy in this scenario yield any benefit, given there is no network traffic involved? it does. IMAPPRoxy caches for example all directories. If the user open a folder, then you have a new connection to the IMAP server, that can take a while (auth ...). After a few seconds the connection is closed (that's quite normal for webapplications like webmail etc.) and than the user requests a new folder ... . So the imapproxy can cache all these things and make the Webmail faster. The only problem is, if there are new folders, so maybe it can take a few minutes, before the folder is visible. Last I looked, imapproxy had some kind of a SELECT cache and possibly something else. They were implemented in an unreliable way and unless something has changed, I wouldn't recommend enabling them. Also I doubt any of them help with performance with Dovecot. Most people seem to be saying imapproxy helps a lot, because they noticed it helped so much with Courier/UW-IMAP..
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 25/06/2010 13:46, Timo Sirainen wrote: Most people seem to be saying imapproxy helps a lot, because they noticed it helped so much with Courier/UW-IMAP.. Agree.
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 2010-06-24 9:18 PM, Timo Sirainen wrote: But with auth cache enabled, there is no extra database load. The index files are also most likely in OS's cache (assuming local disk), so no extra disk I/O to read them either. I'm sure it's a bit more extra CPU usage, but I'm not all that certain that it's really that a big of a difference with Dovecot. We have very few web users, so I really didn't see anything noticeable performance wise, but I'll always use it just for the much cleaner logs. -- Best regards, Charles
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On Friday 25 June 2010 wrote Charles Marcus: On 2010-06-24 9:18 PM, Timo Sirainen wrote: But with auth cache enabled, there is no extra database load. The index files are also most likely in OS's cache (assuming local disk), so no extra disk I/O to read them either. I'm sure it's a bit more extra CPU usage, but I'm not all that certain that it's really that a big of a difference with Dovecot. We have very few web users, so I really didn't see anything noticeable performance wise, but I'll always use it just for the much cleaner logs. For whatever it is worth, we use imapproxy, because it allows us to use one-time passwords for webmail users. One login, not many. Amon Ott -- Dr. Amon Ott - m-privacy GmbH Am Köllnischen Park 1, 10179 Berlin Tel: +49 30 24342334 Fax: +49 30 24342336 Web: http://www.m-privacy.de Handelsregister: Amtsgericht Charlottenburg HRB 84946 Geschäftsführer: Dipl.-Kfm. Holger Maczkowsky, Roman Maczkowsky GnuPG-Key-ID: EA898571
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On Fri, 2010-06-25 at 14:44 +0200, Amon Ott wrote: For whatever it is worth, we use imapproxy, because it allows us to use one-time passwords for webmail users. One login, not many. With large enough auth cache, I think that should also work with Dovecot directly (although if the session takes longer than auth_cache_ttl, the user needs to re-login).
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Quoting Eric Rostetter rostet...@mail.utexas.edu: Quoting Daniel L. Miller dmil...@amfes.com: I know when I was playing web clients - particularly squirrelmail - there was a definite perceived improvement - but I never measured it. Webmail clients are basically stateless. Over time, they have cached some info for performance, but basically they are still stateless. So almost any time you click on a link dealing with an email (delete, purge, next message, forward, etc) it has to open a _new_ connection to the imap (or pop) server, which means a new login. Simple proxies (like say perdition) don't help, as each connection will still be a new login. Some proxies (like imapproxy) however can keep a login session open, with the proxy caching the authentication issue. For each connection after the first, it can do the authentication itself and use the existing login connection to the pop/imap server, avoiding a login for each operation after the first (until a timeout is reached). By avoiding the constant login/logout cycle, it will generally perform at least slightly better with the proxy than without (no new connection overhead, no login and logout overhead, same over head most everywhere else unless the proxy's authentication is slow). However, imapproxy (at least as of v1.2.6) is still less than desirable. Even though you are saving on the authentication overhead, every time your stateless client (i.e. webmail) connects it still needs to potentially issue a CAPABILITY command and/or other enabling commands (e.g. 'ENABLE QRESYNC'). So that's still a bunch of overhead that could potentially be saved. Solution: make the imapproxy layer visible to the client. Patches were added to imapproxy v1.2.7 to do just that. When IMAPPROXY is being used, XIMAPPROXY is announced via the IMAP capability string. More important, if the cached proxy connection is successfully reused, a XPROXYREUSE status response is returned after the authentication commands are completed. If this status return is observed by the client, it knows that it can reuse its cached information about the connection (if it exists locally on the client) without needing to check on CAPABILITIES or issuing ENABLING commands. Additionally, it can prevent unnecessary initialization code on the client side that only needs to be done when a user logs into the server. IMP 5 (more specifically, the Horde 4: Horde_Imap_Client socket driver) does just this to prevent unnecessary traffic to the IMAP server. michael
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Rajesh M wrote: Rajesh M wrote: Rajesh M wrote: eric i studied LDA a bit if i use lda that means all my 5000+ users' email index files will be continuously updated when every email arrives -- means a lot of writes to disk ... is that correct ? Yes, but I think you make it sound worse than it is. Updating the index as each email arrives doesn't involve rebuilding the whole index like what happens at login. It only updates what's there instead of rebuilding the whole thing. I can't speak from experience, but I expect that it's very efficient, and you'll never notice any increased load on the machine. Perhaps the equivalent of adding journaling to a filesystem. You can ask the dovecot list to verify this. Also, it just occurred to me that if you're using ext3, you should be sure to optimize the filesystem. Off hand, I'd use the noatime and nodiratime mount options (man mount) and dir_index filesystem option (man tune2fs). if yes then i dont want that since only around 200 or so people out of the above 5000 use webmail. Just to be clear, it affects all imap usage, not only webmail (although your uses may not use imap other than webmail). i will study a some more and revert I hope you can figure a way to try this out. I expect that QMT's move to dovecot in the future will include the deliver LDA, and your work here will be a big help getting us there. Let me know if I can be of any further help. -- -Eric 'shubes' hi eric few questions i wil enable lda exclusively for this one 30 gb email box using .qmail file see the performance improvement .. is that ok ? I believe that would be fine. can i safely enable atime for this user maildir ? Sure. That's simply a performance consideration. I don't believe that atime is used for much of anything, so either atime or noatime is safe. noatime is just a little faster because the inode (where that piece of information is stored) doesn't need to be updated each time a file is accessed. thanks for your help Sure. I'm eager to see your results. -- -Eric 'shubes' hi eric this is what i plan to do as per http://wiki.dovecot.org/LDA/Qmail i need to enter the following line in a .qmail file under the Maildir directory of the user |/var/qmail/bin/preline -f /usr/local/libexec/dovecot/deliver -d $...@$user the location is /usr/libexec/dovecot/deliver on my system. i would add an email forward using qmailadmin web interface (which creates the .qmail file) and then manually edit the .qmail file and put the above line in it (so that ownership and permissions are taken care off) That sounds ok. The permissions/ownership I'm seeing on .qmail files is: -rw--- 1 vpopmail vchkpw 101 Mar 26 2008 .qmail now i would also need to edit the dovecot.conf file could you guide me on what changes do i need to dovecot.conf file ? Others on this list would be more help than me regarding this. From the looks of it though (after browsing the dovecot.conf file, which is well documented btw) I don't think you'd need to change a thing. We'll certainly find out if that's the case or not. I'd just give it a go. Perhaps on a test account initially. ;) -- -Eric 'shubes'
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On Fri, 2010-06-25 at 13:49 +0100, Timo Sirainen wrote: On Fri, 2010-06-25 at 14:44 +0200, Amon Ott wrote: For whatever it is worth, we use imapproxy, because it allows us to use one-time passwords for webmail users. One login, not many. With large enough auth cache, I think that should also work with Dovecot directly (although if the session takes longer than auth_cache_ttl, the user needs to re-login). As I mentioned earlier It doesnt with squirrelmail, I tried it on production and the I/O increased a bit, watching logging alone was giving me a headache (no, not in debug mode either).. So I thought I'd try it on a dev box, in case the cache wasnt large enough, but nope, incidentally imap with evolution works as you indicate it should, but webmail, not a hope in . it is login -action logout constantly, imapproxy avoids all this crud, so it obviously does something differently, maybe you find that and incorporate it in dovecot :) that way all this login logout stuff is only seen in say, debug mod? Though the overhead for running imapproxy is non existent it is still an needed devil in a busy environment. attachment: stock_smiley-1.png
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 26.6.2010, at 0.57, Noel Butler wrote: As I mentioned earlier It doesnt with squirrelmail, I tried it on production and the I/O increased a bit, watching logging alone was giving me a headache (no, not in debug mode either).. How much is a bit that the I/O increased? Less than 10%? So I thought I'd try it on a dev box, in case the cache wasnt large enough, but nope, incidentally imap with evolution works as you indicate it should, but webmail, not a hope in . it is login -action logout constantly, imapproxy avoids all this crud, so it obviously does something differently, maybe you find that and incorporate it in dovecot :) that way all this login logout stuff is only seen in say, debug mod? It still sounds like the only problem you have is the excessive log messages?.. Sure, Dovecot doesn't even try to get rid of those.
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
apologies, I sent this direct to Eric not the list ore OP, my bad :) On Fri, 2010-06-25 at 13:46 -0700, Eric Shubert wrote: snipped 7 pages of irrelevant un-trimmed crap now i would also need to edit the dovecot.conf file could you guide me on what changes do i need to dovecot.conf file ? Others on this list would be more help than me regarding this. From the looks of it though (after browsing the dovecot.conf file, which is well documented btw) I don't think you'd need to change a thing. We'll certainly find out if that's the case or not. I'd just give it a go. Perhaps on a test account initially. ;) With Qmail (in particular if using vpopmail) from memory you need to include first_valid_uid and first_valid_gid options i attachment: stock_smiley-1.png
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On Sat, 2010-06-26 at 01:02 +0100, Timo Sirainen wrote: On 26.6.2010, at 0.57, Noel Butler wrote: As I mentioned earlier It doesnt with squirrelmail, I tried it on production and the I/O increased a bit, watching logging alone was giving me a headache (no, not in debug mode either).. How much is a bit that the I/O increased? Less than 10%? perhaps around the 10-15% (see below) So I thought I'd try it on a dev box, in case the cache wasnt large enough, but nope, incidentally imap with evolution works as you indicate it should, but webmail, not a hope in . it is login -action logout constantly, imapproxy avoids all this crud, so it obviously does something differently, maybe you find that and incorporate it in dovecot :) that way all this login logout stuff is only seen in say, debug mod? It still sounds like the only problem you have is the excessive log messages?.. Sure, Dovecot doesn't even try to get rid of those. this likely is the sole reason for the I/O increase, but yes. Oh and the local (replicated) DB was seeing maybe 40 requests a second rather than only a few, no big deal there though.
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Timo, On 6/24/10 4:23 AM, Timo Sirainen t...@iki.fi wrote: I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail. Do you have any about a real world numbers about installation with and without imapproxy? We run imapproxy behind our Roundcube instance, and our old in-house Perl mail system has a custom equivalent written in C that also does some caching of folder metadata and message headers. We run a proxy instance on each of the webmail hosts, with the communication between the web application add the proxy being done in cleartext, but with the proxy - Dovecot communication secured over SSL. Besides preventing a lot of extra SSL handshakes and login/logout actions, it also helps tie a user session to a single backend node in our pool of IMAP servers. It seems like there might also be other benefits to having Dovecot not tear down all of the user session state between page loads. A lot of this stuff might be nice to see in the Director some day. If there was an option to not immediately close the Director proxy's backend connections when the user logs out, (ie leave the connection active and logged in for X seconds, and reuse it if the user logs in to the Director again), and if the auth caching works as well as you say, then I could definitely see a day where we replace imapproxy with a director instance on the webmail host. -Brad
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 26.6.2010, at 3.08, Brandon Davidson wrote: We run a proxy instance on each of the webmail hosts, with the communication between the web application add the proxy being done in cleartext, but with the proxy - Dovecot communication secured over SSL. Besides preventing a lot of extra SSL handshakes and login/logout actions, Yes, SSL handshakes are extra. Although SSL supports some kind of quick renegotiation too, but Dovecot doesn't support that yet. No one's ever requested it.. it also helps tie a user session to a single backend node in our pool of IMAP servers. It seems like there might also be other benefits to having Dovecot not tear down all of the user session state between page loads. Some, but I suspect mostly CPU performance related (which usually doesn't matter). A lot of this stuff might be nice to see in the Director some day. If there was an option to not immediately close the Director proxy's backend connections when the user logs out, (ie leave the connection active and logged in for X seconds, and reuse it if the user logs in to the Director again), It hasn't really ever been in my plans to do anything like this. A separate imapproxy program for this is probably better. At least I don't really see any benefits compared to it if it was done by Dovecot. I mainly just wish someone would give some kind of real numbers when running with and without imapproxy with Dovecot (and not some other server they used to run years ago) when they're talking about imapproxy helping the performance a lot. So far the best reports I've heard are imapproxy improves a little bit, but adds too much complexity/fragility. So I'm a bit sceptical about using it. I've also recently heard from a large installation who was complaining about Dovecot's memory usage (compared to Courier). imapproxy would definitely make their memory usage a lot higher, since the connections would stay around for longer. Although I'm still wondering why Dovecot would take so much more memory with large maildirs .. should look into that some day.
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
stan you are right ... i am running qmail toaster for years now and this is production mail server with over 5000 email boxes .. i dont want to migrate at all. i recently upgraded to dovecot only for the imap part because courier was trashing my server .. i was getting load levels of over 30-40 during peak times once courier was implemented my load levels were down to around 2-3 i am using squirrel mail and it has server side sort enabled most users of mine have at the most around 100 to 200 mb of emails and just a few users having 3-4 gb boxes i am absolutely happy with the performance of dovecot a 3.8 gb mail box opens in 8-10 seconds the first time and next time in around 3 seconds this 30 gb email box test is just to see how best dovecot can perform my system is on centos with ext3 and is using 60 percent of drive space so i believe it shud not be fragmented i also feel that this is related to index and cache which is getting expired or something like that. i will try out what you suggested and revert rajesh Ed W put forth on 6/23/2010 4:18 PM: Secondly 7,500 mails over 5 mins means an indexing rate of 25 mails/sec. This would not be out of order for a heavily fragmented drive which is IO bound? Each file needs to be opened to scan the headers so likely you need one disk seek and I guess it's easy to be IO bound? What does iotop show you during dovecot's thrashing? It seems he is I/O bound a degree. If I read his answer to my disk subsystem question correctly he's storing user maildirs on a single local 1TB SATA drive. However, given that his 2nd successive login is 4-5 seconds instead of 5 minutes, it would seem index and cache being current are the problem, not I/O saturation. Faster disk would always help, but it's not close to a total solution to his problem. Putting his maildir on a 16 disk RAID 0 stripe of the same model 1TB disk he already has would yield a 16x improvement in seek throughput, cutting his 'stale' login time to ~20 seconds, if my math is correct. 20 seconds is still unacceptably high IMHO, though it's much better than 300 seconds. Ok so lets assume the filesystem underlying his maildir is heavily fragmented and that defragging it would yield a 100% improvement for argument sake (50% improvement is almost unheard of, normal is about 20%). He'd still be looking at a 10 second login time after spending anywhere from $3k to $8k USD on a 16 disk array depending on what vendor he chooses. Throwing money and hardware at this problem isn't the proper or optimal solution. Dovecot2 has an mdbox option which sounds like it could be beneficial for your performance requirements (but it's not stable yet) I don't think migrating to a new mailbox format is what he really needs, or wants. IIRC he's been using qmail for years, which means he's been using maildir for years. He's likely very comfortable with maildir and probably wants to stick with it. Otherwise I guess you need to investigate dovecot's delivery agent which does incremental index updates at deliver time (pay the cost one email at a time rather than every 7,500 emails). Or consider alternative filesystems which are more performant for this requirement? On this I completely agree with you, and I suggested it previously in the same thread I suggested dirty_syncs. This should be his next step. If LDA alone doesn't fix the problem, then he should try dirty_syncs again in conjunction with LDA. -- Stan
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 24.6.2010, at 5.45, Daniel L. Miller wrote: On 6/19/2010 7:33 PM, Rajesh M wrote: setup dovecot with squirrelmail and logged in I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail. Do you have any about a real world numbers about installation with and without imapproxy?
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Rajesh M wrote: Rajesh M wrote: eric i studied LDA a bit if i use lda that means all my 5000+ users' email index files will be continuously updated when every email arrives -- means a lot of writes to disk ... is that correct ? Yes, but I think you make it sound worse than it is. Updating the index as each email arrives doesn't involve rebuilding the whole index like what happens at login. It only updates what's there instead of rebuilding the whole thing. I can't speak from experience, but I expect that it's very efficient, and you'll never notice any increased load on the machine. Perhaps the equivalent of adding journaling to a filesystem. You can ask the dovecot list to verify this. Also, it just occurred to me that if you're using ext3, you should be sure to optimize the filesystem. Off hand, I'd use the noatime and nodiratime mount options (man mount) and dir_index filesystem option (man tune2fs). if yes then i dont want that since only around 200 or so people out of the above 5000 use webmail. Just to be clear, it affects all imap usage, not only webmail (although your uses may not use imap other than webmail). i will study a some more and revert I hope you can figure a way to try this out. I expect that QMT's move to dovecot in the future will include the deliver LDA, and your work here will be a big help getting us there. Let me know if I can be of any further help. -- -Eric 'shubes' hi eric few questions i wil enable lda exclusively for this one 30 gb email box using .qmail file see the performance improvement .. is that ok ? I believe that would be fine. can i safely enable atime for this user maildir ? Sure. That's simply a performance consideration. I don't believe that atime is used for much of anything, so either atime or noatime is safe. noatime is just a little faster because the inode (where that piece of information is stored) doesn't need to be updated each time a file is accessed. thanks for your help Sure. I'm eager to see your results. -- -Eric 'shubes' hi eric this is what i plan to do as per http://wiki.dovecot.org/LDA/Qmail i need to enter the following line in a .qmail file under the Maildir directory of the user |/var/qmail/bin/preline -f /usr/local/libexec/dovecot/deliver -d $...@$user i would add an email forward using qmailadmin web interface (which creates the .qmail file) and then manually edit the .qmail file and put the above line in it (so that ownership and permissions are taken care off) now i would also need to edit the dovecot.conf file could you guide me on what changes do i need to dovecot.conf file ? rajesh
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 6/24/2010 4:23 AM, Timo Sirainen wrote: I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail. Do you have any about a real world numbers about installation with and without imapproxy? What, you want me to actually back up that statement with data?! Who do you think I am?! Never mind - don't answer that. I know when I was playing web clients - particularly squirrelmail - there was a definite perceived improvement - but I never measured it. -- Daniel
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On Thu, 2010-06-24 at 16:58 -0700, Daniel L. Miller wrote: On 6/24/2010 4:23 AM, Timo Sirainen wrote: I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail. Do you have any about a real world numbers about installation with and without imapproxy? What, you want me to actually back up that statement with data?! Who do you think I am?! Never mind - don't answer that. I know when I was playing web clients - particularly squirrelmail - there was a definite perceived improvement - but I never measured it. Since every request (enter message, delete, forward, reply, etc etc etc) is a login - action - logout, it stands to reason on busy servers this will impact greatly, I saw a remarkable increase when using flat file years ago when changing to using imapproxy, but for many years now, using MySQL, it also was very noticeable improvement using it since it did not need to make all those extra concurrent requests. If its running on a SOHO, you probably wouldn't even be able to measure the difference, but for an ISP/Telco or large institution, you will certainly notice the reduction in I/O (or loads on your database server). This goes for any imap webmail software, not just SM. The advantage to this combination since about 2 months ago, the Squirrelmail team now also look after the imapproxy project :) attachment: stock_smiley-1.png
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 25.6.2010, at 1.37, Noel Butler wrote: If its running on a SOHO, you probably wouldn't even be able to measure the difference, but for an ISP/Telco or large institution, you will certainly notice the reduction in I/O (or loads on your database server). This goes for any imap webmail software, not just SM. But with auth cache enabled, there is no extra database load. The index files are also most likely in OS's cache (assuming local disk), so no extra disk I/O to read them either. I'm sure it's a bit more extra CPU usage, but I'm not all that certain that it's really that a big of a difference with Dovecot.
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
In my measurements I did years ago, I found imapproxy to give a small improvement compared to 0.9x dovecot, I believe that went away in 1.x versions. I did use imapproxy on a few systems. But over time I finally gave up on it, mainly cause it kept crashing, causing webmail to stop working, and I didn't feel adding a watcher or looping it to keep it running to be the correct fix. And given the very small improvment it gave, felt it wasn't worth the hassle of dealing with. Quoting Timo Sirainen t...@iki.fi: On 25.6.2010, at 1.37, Noel Butler wrote: If its running on a SOHO, you probably wouldn't even be able to measure the difference, but for an ISP/Telco or large institution, you will certainly notice the reduction in I/O (or loads on your database server). This goes for any imap webmail software, not just SM. But with auth cache enabled, there is no extra database load. The index files are also most likely in OS's cache (assuming local disk), so no extra disk I/O to read them either. I'm sure it's a bit more extra CPU usage, but I'm not all that certain that it's really that a big of a difference with Dovecot.
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On Fri, 2010-06-25 at 02:18 +0100, Timo Sirainen wrote: On 25.6.2010, at 1.37, Noel Butler wrote: If its running on a SOHO, you probably wouldn't even be able to measure the difference, but for an ISP/Telco or large institution, you will certainly notice the reduction in I/O (or loads on your database server). This goes for any imap webmail software, not just SM. But with auth cache enabled, there is no extra database load. The index files are also most likely in OS's cache (assuming local disk), so no extra disk I/O to read them either. I'm sure it's a bit more extra CPU usage, but I'm not all that certain that it's really that a big of a difference with Dovecot. auth_cache_size = 512 auth_cache_ttl = 3600 auth_cache_negative_ttl = 0 So, we use it at 512KB, perhaps this value was just too small, but admittedly, this change was made back in the early 0.9 days, and since imapproxy cured the problems, we (ok.. I :-) decided it shall be mandatory with all webmail servers since... I'll drop it off on one of them for a day or so and keep an eye on it.
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Timo Sirainen put forth on 6/24/2010 6:23 AM: On 24.6.2010, at 5.45, Daniel L. Miller wrote: On 6/19/2010 7:33 PM, Rajesh M wrote: setup dovecot with squirrelmail and logged in I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail. Do you have any about a real world numbers about installation with and without imapproxy? IIRC the OP is running SM and Dovecot on the same host. Does running imapproxy in this scenario yield any benefit, given there is no network traffic involved? -- Stan
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 23 Jun 2010, Rajesh M wrote: Roundcube 0.3.1 seems to have a hard coded limit of 200 mails per page. it again took around 4-5 mins I do not know nothing about Roundcube, therefore I wonder where the 5min delay come from: a) Do you keep index files on Dovecot side? b) Do you deliver / manipulate the mailbox via Dovecot only? E.g. do you use Dovecot deliver? c) If you rawlog or sniff the connection and issue the same commands as roundcube to Dovecot via, say, telnet next morning, is the delay the same? http://wiki.dovecot.org/Debugging/Rawlog http://wiki.dovecot.org/TestInstallation?highlight=%28telnet%29 a) and b) should avoid rereading of the mailbox. c) is to ensure that Roundcube does not add the delay itself. the extra 1000 emails were to be added to the cache .. does dovecot go thru all the 17 emails again to form the meta data ? or is there some cache expiry period ? Usually no. Regards, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBTCG2V7+Vh58GPL/cAQJyHAf+MayOiuayYay2mCM5i0DJH296pbuDMAXc kbXKMRXFOP/KGeAYmHirL/9/63h0IjeRtSuBe0HJxmbg87aZVWOejVaXSPOPTAUR n8d++siuBnQ8xiDFw0LKgep2HetYFdxepDRjdPTOM/Drpv+THFRGOVnem6JYb7yw 0XqkQG7M4pKBaxZA+0HFH4c4ADz3XykkOluTHhmU0zDXcdkK0TybfEiwGRY3R7i8 IozQ62pKv/v2t2z2clYOXsqLe6pM14B8/2SXUX4Anrkz8AoKyocT1Ks9hOCkizEn bG99r7U+WObrMmbTpy6jsq//amh496ot37S4DO7DBKa3cKYqOJwwXg== =gqnn -END PGP SIGNATURE-
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Wed, 23 Jun 2010, Rajesh M wrote:
Roundcube 0.3.1 seems to have a hard coded limit of 200 mails per page.
it again took around 4-5 mins
I do not know nothing about Roundcube, therefore I wonder where the 5min
delay come from:
a) Do you keep index files on Dovecot side?
b) Do you deliver / manipulate the mailbox via Dovecot only? E.g. do you
use Dovecot deliver?
c) If you rawlog or sniff the connection and issue the same commands as
roundcube to Dovecot via, say, telnet next morning, is the delay the same?
http://wiki.dovecot.org/Debugging/Rawlog
http://wiki.dovecot.org/TestInstallation?highlight=%28telnet%29
a) and b) should avoid rereading of the mailbox.
c) is to ensure that Roundcube does not add the delay itself.
the extra 1000 emails were to be added to the cache .. does dovecot go
thru all the 17 emails again to form the meta data ? or is there
some
cache expiry period ?
Usually no.
Regards,
- --
Steffen Kaiser
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
iQEVAwUBTCG2V7+Vh58GPL/cAQJyHAf+MayOiuayYay2mCM5i0DJH296pbuDMAXc
kbXKMRXFOP/KGeAYmHirL/9/63h0IjeRtSuBe0HJxmbg87aZVWOejVaXSPOPTAUR
n8d++siuBnQ8xiDFw0LKgep2HetYFdxepDRjdPTOM/Drpv+THFRGOVnem6JYb7yw
0XqkQG7M4pKBaxZA+0HFH4c4ADz3XykkOluTHhmU0zDXcdkK0TybfEiwGRY3R7i8
IozQ62pKv/v2t2z2clYOXsqLe6pM14B8/2SXUX4Anrkz8AoKyocT1Ks9hOCkizEn
bG99r7U+WObrMmbTpy6jsq//amh496ot37S4DO7DBKa3cKYqOJwwXg==
=gqnn
-END PGP SIGNATURE-
hi
did the tests once again on this large email box
i set
maildir_very_dirty_syncs = yes
this was done in the morning and dovecot was restarted
i logged after about 4 hours after the previous login
again it took around 5 mins to login
i was monitoring my server load which around 1.5 - 2 on my dual core dual
xeon machine
this increased to around 8-9 during the login process
the no of emails in the box had increase by around 7500 emails
setting
maildir_very_dirty_syncs = yes
does not seem to help
i have pasted below my dovecot.conf file
base_dir = /var/run/dovecot/
protocols = imap imaps
log_path = /backup1/qmaillog/dovecot.log
#ssl_disable = no
ssl_cert_file = /var/qmail/control/servercert.pem
ssl_key_file = /var/qmail/control/servercert.pem
ssl_cipher_list = djdjjd
verbose_ssl = yes
protocol imap {
listen = *:143
ssl_listen = *:993
}
## Login processes
#login_dir = /usr/local/var/run/dovecot/login
login_user = dovecot
login_process_per_connection = no
login_processes_count = 3
login_process_size = 128
login_max_processes_count = 512
login_greeting = Ready
#login_log_format_elements = user=%u method=%m rip=%r lip=%l %c
## Mailbox locations and namespaces
mail_location = maildir:~/Maildir
namespace private {
separator = .
prefix = INBOX.
inbox = yes
}
# Mail processes
verbose_proctitle = yes
first_valid_uid = 89
last_valid_uid = 89
# Maximum number of running mail processes. When this limit is reached,
max_mail_processes = 200
# Set max. process size in megabytes. Most of the memory goes to mmap()ing
# files, so it shouldn't harm much even if this limit is set pretty high.
mail_process_size = 256
## Maildir-specific settings
maildir_very_dirty_syncs = yes
## Authentication processes
disable_plaintext_auth = yes
auth default {
mechanisms = plain login digest-md5 cram-md5
passdb vpopmail {
args = webmail=127.0.0.1
}
userdb vpopmail {
}
user = vpopmail
count = 1
ssl_require_client_cert = no
}
thanks
rajesh
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 23.06.2010 13:22, Rajesh M wrote: did the tests once again on this large email box First of all, are you using one client app or many? Make sure it's not the client issue. You should also enable some debug to see what commands are sent to IMAP server and what is the response and what is response time. For example, there's imap_debug option in Roundcube. -- Aleksander 'A.L.E.C' Machniak http://alec.pl gg:2275252 LAN Management System Developer http://lms.org.pl Roundcube Webmail Developer http://roundcube.net
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 23 Jun 2010, Rajesh M wrote: a) Do you keep index files on Dovecot side? b) Do you deliver / manipulate the mailbox via Dovecot only? E.g. do you use Dovecot deliver? c) If you rawlog or sniff the connection and issue the same commands as roundcube to Dovecot via, say, telnet next morning, is the delay the same? http://wiki.dovecot.org/Debugging/Rawlog http://wiki.dovecot.org/TestInstallation?highlight=%28telnet%29 a) and b) should avoid rereading of the mailbox. c) is to ensure that Roundcube does not add the delay itself. did the tests once again on this large email box Your reply seems to answer a) as your conf does not contain INDEX=MEMORY, so I guess there are current index files. Can you verify there are? What about b) and c)? Do you have log entries like UID inserted in the middle, Invalid cache or something? Is the mail storage located locally or remote / NFS / ... ? Regards, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBTCH297+Vh58GPL/cAQIFdQf/fsARIYwP4zSAwLliGmklLU2D0QqRpH+W beiWgPcxC6Bp3xxkMOKUvtD6fzK6mSEFDZ3q0tlw4edhV2fYKCA2CvhgujTPKGWD 9muH2FGD8WJICqrFbqXCRkiuj7HKWFu7wuaEX7Wxz+YYZLiyELecAHVtOQ4Hx65T MqyRhYhE7J4Ec9LFwF+H2uwmMIL8aw/NpcEEJ8EQ9Iria0NfINZV/bshXAa11GHi A8NMLIrXi5kW2kKoKzbBJKTsXDuIqeIZFQZcVeGg0Y2Jhs5fYnv944MozpE2VikQ E3TsQEGtaIOCKznFurZCoCEMY3XMh/fkUv5b6Brdni5HOlRbAcirSw== =r/IY -END PGP SIGNATURE-
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 06/23/2010 01:22 PM Rajesh M wrote:
i set
maildir_very_dirty_syncs = yes
this was done in the morning and dovecot was restarted
i logged after about 4 hours after the previous login
again it took around 5 mins to login
i was monitoring my server load which around 1.5 - 2 on my dual core dual
xeon machine
this increased to around 8-9 during the login process
the no of emails in the box had increase by around 7500 emails
setting
maildir_very_dirty_syncs = yes
does not seem to help
i have pasted below my dovecot.conf file
base_dir = /var/run/dovecot/
protocols = imap imaps
log_path = /backup1/qmaillog/dovecot.log
#ssl_disable = no
ssl_cert_file = /var/qmail/control/servercert.pem
ssl_key_file = /var/qmail/control/servercert.pem
ssl_cipher_list = djdjjd
verbose_ssl = yes
protocol imap {
listen = *:143
ssl_listen = *:993
}
## Login processes
#login_dir = /usr/local/var/run/dovecot/login
login_user = dovecot
login_process_per_connection = no
login_processes_count = 3
login_process_size = 128
login_max_processes_count = 512
login_greeting = Ready
#login_log_format_elements = user=%u method=%m rip=%r lip=%l %c
## Mailbox locations and namespaces
mail_location = maildir:~/Maildir
namespace private {
separator = .
prefix = INBOX.
inbox = yes
}
# Mail processes
verbose_proctitle = yes
first_valid_uid = 89
last_valid_uid = 89
# Maximum number of running mail processes. When this limit is reached,
max_mail_processes = 200
# Set max. process size in megabytes. Most of the memory goes to mmap()ing
# files, so it shouldn't harm much even if this limit is set pretty high.
mail_process_size = 256
## Maildir-specific settings
maildir_very_dirty_syncs = yes
## Authentication processes
disable_plaintext_auth = yes
auth default {
mechanisms = plain login digest-md5 cram-md5
passdb vpopmail {
args = webmail=127.0.0.1
}
userdb vpopmail {
}
user = vpopmail
count = 1
ssl_require_client_cert = no
}
thanks
rajesh
Hm, I can't see a auth master socket in your pasted configuration. (BTW:
send only `dovecot -n` output.) So I guess, you let Qmail store the
messages into the Maildir.
If you would use deliver, Dovecot's LDA http://wiki.dovecot.org/LDA,
deliver would keep your index files up to date. This should reduce the
delay at login time.
Regards,
Pascal
--
The trapper recommends today: deadbeef.1017...@localdomain.org
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Pascal Volk wrote:
On 06/23/2010 01:22 PM Rajesh M wrote:
i set
maildir_very_dirty_syncs = yes
this was done in the morning and dovecot was restarted
i logged after about 4 hours after the previous login
again it took around 5 mins to login
i was monitoring my server load which around 1.5 - 2 on my dual core dual
xeon machine
this increased to around 8-9 during the login process
the no of emails in the box had increase by around 7500 emails
setting
maildir_very_dirty_syncs = yes
does not seem to help
i have pasted below my dovecot.conf file
base_dir = /var/run/dovecot/
protocols = imap imaps
log_path = /backup1/qmaillog/dovecot.log
#ssl_disable = no
ssl_cert_file = /var/qmail/control/servercert.pem
ssl_key_file = /var/qmail/control/servercert.pem
ssl_cipher_list = djdjjd
verbose_ssl = yes
protocol imap {
listen = *:143
ssl_listen = *:993
}
## Login processes
#login_dir = /usr/local/var/run/dovecot/login
login_user = dovecot
login_process_per_connection = no
login_processes_count = 3
login_process_size = 128
login_max_processes_count = 512
login_greeting = Ready
#login_log_format_elements = user=%u method=%m rip=%r lip=%l %c
## Mailbox locations and namespaces
mail_location = maildir:~/Maildir
namespace private {
separator = .
prefix = INBOX.
inbox = yes
}
# Mail processes
verbose_proctitle = yes
first_valid_uid = 89
last_valid_uid = 89
# Maximum number of running mail processes. When this limit is reached,
max_mail_processes = 200
# Set max. process size in megabytes. Most of the memory goes to mmap()ing
# files, so it shouldn't harm much even if this limit is set pretty high.
mail_process_size = 256
## Maildir-specific settings
maildir_very_dirty_syncs = yes
## Authentication processes
disable_plaintext_auth = yes
auth default {
mechanisms = plain login digest-md5 cram-md5
passdb vpopmail {
args = webmail=127.0.0.1
}
userdb vpopmail {
}
user = vpopmail
count = 1
ssl_require_client_cert = no
}
thanks
rajesh
Hm, I can't see a auth master socket in your pasted configuration. (BTW:
send only `dovecot -n` output.) So I guess, you let Qmail store the
messages into the Maildir.
If you would use deliver, Dovecot's LDA http://wiki.dovecot.org/LDA,
deliver would keep your index files up to date. This should reduce the
delay at login time.
Regards,
Pascal
Not that it matters, but I think Pascal's correct about this.
Rajesh, can you give dovecot's 'deliver' a go? Let us know if you need a
hand with that. Oh, and try to take good notes. ;)
--
-Eric 'shubes'
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
A.L.E.C put forth on 6/23/2010 6:51 AM: On 23.06.2010 13:22, Rajesh M wrote: did the tests once again on this large email box First of all, are you using one client app or many? Make sure it's not the client issue. You should also enable some debug to see what commands are sent to IMAP server and what is the response and what is response time. For example, there's imap_debug option in Roundcube. I think he's actually using Squirrelmail guys. I previously mentioned performance I get with Roundcube+Dovecot+mbox strictly as a comparison. That's how RC sneaked into the thread. -- Stan
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 23/06/2010 12:22, Rajesh M wrote: did the tests once again on this large email box i set maildir_very_dirty_syncs = yes this was done in the morning and dovecot was restarted i logged after about 4 hours after the previous login again it took around 5 mins to login i was monitoring my server load which around 1.5 - 2 on my dual core dual xeon machine this increased to around 8-9 during the login process the no of emails in the box had increase by around 7500 emails When you ask Dovecot to open a mailbox I believe it builds an index of the key fields in all messages in the inbox. This means it needs to scan the list of all files in the mailbox to see if any new files are in the directory (so login to the server and time time ls -al just before using dovecot - this is a measure of the time the OS needs just to list all the files in the mailbox (I presume maildir, not mbox?) Secondly 7,500 mails over 5 mins means an indexing rate of 25 mails/sec. This would not be out of order for a heavily fragmented drive which is IO bound? Each file needs to be opened to scan the headers so likely you need one disk seek and I guess it's easy to be IO bound? What does iotop show you during dovecot's thrashing? Dovecot2 has an mdbox option which sounds like it could be beneficial for your performance requirements (but it's not stable yet) Otherwise I guess you need to investigate dovecot's delivery agent which does incremental index updates at deliver time (pay the cost one email at a time rather than every 7,500 emails). Or consider alternative filesystems which are more performant for this requirement? Good luck! Ed W
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 6/23/2010 12:27, Stan Hoeppner wrote: A.L.E.C put forth on 6/23/2010 6:51 AM: On 23.06.2010 13:22, Rajesh M wrote: did the tests once again on this large email box First of all, are you using one client app or many? Make sure it's not the client issue. You should also enable some debug to see what commands are sent to IMAP server and what is the response and what is response time. For example, there's imap_debug option in Roundcube. I think he's actually using Squirrelmail guys. I previously mentioned performance I get with Roundcube+Dovecot+mbox strictly as a comparison. That's how RC sneaked into the thread. When using SquirrelMail, make sure these settings are 'true': Allow server thread sort : true Allow server-side sorting: true Otherwise performance will suffer because it will do these things itself rather than letting the server do them. ~Seth
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Ed W put forth on 6/23/2010 4:18 PM: Secondly 7,500 mails over 5 mins means an indexing rate of 25 mails/sec. This would not be out of order for a heavily fragmented drive which is IO bound? Each file needs to be opened to scan the headers so likely you need one disk seek and I guess it's easy to be IO bound? What does iotop show you during dovecot's thrashing? It seems he is I/O bound a degree. If I read his answer to my disk subsystem question correctly he's storing user maildirs on a single local 1TB SATA drive. However, given that his 2nd successive login is 4-5 seconds instead of 5 minutes, it would seem index and cache being current are the problem, not I/O saturation. Faster disk would always help, but it's not close to a total solution to his problem. Putting his maildir on a 16 disk RAID 0 stripe of the same model 1TB disk he already has would yield a 16x improvement in seek throughput, cutting his 'stale' login time to ~20 seconds, if my math is correct. 20 seconds is still unacceptably high IMHO, though it's much better than 300 seconds. Ok so lets assume the filesystem underlying his maildir is heavily fragmented and that defragging it would yield a 100% improvement for argument sake (50% improvement is almost unheard of, normal is about 20%). He'd still be looking at a 10 second login time after spending anywhere from $3k to $8k USD on a 16 disk array depending on what vendor he chooses. Throwing money and hardware at this problem isn't the proper or optimal solution. Dovecot2 has an mdbox option which sounds like it could be beneficial for your performance requirements (but it's not stable yet) I don't think migrating to a new mailbox format is what he really needs, or wants. IIRC he's been using qmail for years, which means he's been using maildir for years. He's likely very comfortable with maildir and probably wants to stick with it. Otherwise I guess you need to investigate dovecot's delivery agent which does incremental index updates at deliver time (pay the cost one email at a time rather than every 7,500 emails). Or consider alternative filesystems which are more performant for this requirement? On this I completely agree with you, and I suggested it previously in the same thread I suggested dirty_syncs. This should be his next step. If LDA alone doesn't fix the problem, then he should try dirty_syncs again in conjunction with LDA. -- Stan
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Seth Mattinen put forth on 6/23/2010 4:30 PM: When using SquirrelMail, make sure these settings are 'true': Allow server thread sort : true Allow server-side sorting: true Otherwise performance will suffer because it will do these things itself rather than letting the server do them. Is this still true if both SM and Dovecot are on the same box? I.e. is this a CPU bandwidth issue or a network bandwidth issue? If these settings solve a network b/w issue then they're irrelevant when both SM and Dovecot are on the same host. -- Stan
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 6/23/2010 15:41, Stan Hoeppner wrote: Seth Mattinen put forth on 6/23/2010 4:30 PM: When using SquirrelMail, make sure these settings are 'true': Allow server thread sort : true Allow server-side sorting: true Otherwise performance will suffer because it will do these things itself rather than letting the server do them. Is this still true if both SM and Dovecot are on the same box? I.e. is this a CPU bandwidth issue or a network bandwidth issue? If these settings solve a network b/w issue then they're irrelevant when both SM and Dovecot are on the same host. Found it in the docs: http://squirrelmail.org/docs/admin/admin-6.html#ss6.3 I would very strongly suggest that IMAP SORT be enabled if the server supports it, which of course Dovecot does. IMAP THREAD is the other one. ~Seth
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 6/23/2010 15:41, Stan Hoeppner wrote: Seth Mattinen put forth on 6/23/2010 4:30 PM: When using SquirrelMail, make sure these settings are 'true': Allow server thread sort : true Allow server-side sorting: true Otherwise performance will suffer because it will do these things itself rather than letting the server do them. Is this still true if both SM and Dovecot are on the same box? I.e. is this a CPU bandwidth issue or a network bandwidth issue? If these settings solve a network b/w issue then they're irrelevant when both SM and Dovecot are on the same host. It's more along the lines of SM will not use commands like SORT and attempt to do things itself because your server does not implement it. For example, I believe SurgeMail does not implement IMAP SORT so you'd have to set it to 'false'. Dovecot does, and is much better at executing SORT than having squirrel simulate it. ~Seth
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 6/19/2010 7:33 PM, Rajesh M wrote: setup dovecot with squirrelmail and logged in I'd recommend also installing and configuring imapproxy - it can be beneficial with squirrelmail. -- Daniel
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Timo Sirainen put forth on 6/19/2010 9:43 PM: On 20.6.2010, at 3.33, Rajesh M wrote: 3) next i changed squirrelmail display preference to list 500 mails per page and then clicked on inbox ... expecting dovecot to take atleast around 1-2 mins to show the mails ... the mails were displayed in just 8 seconds .. awesome performance Fetching metadata for 500 messages took 8 seconds? Still sounds slow. Although maybe it's because of maildir. I'm anyway not happy until Dovecot can open a mailbox with millions of messages and access them in nearly constant time (few milliseconds, with SSD). Roundcube 0.3.1 seems to have a hard coded limit of 200 mails per page. On a dual CPU 500 MHz Celeron with 384MB and a single 500GB SATA drive, running both RC and Dovecot 1.2.11 on the box, it takes about 5 seconds to pull metadata for each set of 200 messages on a 13k+ message mailbox. This is using mbox. I agree with Timo, 8 seconds for 500 messages on that hardware seems a bit slow given what my lowly setup can do. Was the box under load when you tested or idle? What disk subsystem is the maildir stored on? Single disk or high performance RAID? Was the disk subsystem under load or idle when you tested? -- Stan stan my machine is dual core dual processor xeon 1.6 gig proc 4 gb ram 1 tb sata for centos linux 1 tb sata for vpopmail domains data, mysql db and log data my hdd usage is at 60 percent now. i accessed the large email box (this is qmail tap account in which emails are continuously added) yesterday 10 pm at night login took around 5 mins Around 169000 emails in the box I logged out and logged in again after around 5 mins -- access took just around 4-5 seconds. Around 10 emails had been added. at 10 am in the morning today i logged in again ie around 12 hours after above login. it again took around 4-5 mins i have currently over 17 emails in the mailbox ie overnight around 1000 emails were added i logged off and access again -- the mailbox opened in around 5 seconds so what i can conclude is that if relogin after a few minutes then the access is very fast whereas if i do the same after a few hours then it takes time. i am new to dovecot and a few questions 1) is it ok that dovecot takes 5 mins every time i login if there is significant time difference between the two logins 2) as per what timo said the metadata is cached so the data pertaining to the extra 1000 emails were to be added to the cache .. does dovecot go thru all the 17 emails again to form the meta data ? or is there some cache expiry period ? thanks rajesh
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Rajesh M put forth on 6/23/2010 12:12 AM: 1) is it ok that dovecot takes 5 mins every time i login if there is significant time difference between the two logins No, 5 minutes is unacceptable to pretty much anyone. 2) as per what timo said the metadata is cached so the data pertaining to the extra 1000 emails were to be added to the cache .. does dovecot go thru all the 17 emails again to form the meta data ? or is there some cache expiry period ? It sounds like the entire mailbox is being read, and that would explain the 5 minute issue. I had a very similar problem not long ago with large mbox files which was solved with the mbox version of the parameter below. Try setting: maildir_very_dirty_syncs = yes in dovecot.conf and see if that helps. Please read the documentation before doing so. If you do this it would probably be a really good idea to use Dovecot LDA for the mailbox delivery instead of having qmail write the mail. Having both qmail and dovecot writing and reading the same maildir could be problematic. http://wiki.dovecot.org/LDA/Qmail -- Stan
[Dovecot] dovecot evaluation on a 30 gb mailbox
hi all i use qmail toaster. i recently changed over to dovecot. posting below my evaluation on the same my machine dual core dual xeon 5140 processor, with 4 gb ram, centos linux with qmailtoaster i had setup a qmail tap account which backups up data for a client of mine mailbox size over 30 gb setup dovecot with squirrelmail and logged in it had over one hundred and thirty nine thousand emails 139019 emails in the 30 gb mailbox Reports 1) first login : took 7 minutes to parse 139019 mails but never timed out. this was simply impossible with courier 2) the next login it took 5 seconds by which time the mail numbers increased to 139045 3) next i changed squirrelmail display preference to list 500 mails per page and then clicked on inbox ... expecting dovecot to take atleast around 1-2 mins to show the mails ... the mails were displayed in just 8 seconds .. awesome performance qmailtoaster is simply waiting for a dovecot integration i am evaluating the stability of dovecot now in the next 2-3 weeks i post stability report rajesh
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
On 20.6.2010, at 3.33, Rajesh M wrote: 3) next i changed squirrelmail display preference to list 500 mails per page and then clicked on inbox ... expecting dovecot to take atleast around 1-2 mins to show the mails ... the mails were displayed in just 8 seconds .. awesome performance Fetching metadata for 500 messages took 8 seconds? Still sounds slow. Although maybe it's because of maildir. I'm anyway not happy until Dovecot can open a mailbox with millions of messages and access them in nearly constant time (few milliseconds, with SSD).
Re: [Dovecot] dovecot evaluation on a 30 gb mailbox
Timo Sirainen put forth on 6/19/2010 9:43 PM: On 20.6.2010, at 3.33, Rajesh M wrote: 3) next i changed squirrelmail display preference to list 500 mails per page and then clicked on inbox ... expecting dovecot to take atleast around 1-2 mins to show the mails ... the mails were displayed in just 8 seconds .. awesome performance Fetching metadata for 500 messages took 8 seconds? Still sounds slow. Although maybe it's because of maildir. I'm anyway not happy until Dovecot can open a mailbox with millions of messages and access them in nearly constant time (few milliseconds, with SSD). Roundcube 0.3.1 seems to have a hard coded limit of 200 mails per page. On a dual CPU 500 MHz Celeron with 384MB and a single 500GB SATA drive, running both RC and Dovecot 1.2.11 on the box, it takes about 5 seconds to pull metadata for each set of 200 messages on a 13k+ message mailbox. This is using mbox. I agree with Timo, 8 seconds for 500 messages on that hardware seems a bit slow given what my lowly setup can do. Was the box under load when you tested or idle? What disk subsystem is the maildir stored on? Single disk or high performance RAID? Was the disk subsystem under load or idle when you tested? -- Stan
