Re: [Dovecot] Migration questions...
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 13 May 2009, Richard Hobbs wrote: difference for you. A single server with recent hardware should be able to easily handle hundreds of simultaneous users. That's also good to know... i like to do a job right instead of relying on faster hardware, as i'm sure you all do too, but it's good to know that if i make one or two non-optimal choices along the way, it'll probably be lightning fast anyway! Well, IMHO there is some advantage to split index data and mail data on separate disks, if they are located at different channels, so the load per channel drops. But this is probably not so significant, if your OS uses one CPU for all I/O anyway The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: a) hits a 2GB limit on a mailbox in the mail client, b) try to move all their 20'000+ messages at once to an archive folder once a year, when their quota limit is exceeded. This can make the whole server irresponsible slow, because I have the mail_log plugin running as well. With maildir, and especially dovecot, this problem effectively disappears! That's the theory anyway, right? Bye, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBSgvEHnWSIuGy1ktrAQLsswgAvW1yUeDUVCvivf/67C47aJF4Vahuovzu kMBcUlrB9K2kxxz4yP2F+UVSnsQAVyo5I7XztFQ81mCZdNKStFO8k4booI75zpW/ Qwm6d9jezReMMzL/D73NKNwi1hQwK+CDsSIuuIFRfFuHkH7q02BAT2SF66dTeqM3 e08g66b0Ki2qlSWfE4smAjZFrVZCPgDlYgSRYrUK0VixV6ljyZLU+KFrl97RkHOY qMvxM6WfQ7TsejcU5UrFGL/J0Wyh5TOzO6iYSAXhsWM6AEthfDvz1C5rpcBmk1V4 BYQYe4cnG7b7slb0SsMed6SVuyzVfKnoZo+Wm5vEy27DGy8JfVljuA== =wzNa -END PGP SIGNATURE-
Re: [Dovecot] id:N13388 Disable storage of messages on server when read (pop3)
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 13 May 2009, Timo Sirainen wrote: On Tue, 2009-05-12 at 14:13 +0300, Dimitrios Karapiperis wrote: Is there any way in dovecot to disable the storage of messages on server when they are read/downloaded by the users (POP3). No. I remember some discussion some time ago about to or not to update IMAP flags when reading via POP. If your Dovecot POP server updates the READ flag, you could delete those READ messages via cron job. Bye, - -- Steffen Kaiser -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) iQEVAwUBSgvFbHWSIuGy1ktrAQKMRwf8CktRoO74kQHMIBCjElF9UL2OjWMr5qVS GPADm3wZ/mdHhHZLf0pazu/82BAlVNkah/k6BqKM29uOZhpoOoLu1KsUJtKIiEjH h7+bVfyN3/Avm6ViInYE/RTm+AxDnDcdEURDnFH7U4H9eIVp4rCKoMQv52qBCxEz A2EDHlsGtyqVWIV67XXZX0YLQzmKXcy72L/TTo6evlLZQR3kh7ugu7q6maAI7CaV tGBaJmiHfY/EmsI6mM1hQgCxFq0MtpkUighxNginsojcDj5mJSh3efxH1yc53GKi VUV8lGP/4IVefO+fx7KgSRPawJPs+4kXz8dn0WwZCCKWAzbv2xnXNQ== =kpk5 -END PGP SIGNATURE-
Re: [Dovecot] Migration questions...
Richard Hobbs wrote: That's also good to know... i like to do a job right instead of relying on faster hardware, as i'm sure you all do too, but it's good to know that if i make one or two non-optimal choices along the way, it'll probably be lightning fast anyway! The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Most of my mailing list folders that I work with on a daily basis like NANOG (over 20,000) and cisco-nsp (over 35,000) are no slower than folders with a handful of messages in them. It's not a private server either, it's the same one my customers use. If you're seeing a slowdown at 5k messages, either your server is woefully underpowered or something isn't quite right. ~Seth
Re: [Dovecot] id:N13388 Disable storage of messages on server when read (pop3)
On Tue, 2009-05-12 at 14:13 +0300, Dimitrios Karapiperis wrote: Is there any way in dovecot to disable the storage of messages on server when they are read/downloaded by the users (POP3). No. I remember some discussion some time ago about to or not to update IMAP flags when reading via POP. If your Dovecot POP server updates the READ flag, you could delete those READ messages via cron job. If I just clear the cur directory (maildir) via cron , is it ok?
Re: [Dovecot] Migration questions...
On 5/14/2009, Steffen Kaiser (skdove...@smail.inf.fh-brs.de) wrote: b) try to move all their 20'000+ messages at once to an archive folder once a year, when their quota limit is exceeded. This can make the whole server irresponsible slow, because I have the mail_log plugin running as well. Hmmm... it would be nice if there was some way to check for mass moves like that and just log one line like: imap(user): copy - Archive: uid=908, IDIOT_USER moved 20K+ messages at once - FLOG HIM!! Don't know if its possible to detect a mass move though... -- Best regards, Charles
[Dovecot] IMAP subfolders and MBOX format
Hello folks I have a problem related to the mbox format I think First we use MBOX format with Dovecot 1.14 A user's here need to create some subfolders in his Imap space. Due to the use of MBOX format this is impossible from thunderbird as you cannot creat a directory in a file :-) So to help him I created manually on the mail server some directories into his imap space, Doing so we are able to see the folder and subfolders but it is not possible to move some other folders into the subfolder via the thunderdird client. Anyone to help me a bit ? Note : on another machine which use the Maildir format I can do whatever I want without problem. Thanks a lot.
[Dovecot] Unable to subscribe to newly created subfolders under public mailboxes with acl plugin enabled
I've group namespace and b2b mailbox under it. I'm unable to subscribe to it and deliver emails there. Here is my problem: if I create subfolder (lets name it subf) under b2b mailbox I'm unable to subscribe to it until acl plugin is enabled. Even if I put dovecot-acl file in .b2b.subf folder with anyone full rights dovecot-acl-list doesn't get populated with new subfoler. If I put it there manually I'm able to see subf folder in subscribe dialog in TB, but dovecot-acl-list is regulary updated, isn't it? Updated file is missing subf entry again. If I disable acl plugin everything works fine, but I'd like to keep it enabled. Is it a bug or misconfiguration? My dovecot version is 1.2 rc3 here is dovecot -n authput protocols: imap managesieve ssl: yes ssl_ca_file: /usr/share/ca-certificates/floristCA-cacert.pem ssl_cert_file: /var/lib/ssl/certs/imap.florist.my-cert.pem ssl_key_file: /var/lib/ssl/private/imap.florist.my-key.pem ssl_cipher_list: ALL:!LOW:!SSLv2 disable_plaintext_auth: yes verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable(default): /usr/libexec/dovecot/imap-login login_executable(imap): /usr/libexec/dovecot/imap-login login_executable(managesieve): /usr/libexec/dovecot/managesieve-login login_processes_count: 5 verbose_proctitle: yes first_valid_uid: 1000 mail_privileged_group: mail mail_location: maildir:/var/spool/mail/dovecot/%n:INBOX=/var/spool/mail/dovecot/%n/INBOX mail_debug: yes mail_executable(default): /usr/libexec/dovecot/imap mail_executable(imap): /usr/libexec/dovecot/imap mail_executable(managesieve): /usr/libexec/dovecot/managesieve mail_plugins(default): acl fts fts_squat mail_plugins(imap): acl fts fts_squat mail_plugins(managesieve): mail_plugin_dir(default): /usr/lib/dovecot/modules/imap mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap mail_plugin_dir(managesieve): /usr/lib/dovecot/modules/managesieve namespace: type: private separator: / inbox: yes list: yes subscriptions: yes namespace: type: public separator: / prefix: group/ location: maildir:/var/spool/mail/groupmail list: yes subscriptions: yes auth default: verbose: yes passdb: driver: pam args: session=yes userdb: driver: passwd args: blocking=yes socket: type: listen client: path: /var/run/dovecot/auth-client mode: 432 master: path: /var/run/dovecot/auth-master mode: 438 plugin: acl: vfile lazy_expunge: .EXPUNGED/ .DELETED/ .DELETED/.EXPUNGED/ sieve: ~/.dovecot.sieve sieve_storage: ~/sieve fts: squat fts_squat: partial=4 full=4
Re: [Dovecot] Migration questions...
I'd point out that the big *practical* issue with mbox is the reality of big inboxes. While you can restrict the hoi polloi to something limited like a quota of under 60MB (and remember that inbox is one big honking file), the powers that be will not allow themselves to be so limited...nor will they be particularly good about cleaning up. I dunno how it is with you and your hardware/OS implementation, but there is a serious CPU hit when somebody with a 1GB inbox (one big file, remember) deletes a message...or gets new mail...or searches their inbox (I call this the python swallowing the pig). The first two will be trivial when we switch to maildir. OTOH, boy is it quick to do a backup with mbox. I dread that part of our move from mbox to maildir format. We will probably go from 2 hours to a day in the switch from 3000 inboxes of one file each (mbox) to 3000 directories with hundreds or thousands of files in each (maildir). Pick your poison Timo Sirainen wrote: On May 13, 2009, at 9:57 AM, Richard Hobbs wrote: OK... so Dovecot is certainly significantly faster that uw-imapd in both cases, but is dovecot fastest with mbox or maildir? I would assume maildir, but you never know... It's not that simple to answer. With mbox it's probably faster to read through all mails, because they're in a single file. With Maildir it's faster to delete mails, because it only needs to delete a single file, instead of moving data around in the mbox file. But Maildir has less problems and it's much less likely to get corrupted, so even if mbox performance would be better in some cases I'd recommend Maildir. -- One must think like a hero to behave like a merely decent human being. - May Sarton Stewart Dean, Unix System Admin, Bard College, New York 12504 sd...@bard.edu voice: 845-758-7475, fax: 845-758-7035
Re: [Dovecot] Unable to subscribe to newly created subfolders under public mailboxes with acl plugin enabled
On 5/14/2009, Max Ivanov (ivanov.ma...@gmail.com) wrote: here is dovecot -n authput Please don't edit this output... the first two lines are dovecot version and info on system... -- Best regards, Charles
Re: [Dovecot] IMAP subfolders and MBOX format
on 5-14-2009 5:11 AM Frank Bonnet spake the following: Hello folks I have a problem related to the mbox format I think First we use MBOX format with Dovecot 1.14 A user's here need to create some subfolders in his Imap space. Due to the use of MBOX format this is impossible from thunderbird as you cannot creat a directory in a file :-) So to help him I created manually on the mail server some directories into his imap space, Doing so we are able to see the folder and subfolders but it is not possible to move some other folders into the subfolder via the thunderdird client. Anyone to help me a bit ? Note : on another machine which use the Maildir format I can do whatever I want without problem. Thanks a lot. You can create subfolders on a mbox system, but you can't have folders that contain messages and folders. When you create folders you end them with a / like new folder/. signature.asc Description: OpenPGP digital signature
[Dovecot] Possibly dumb questions about DC and user/system limits
Because of user access growth, the number of processes associated with IMAP has increased and I thought to change some of the configured parms. dovecot -n yields: # 1.1.14: /usr/local/etc/dovecot.conf # OS: AIX 1 005A928C4C00 listen: *:143 ssl_listen: *:993 disable_plaintext_auth: no verbose_ssl: yes login_dir: /var/run/dovecot/login login_executable: /usr/local/libexec/dovecot/imap-login login_processes_count: 12 login_max_processes_count: 774 max_mail_processes: 1280 mail_max_userip_connections: 12 verbose_proctitle: yes first_valid_uid: 200 mail_location: mbox:~/mail:INBOX=/var/spool/mail/%u:INDEX=/var/dcindx/%u mbox_write_locks: fcntl mbox_dirty_syncs: no auth default: passdb: driver: pam userdb: driver: passwd The only changes was that max_mail_processes went from 1024 to 1280. Now I get a error message when I start DC: Warning: fd limit 2000 is lower than what Dovecot can use under full load (more than 2054). Either grow the limit or change login_max_processes_count and max_mail_processes settings So I changed the no_size and no_size_hard to 3500 and 4000 respectively in both dovecot and root AIX defines nosize:* *Sets the soft limit for the number of file descriptors a user process may have open at one time. an lsuser dovecot returns: dovecot id=417 pgrp=dovecot groups=dovecot shell=/bin/false daemon=true admin=false ... fsize=2097151 cpu=-1 data=262144 stack=65536 core=2097151 rss=65536 nofiles=3500 nofiles_hard=4000 I kill dovecot and all children and restart itsame error message What am I missing? Does the machine have to be rebooted for the no_size to be updated? -- Once upon a time, the Internet was a friendly, neighbors-helping-neighbors small town, and no one locked their doors. Now it's like an apartment in Bed-Stuy: you need three heavy duty pick-proof locks, one of those braces that goes from the lock to the floor, and bars on the windows Stewart Dean, Unix System Admin, Bard College, New York 12504 sd...@bard.edu voice: 845-758-7475, fax: 845-758-7035
Re: [Dovecot] Possibly dumb questions about DC and user/system limits
On Thu, 2009-05-14 at 14:17 -0400, Stewart Dean wrote: Warning: fd limit 2000 is lower than what Dovecot can use under full load (more than 2054). Either grow the limit or change login_max_processes_count and max_mail_processes settings .. I kill dovecot and all children and restart itsame error message What am I missing? Does the machine have to be rebooted for the no_size to be updated? I've no idea about AIX, but it sounds similar to how Linux distros set them up. They probably affect only the logging in users' processes and not services that start up. Anyway you probably still have ulimit command in AIX shells too? So before starting up Dovecot, make sure ulimit -n returns enough files. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Migration questions...
Phillip Macey wrote: Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) At first glance this sounds like a large folder is being indexed... are you using Dovecot deliver (which updates indices on deliver)? With maildir, and especially dovecot, this problem effectively disappears! That's the theory anyway, right? Theory is wonderful when it works :-) In theory, Theory and Practice are the same. In practice, this isn't always the case. :) -- Curtis Maloney cmalo...@cardgate.net
Re: [Dovecot] Migration questions...
Phillip Macey wrote: On 14/05/2009 5:11 PM, Steffen Kaiser wrote: On Wed, 13 May 2009, Richard Hobbs wrote: The main complaint we have from users is that their IMAP Inbox, with 5000 emails in it takes ages to appear, and no amount of coaxing will convince them to split their inbox into multiple folders. Oh, we serve Maildir via Dovecot IMAP and 5000 messages per folder are a wimp. Problems start if the user: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Are you using ext3 by chance? Vanilla ext3 without directory indexing (or whatever it's called) *hates* directories with a lot of files - like maildir. Personally, I use XFS, which doesn't suffer from this problem since it uses b-trees instead of a table(!) like ext3 does. ~Seth
Re: [Dovecot] Migration questions...
On Fri, 2009-05-15 at 09:35 +1000, Phillip Macey wrote: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Do you have POP3 users? What clients do your users typically use? BTW. Kind of nasty to hijack a pretty much unrelated thread to your question.. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] Migration questions...
On 15/05/2009 9:49 AM, Timo Sirainen wrote: On Fri, 2009-05-15 at 09:35 +1000, Phillip Macey wrote: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Do you have POP3 users? What clients do your users typically use? 50% pop. 50% imap.. Roughly. BTW. Kind of nasty to hijack a pretty much unrelated thread to your question.. Apologies, I wasn't intending it to be a hijack. Steffen said he had no problems. I was only making trying to point out that maildir can also have its problems as well. I dont know how MBox would handle the same situation because we dont have any MBox storage. I may post another thread if we still have problems after I have finished tidying up some peoples maildirs. -- Thanks, Phill Macey begin:vcard fn:Phillip Macey n:Macey;Phillip org:Canon Information Systems Research Australia (CISRA);IT Services adr:;;3 Thomas Holt Drive;North Ryde;;2113;Australia email;internet:phillip.ma...@cisra.canon.com.au title:Systems Administrator tel;work:(02) 9805 2645 x-mozilla-html:FALSE version:2.1 end:vcard
Re: [Dovecot] Migration questions...
On May 14, 2009, at 8:19 PM, Phillip Macey wrote: On 15/05/2009 9:49 AM, Timo Sirainen wrote: On Fri, 2009-05-15 at 09:35 +1000, Phillip Macey wrote: We are having some performancec issues on our server at the moment - all I can put it down to is the large size of some maildirs. Eg. `ls -ld Maildir/cur` might show a directory 20Mb in size for some of our users (20-30k emails). (Performance issues == everything is running ok then all of a sudden load avg goes through the roof, system cpu time goes crazy. Reading mail grinds to a halt. Then everything recovers just as suddenly and the load avg gradually returns to normal levels) Do you have POP3 users? What clients do your users typically use? 50% pop. 50% imap.. Roughly. With POP3 users those kinds of I/O bursts can happen when a lot of messages don't have their virtual size known. See Maildir performance in http://wiki.dovecot.org/POP3Server
Re: [Dovecot] Migration questions...
On 15/05/2009 10:33 AM, Timo Sirainen wrote: On May 14, 2009, at 8:19 PM, Phillip Macey wrote: On 15/05/2009 9:49 AM, Timo Sirainen wrote: Do you have POP3 users? What clients do your users typically use? 50% pop. 50% imap.. Roughly. With POP3 users those kinds of I/O bursts can happen when a lot of messages don't have their virtual size known. See Maildir performance in http://wiki.dovecot.org/POP3Server Ok. Thanks. I will have a read and keep that in mind ;-)
