Re: [Dovecot] Dovecot crashes totally - SOLVED
On 11/06/2011 07:56 PM, Gordon Grubert wrote: On 11/04/2011 08:43 PM, Timo Sirainen wrote: On Sat, 2011-10-22 at 21:21 +0200, Gordon Grubert wrote: Hello, our dovecot server crashes totally without any really useful log messages. The error log can be found in the attachment. The only way to get dovecot running again is a complete system restart. How often does it break? If really a complete system restart is needed to fix it, it doesn't sound like a Dovecot problem. Check if it's enough to stop dovecot and then make sure there aren't any dovecot processes lying around afterwards. Currently, the problem occurred three times. The last time some days ago. The last crash was in the night and, therefore, we used the chance for a detailed debugging of the system. You could be right, that it's not a dovecot problem. Next to dovecot, we found other processes hanging and could not be killed by kill -9. Additionally, we found a commonness of all of these processes: They hanged while trying to access the mailbox volume. Therefore, we repaired the filesystem. Now, we're watching the system ... Oct 11 09:55:23 mailserver2 dovecot: master: Error: service(imap): Initial status notification not received in 30 seconds, killing the process Oct 11 09:56:23 mailserver2 dovecot: imap-login: Error: master(imap): Auth request timed out (received 0/12 bytes) Kind of looks like auth process is hanging. You could see if stracing it shows anything useful. Also are any errors logged about LDAP? Is LDAP running on the same server? Dovecot authenticates against postfix and postfix has an LDAP connection. The LDAP is running on an external cluster. Here, no errors are reported. We hope, that the filesystem error was the reason for the problem and, that the problem is fixed by repairing it. During the last two month, no error occurred. Therefore, the problem in the filesystem seems to be the reason for the dovecot crash. Thx and best regards, Gordon smime.p7s Description: S/MIME Cryptographic Signature
Re: [Dovecot] imap-login process_limit reached
On Mon, 2012-01-16 at 14:41 -0800, Don Buchholz wrote: I've been having some problems with IMAP user connections to the Dovecot (v2.0.8) server. The following message is being logged. Jan 16 10:51:36 postal dovecot: master: Warning: service(imap-login): process_limit reached, client connections are being dropped Maybe this will help some in future: http://hg.dovecot.org/dovecot-2.1/rev/a4e61c99c7eb The new error message is: service(imap-login): process_limit (100) reached, client connections are being dropped
[Dovecot] Performance of Maildir vs sdbox/mdbox
Hi Guys, I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. This is a brand new build, with customer mailboxes to be migrated in over the course of 3-4 months. Some details on our new environment: * Approximately 1.6M+ mailboxes once all legacy systems are combined * NetApp FAS6280 storage w/ 120TB usable for mail storage, 1TB of FlashCache in each controller * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers * Apache Lucene indexes will be used to accelerate IMAP search for users Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? * What kind of IOPS/User reduction could we expect to see under mdbox? * If someone can give some technical reasoning behind why mdbox does less IOPS than Maildir? I understand some of the reasons for the mdbox IOPS question, but I need some more information so we can discuss internally and make a decision as to whether we're comfortable going with mdbox from day one. We're very familiar with Maidlir, and there's just some uneasiness internally around going to a new mail storage format. Thanks!
[Dovecot] Dovecot Solutions company update
Hi, A small update: My Dovecot support company finally has web pages: http://www.dovecot.fi/ We've also started providing 24/7 support.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
Am 18.01.2012 13:44, schrieb Lee Standen: Hi Guys, I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. This is a brand new build, with customer mailboxes to be migrated in over the course of 3-4 months. Some details on our new environment: * Approximately 1.6M+ mailboxes once all legacy systems are combined * NetApp FAS6280 storage w/ 120TB usable for mail storage, 1TB of FlashCache in each controller * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) nfs may not be optimal clusterfilesystem might better, but this is an heavy seperate discussion * Postfix will feed new email to Dovecot via LMTP perfect * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers * Apache Lucene indexes will be used to accelerate IMAP search for users sounds ok Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak wow thats big Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? * What kind of IOPS/User reduction could we expect to see under mdbox? there should be people on the list , knowing this , by migration done * If someone can give some technical reasoning behind why mdbox does less IOPS than Maildir? as far i remember mdbox takes 8 mails per file ( i am not using it currently, so i didnt investigate it ), better wait for more qualified answer, anyway mdbox seems recommended in your case from our last plans about 25k mailboxes we decide using mdbox, as far i remember I understand some of the reasons for the mdbox IOPS question, but I need some more information so we can discuss internally and make a decision as to whether we're comfortable going with mdbox from day one. We're very familiar with Maidlir, and there's just some uneasiness internally around going to a new mail storage format. Thanks! from my personal knowledge io on storage has most influance of performance, if at last ,all other setup parts are solved optimal wait a little bit , i guess more matching answers will come up after all ,you can hire someone, perhaps Timo, if you stuck in something -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
Spanish edu site here, 80k users, 4,5 TB of email, 6.000 iops (indexes) + 9.000 iops (mdboxes) in working hours here. We evaluated mdbox against Maildir and we found that with these setting dovecot 2 perfoms better than Maildir: mdbox_rotate_interval = 1d mdbox_rotate_size=60m zlib_save_level = 9 # 1..9 zlib_save = gz # or bz2 We detected 40% less iops with this setup *in working hours (more info below)*. Zlib saved some writes (15-30%). With mdbox, deletion of a message is written to indexes (use SSD for this), and a nightly cronjob deletes the real message from the mdbox, this saves us some iops in working hours. Also, backup software is MUCH happier handling hundreds of thousands files (mdbox) versus tens of millions (maildir) Mdbox has also drawbacks: you have to be VERY careful with your indexes, they contain data that can not be rebuilt from mdboxes. The nightly cronjob purging the mdboxes hammers the SAN. Full backup time is reduced, but incremental backup space time increases: if you delete a message, after purging it from the mdbox the mdbox file changes (size and date), so the incremental backup has to copy it again. Regards Javier
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, 2012-01-18 at 20:44 +0800, Lee Standen wrote: I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. Unfortunately there aren't really any. Everyone who seems to switch to sdbox/mdbox usually also change their hardware at the same time, so there aren't really any before/after metrics. I've of course some unrealistic synthetic benchmarks, but I don't think they are very useful. So, I would also be very interested in seeing some before/after graphs of disk IO, CPU and memory usage of Maildir - dbox switch in same hardware. Maildir is anyway definitely worse performance then sdbox or mdbox. mdbox also uses less NFS operations, but I don't know how much faster (if any) it is with Netapps. * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers Can the same mailbox still be accessed via multiple geographic servers? I've had some plans for doing this kind of access/replication using dsync.. * Apache Lucene indexes will be used to accelerate IMAP search for users Dovecot's fts-solr or fts-lucene? Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? If you have webmail type of clients, definitely. For Outlook/Thunderbird you should still see improvement, but not necessarily as much. You didn't mention POP3. That isn't Dovecot's strong point. Its performance should be about the same as Courier-POP3, but could be less than QMail-POP3. Although if many of your POP3 users keep a lot of mails on server it * If someone can give some technical reasoning behind why mdbox does less IOPS than Maildir? Maildir renames files a lot. From new/ - to cur/ and then every time message flag changes. That's why sdbox is faster. Why mdbox should be faster than sdbox is because mdbox puts (or should put) more mail data physically closer in disks to make reading it faster. I understand some of the reasons for the mdbox IOPS question, but I need some more information so we can discuss internally and make a decision as to whether we're comfortable going with mdbox from day one. We're very familiar with Maidlir, and there's just some uneasiness internally around going to a new mail storage format. It's at least safer to first switch to Dovecot+Maildir to make sure that any problems you might find aren't related to the mailbox format..
[Dovecot] shared folder files not displaying in thunderbird
Hello,
I have dovecot installed with the configuration below.
One of the subfolders created (using the email client) under the
'/home/vpopmail/domains/mydomain.com/shared/projects' share no longer
(it used to) displays the files located in it. There are about 150
folders under the '/home/vpopmail/domains/mydomain.com/shared/projects'
share all of which display the files located in them, the one mentioned
used to display the contents but no longer does.
What would be the reason that one folder would no longer display
existing files in the email client (Thunderbird) and the other folders
would? And, how do I fix this?
I've already tried unsubscribing and resubscribing the folder. This did
not work.
Would it now be simply a matter of unsubscribing the folder, deleting
the dovecot files, and resubscribing to the folder?
Eric
# 2.0.11: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.18-238.19.1.el5 i686 CentOS release 5.7 (Final)
auth_cache_size = 32 M
auth_mechanisms = plain login digest-md5 cram-md5
auth_username_format = %Lu
disable_plaintext_auth = no
first_valid_uid = 89
log_path = /var/log/dovecot.log
login_greeting = Dovecot toaster ready.
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
namespace {
location = maildir:/home/vpopmail/domains/mydomain.com/shared/projects
prefix = projects.
separator = .
type = public
}
passdb {
args = cache_key=%u webmail=127.0.0.1
driver = vpopmail
}
plugin/quota = maildir
protocols = imap
ssl_cert = /var/qmail/control/servercert.pem
ssl_key = /var/qmail/control/servercert.pem
userdb {
args = cache_key=%u quota_template=quota_rule=*:backend=%q
driver = vpopmail
}
protocol imap {
imap_client_workarounds = delay-newmail
}
Re: [Dovecot] Dovecot Solutions company update
Am 18.01.2012 13:58, schrieb Timo Sirainen: Hi, A small update: My Dovecot support company finally has web pages: http://www.dovecot.fi/ We've also started providing 24/7 support. Hi Timo, very cool ! -- Best Regards MfG Robert Schetterer Germany/Munich/Bavaria
Re: [Dovecot] Dovecot unable to locate mailbox
On Mon, 2012-01-16 at 14:38 +0200, Jason X, Maney wrote:
Jan 16 14:18:16 myservername dovecot: pop3(userA): Error: user molla:
Initialization failed: mail_location not set and autodetection failed: Mail
storage autodetection failed with home=/home/userA
As it says.
Yet my config also come out strangely as below:
=
root@guyana:~# dovecot -n
# 2.0.13: /etc/dovecot/dovecot.conf
# OS: Linux 3.0.0-12-server x86_64 Ubuntu 11.10
passdb {
driver = pam
}
protocols = imap pop3
ssl_cert = /etc/ssl/certs/dovecot.pem
ssl_key = /etc/ssl/private/dovecot.pem
userdb {
driver = passwd
}
root@guyana:~#
=
There is no mail_location above. This is the configuration Dovecot sees.
My mailbox location setting is as follows:
=
cat conf.d/10-mail.conf |grep mail_location
Look at /etc/dovecot/dovecot.conf file. Do you see !include
conf.d/*.conf in there? Probably not, so those files aren't being read.
Re: [Dovecot] Antispam plugin not compatible with Dovecot 2.1
On Tue, 2012-01-17 at 11:07 +, interfaSys sàrl wrote: Here is what I get when I try to compile the antispam plugin agaisnt Dovecot 2.1 ** mailbox.c: In function 'antispam_save_begin': mailbox.c:138:12: error: 'struct mail_save_context' has no member named 'copying' The copying should be changed to copying_via_save.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.01.2012 21:54, Timo Sirainen wrote: On Wed, 2012-01-18 at 20:44 +0800, Lee Standen wrote: I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. Unfortunately there aren't really any. Everyone who seems to switch to sdbox/mdbox usually also change their hardware at the same time, so there aren't really any before/after metrics. I've of course some unrealistic synthetic benchmarks, but I don't think they are very useful. So, I would also be very interested in seeing some before/after graphs of disk IO, CPU and memory usage of Maildir - dbox switch in same hardware. Maildir is anyway definitely worse performance then sdbox or mdbox. mdbox also uses less NFS operations, but I don't know how much faster (if any) it is with Netapps. We have bought new hardware for this project too, so we might not be able to help out massively on that front... we do have NFS operations monitored though so we should at least be able to compare that metric since the underlying storage operating system is the same. All NetApp hardware runs their Data ONTAP operating system, so the metrics are assured to be the same :) How about this... are there any tools available (that you know of) to capture real live customer POP3/IMAP traffic and replay it against a separate system? That might be a feasible option for doing a like-for-like comparison in our environment? We could probably get something in place to simulate the load if we can do something like that... * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director That might be the one thing (unfortunately) which prevents us from going with the dbox format. I understand the same issue can actually occur on Dovecot Maildir as well, but because Maildir works without these index files, we were willing to just go with it. I will raise it again, but there has been a lot of push back about introducing a single point of failure, even though this is a perceived one. The biggest challenge I have at the moment if I try to sell the dbox format is providing some kind of data on the expected gains from this. If it's only a 10% reduction in NFS operations for the typical user, then it's probably not worth our while. - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers Can the same mailbox still be accessed via multiple geographic servers? I've had some plans for doing this kind of access/replication using dsync.. No, we're using the nginx proxy layer to ensure that if a user in Sydney (for example) tries to access a Perth mailbox, their connection is redirected (by nginx) to the Perth POP/IMAP servers. Postfix configuration is handling the same thing on the LMTP side. The requirement here is for all users to have the same settings regardless of location, but still be able to locate the email servers and data close to the customer. * Apache Lucene indexes will be used to accelerate IMAP search for users Dovecot's fts-solr or fts-lucene? fts-solr. I've been using Lucene/Solr interchangeably when discussing this project with my peers :) Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? If you have webmail type of clients, definitely. For Outlook/Thunderbird you should still see improvement, but not necessarily as much. You didn't mention POP3. That isn't Dovecot's strong point. Its performance should be about the same as Courier-POP3, but could be less than QMail-POP3. Although if many of your POP3 users keep a lot of mails on server it Our existing systems run with about 21K concurrent IMAP connections at any one point in time, not counting Webmail POP3 runs at about 3600 concurrent connections, but since those are not long lived it's not particularly indicative of customer numbers. Vague recollection is something like 25% IMAP, 55-60% POP3, rest 20% Webmail. I'd have to go back and check the breakdown again. * If someone can give some
Re: [Dovecot] LMTP Logging
On Mon, 2012-01-16 at 17:17 -0800, Mark Moseley wrote: Just had a minor suggestion, with no clue how hard/easy it would be to implement: The %f flag in deliver_log_format seems to pick up the From: header, instead of the MAIL FROM:... arg. It'd be handy to have a %F that shows the MAIL FROM arg instead. I'm looking at tracking emails through logs from Exim to Dovecot easily. I know Message-ID can be used for correlation but it adds some complexity to searching, i.e. I can't just grep for the sender (as logged by Exim), unless I assume MAIL FROM always == From: Added to v2.1: http://hg.dovecot.org/dovecot-2.1/rev/7ee2cfbcae2e http://hg.dovecot.org/dovecot-2.1/rev/08cc9d2a79e6
Re: [Dovecot] Quota is not working (Debian Squeeze - Dovecot 1.2)
On Mon, 2012-01-16 at 11:42 +0100, RaSca wrote:
passdb sql {
args = /etc/dovecot/dovecot-sql.conf
}
userdb passwd {
}
userdb static {
args = uid=5000 gid=5000 home=/mail/mailboxes/%d/%n@%d
allow_all_users=yes
}
You're using SQL only for passdb lookup.
plugin {
quota = maildir:/mail/mailboxes/%d/%n@%d
The above path probably doesn't do what you intended. It's only the
user-visible quota root name. It could just as well be User quota or
something.
The db connection works, this is /etc/dovecot/dovecot-sql.conf:
driver = mysql
connect = host=myserver dbname=mail user=myuser password=mypassword
default_pass_scheme = CRYPT
password_query = SELECT username, password FROM mailbox WHERE username='%u'
user_query = SELECT username AS user, maildir AS home,
CONCAT('*:storage=', quota , 'B') AS quota_rule FROM mailbox WHERE
username = '%u' AND active = '1'
user_query isn't used, because you aren't using userdb sql.
Re: [Dovecot] v2.1.rc3 released
On Mon, 2012-01-16 at 17:05 +0200, Kirill A. Shutemov wrote: ./autogen failed: $ ./autogen.sh libtoolize: Consider adding `AC_CONFIG_MACRO_DIR([m4])' to configure.in and libtoolize: rerunning libtoolize, to keep the correct libtool macros in-tree. libtoolize: Consider adding `-I m4' to ACLOCAL_AMFLAGS in Makefile.am. src/plugins/fts/Makefile.am:52: `pkglibexecdir' is not a legitimate directory for `SCRIPTS' Makefile.am:24: `pkglibdir' is not a legitimate directory for `DATA' autoreconf: automake failed with exit status: 1 $ automake --version | head -1 automake (GNU automake) 1.11.2 Looks like automake bug: http://old.nabble.com/Re%3A-Scripts-in-pkglibexecdir--p33070266.html
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
Out of interest, has the NFS issue been tested on NFS4? My understanding is that NFS4 has a lot of fixes for the locking/caching problems that plague NFS3, and we were planning to use NFS4 from day one. If this hasn't been tested, is there some kind of load simulator that we could run to see if the issue does occur in our environment? On 18.01.2012 21:54, Timo Sirainen wrote: On Wed, 2012-01-18 at 20:44 +0800, Lee Standen wrote: I've been desperately trying to find some comparative performance information about the different mailbox formats supported by Dovecot in order to make an assessment on which format is right for our environment. Unfortunately there aren't really any. Everyone who seems to switch to sdbox/mdbox usually also change their hardware at the same time, so there aren't really any before/after metrics. I've of course some unrealistic synthetic benchmarks, but I don't think they are very useful. So, I would also be very interested in seeing some before/after graphs of disk IO, CPU and memory usage of Maildir - dbox switch in same hardware. Maildir is anyway definitely worse performance then sdbox or mdbox. mdbox also uses less NFS operations, but I don't know how much faster (if any) it is with Netapps. * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director - LDA POP/IMAP servers are segmented into geographically split groups (so no server sees every single mailbox) - Nginx proxy used to terminate customer connections, connections are redirected to the appropriate geographic servers Can the same mailbox still be accessed via multiple geographic servers? I've had some plans for doing this kind of access/replication using dsync.. * Apache Lucene indexes will be used to accelerate IMAP search for users Dovecot's fts-solr or fts-lucene? Our closest current live configuration (Qmail SMTP, Courier IMAP, Maildir) has 600K mailboxes and pushes ~ 35,000 NFS operations per second at peak Some of the things I would like to know: * Are we likely to see a reduction in IOPS/User by using Maildir alone under Dovecot? If you have webmail type of clients, definitely. For Outlook/Thunderbird you should still see improvement, but not necessarily as much. You didn't mention POP3. That isn't Dovecot's strong point. Its performance should be about the same as Courier-POP3, but could be less than QMail-POP3. Although if many of your POP3 users keep a lot of mails on server it * If someone can give some technical reasoning behind why mdbox does less IOPS than Maildir? Maildir renames files a lot. From new/ - to cur/ and then every time message flag changes. That's why sdbox is faster. Why mdbox should be faster than sdbox is because mdbox puts (or should put) more mail data physically closer in disks to make reading it faster. I understand some of the reasons for the mdbox IOPS question, but I need some more information so we can discuss internally and make a decision as to whether we're comfortable going with mdbox from day one. We're very familiar with Maidlir, and there's just some uneasiness internally around going to a new mail storage format. It's at least safer to first switch to Dovecot+Maildir to make sure that any problems you might find aren't related to the mailbox format..
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, 2012-01-18 at 22:36 +0800, Lee Standen wrote: How about this... are there any tools available (that you know of) to capture real live customer POP3/IMAP traffic and replay it against a separate system? That might be a feasible option for doing a like-for-like comparison in our environment? We could probably get something in place to simulate the load if we can do something like that... I've thought about that too before, but with IMAP traffic it doesn't work very well. Even if the storages were 100% synchronized at startup, the session states could easily become desynced. For example if client does a NOOP at the same time when two mails are being delivered to the mailbox, serverA might show only one of them while serverB would show two of them because it was executed a tiny bit later. All of the client's future commands could then be affected by this desync. (OK, I wrote the above thinking about a real-time system where you could redirect the client's traffic to two systems, but basically same problems exist for offline replays too. Although it would be easier to fix the replays to handle this.) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director That might be the one thing (unfortunately) which prevents us from going with the dbox format. I understand the same issue can actually occur on Dovecot Maildir as well, but because Maildir works without these index files, we were willing to just go with it. Are you planning on also redirecting POP3/IMAP connections to somewhat randomly to the different servers? I really don't recommend that, even with Maildir.. Some of the errors will be user visible, even if no actual data loss happens. Users may get disconnected, and sometimes might have to clean their client's cache. I will raise it again, but there has been a lot of push back about introducing a single point of failure, even though this is a perceived one. What is a single point of failure there? It's at least safer to first switch to Dovecot+Maildir to make sure that any problems you might find aren't related to the mailbox format.. Yep, I'm considering that. The flip side is that it's actually going to be difficult for us to change mail format once we've migrated into this system, but we have an opportunity for (literally) a month long testing phase beginning in Feb/March which will let us test as many possibilities as we can. The mailbox format switching can be done one user at a time with zero downtime with dsync.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, 2012-01-18 at 23:21 +0800, Lee Standen wrote: Out of interest, has the NFS issue been tested on NFS4? My understanding is that NFS4 has a lot of fixes for the locking/caching problems that plague NFS3, and we were planning to use NFS4 from day one. I've tried with Linux NFS4 server+client a few years ago. It seemed to have all the same caching problems as NFS3. If this hasn't been tested, is there some kind of load simulator that we could run to see if the issue does occur in our environment? http://imapwiki.org/ImapTest should easily trigger it. Just run it against two servers, both hammering the same mailbox.
Re: [Dovecot] 2.1.rc3 (1a722c7676bb): doveadm mailbox list - Segmentation fault
On Sun, 2012-01-15 at 14:11 +0100, Pascal Volk wrote: Core was generated by `doveadm mailbox list -u jane@example.com /*'. Finally fixed: http://hg.dovecot.org/dovecot-2.1/rev/99ea6da7dc99
Re: [Dovecot] v2.x services documentation
On Sat, 2012-01-14 at 18:03 +0100, Axel Luttgens wrote: Up to now, I only had the opportunity to quickly read the wiki page, and have a small question; one may read: process_min_avail Minimum number of processes that always should be available to accept more client connections. For service_limit=1 processes this decreases the latency for handling new connections. For service_limit!=1 processes it could be set to the number of CPU cores on the system to balance the load among them. What's that service_limit setting? Thanks, fixed. Was supposed to be service_count.
Re: [Dovecot] Antispam plugin not compatible with Dovecot 2.1
On Wed, 18 Jan 2012 16:34:18 +0200, Timo Sirainen t...@iki.fi wrote: On Tue, 2012-01-17 at 11:07 +, interfaSys sàrl wrote: Here is what I get when I try to compile the antispam plugin agaisnt Dovecot 2.1 ** mailbox.c: In function 'antispam_save_begin': mailbox.c:138:12: error: 'struct mail_save_context' has no member named 'copying' The copying should be changed to copying_via_save. Thank you, Timo. Would #if DOVECOT_IS_GE(2,1) suffice or do I need anything more specific? -- Eugene Paskevich | *==)--- | Plug me into eug...@raptor.kiev.ua| ---(==* | The Matrix
Re: [Dovecot] Antispam plugin not compatible with Dovecot 2.1
On Wed, 2012-01-18 at 18:19 +0200, Eugene Paskevich wrote: mailbox.c: In function 'antispam_save_begin': mailbox.c:138:12: error: 'struct mail_save_context' has no member named 'copying' The copying should be changed to copying_via_save. Thank you, Timo. Would #if DOVECOT_IS_GE(2,1) suffice or do I need anything more specific? Where do you expect to find such macro? ;) Hm. Perhaps I should try to add one.
Re: [Dovecot] Antispam plugin not compatible with Dovecot 2.1
On Wed, 18 Jan 2012 18:31:49 +0200, Timo Sirainen t...@iki.fi wrote: On Wed, 2012-01-18 at 18:19 +0200, Eugene Paskevich wrote: mailbox.c: In function 'antispam_save_begin': mailbox.c:138:12: error: 'struct mail_save_context' has no member named 'copying' The copying should be changed to copying_via_save. Thank you, Timo. Would #if DOVECOT_IS_GE(2,1) suffice or do I need anything more specific? Where do you expect to find such macro? ;) Hm. Perhaps I should try to add one. Heh. That's Johannes' package private macro... :) -- Eugene Paskevich | *==)--- | Plug me into eug...@raptor.kiev.ua| ---(==* | The Matrix
Re: [Dovecot] LMTP Logging
On Wed, Jan 18, 2012 at 6:52 AM, Timo Sirainen t...@iki.fi wrote: On Mon, 2012-01-16 at 17:17 -0800, Mark Moseley wrote: Just had a minor suggestion, with no clue how hard/easy it would be to implement: The %f flag in deliver_log_format seems to pick up the From: header, instead of the MAIL FROM:... arg. It'd be handy to have a %F that shows the MAIL FROM arg instead. I'm looking at tracking emails through logs from Exim to Dovecot easily. I know Message-ID can be used for correlation but it adds some complexity to searching, i.e. I can't just grep for the sender (as logged by Exim), unless I assume MAIL FROM always == From: Added to v2.1: http://hg.dovecot.org/dovecot-2.1/rev/7ee2cfbcae2e http://hg.dovecot.org/dovecot-2.1/rev/08cc9d2a79e6 You're awesome, thanks!
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
snip * All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames) * Postfix will feed new email to Dovecot via LMTP * Dovecot servers have been split based on their role - Dovecot LDA Servers (running LMTP protocol) - Dovecot POP/IMAP servers (running POP/IMAP protocols) You're going to run into NFS caching troubles with the above split setup. I don't recommend it. You will see error messages about index corruption with it, and with dbox it can cause metadata loss. http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director That might be the one thing (unfortunately) which prevents us from going with the dbox format. I understand the same issue can actually occur on Dovecot Maildir as well, but because Maildir works without these index files, we were willing to just go with it. I will raise it again, but there has been a lot of push back about introducing a single point of failure, even though this is a perceived one. /snip I'm in the middle of working on a Maildir-mdbox migration as well, and likewise, over NFS (all Netapps but moving to Sun), and likewise with split LDA and IMAP/POP servers (and both of those served out of pools). I was hoping doing things like setting mail_nfs_index = yes and mmap_disable = yes and mail_fsync = always/optimized would mitigate most of the risks of index corruption, as well as probably turning indexing off on the LDA side of things--i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.1.2012, at 19.54, Mark Moseley wrote: I'm in the middle of working on a Maildir-mdbox migration as well, and likewise, over NFS (all Netapps but moving to Sun), and likewise with split LDA and IMAP/POP servers (and both of those served out of pools). I was hoping doing things like setting mail_nfs_index = yes and mmap_disable = yes and mail_fsync = always/optimized would mitigate most of the risks of index corruption, They help, but aren't 100% effective and they also make the performance worse. as well as probably turning indexing off on the LDA side of things You can't turn off indexing with dbox. --i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed. What's the problem with director-based architecture?
Re: [Dovecot] imap-login process_limit reached
Timo Sirainen wrote: On Mon, 2012-01-16 at 14:41 -0800, Don Buchholz wrote: I've been having some problems with IMAP user connections to the Dovecot (v2.0.8) server. The following message is being logged. Jan 16 10:51:36 postal dovecot: master: Warning: service(imap-login): process_limit reached, client connections are being dropped Maybe this will help some in future: http://hg.dovecot.org/dovecot-2.1/rev/a4e61c99c7eb The new error message is: service(imap-login): process_limit (100) reached, client connections are being dropped Great idea! Thanks, Timo. - Don
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, Jan 18, 2012 at 07:58:31PM +0200, Timo Sirainen wrote: --i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed. What's the problem with director-based architecture? It hasn't been working reliably for lmtp in v2.0. To quote yourself: 888-8-8-88-888-- I think the way I originally planned LMTP proxying to work is simply too complex to work reliably, perhaps even if the code was bug-free. So instead of reading+writing DATA at the same time, this patch changes the DATA to be first read into memory or temp file, and then from there read and sent to the LMTP backends: http://hg.dovecot.org/dovecot-2.1/raw-rev/51d87deb5c26 888-8-8-88-888-- unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not... -jf
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.1.2012, at 20.51, Jan-Frode Myklebust wrote: On Wed, Jan 18, 2012 at 07:58:31PM +0200, Timo Sirainen wrote: --i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed. What's the problem with director-based architecture? It hasn't been working reliably for lmtp in v2.0. Yes, besides that :) To quote yourself: 888-8-8-88-888-- I think the way I originally planned LMTP proxying to work is simply too complex to work reliably, perhaps even if the code was bug-free. So instead of reading+writing DATA at the same time, this patch changes the DATA to be first read into memory or temp file, and then from there read and sent to the LMTP backends: http://hg.dovecot.org/dovecot-2.1/raw-rev/51d87deb5c26 888-8-8-88-888-- unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not... I'm not sure if that patch is useful or not. The important patch to fix it is http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, Jan 18, 2012 at 9:58 AM, Timo Sirainen t...@iki.fi wrote: On 18.1.2012, at 19.54, Mark Moseley wrote: I'm in the middle of working on a Maildir-mdbox migration as well, and likewise, over NFS (all Netapps but moving to Sun), and likewise with split LDA and IMAP/POP servers (and both of those served out of pools). I was hoping doing things like setting mail_nfs_index = yes and mmap_disable = yes and mail_fsync = always/optimized would mitigate most of the risks of index corruption, They help, but aren't 100% effective and they also make the performance worse. In testing, it seemed very much like the benefits of reducing IOPS by up to a couple orders of magnitude outweighed having to use those settings. Both in scripted testing and just using a mail UI, with the NFS-ish settings, I didn't notice any lag and doing things like checking a good-sized mailbox were at least as quick as Maildir. And I'm hoping that reducing IOPS across the entire set of NFS servers will compound the benefits quite a bit. as well as probably turning indexing off on the LDA side of things You can't turn off indexing with dbox. Ah, too bad. I was hoping I could get away with the LDA not updating the index but just dropping the message into storage/m.# but it'd still be seen on the IMAP/POP side--but hadn't tested that. Guess that's not the case. --i.e. all the suggestions at http://wiki2.dovecot.org/NFS. Is that definitely not the case? Is there anything else (beyond moving to a director-based architecture) that can mitigate the risk of index corruption? In our case, incoming IMAP/POP are 'stuck' to servers based on IP persistence for a given amount of time, but incoming LDA is randomly distributed. What's the problem with director-based architecture? Nothing, per se. It's just that migrating to mdbox *and* to a director architecture is quite a bit more added complexity than simply migrating to mdbox alone. Hopefully, I'm not hijacking this thread. This seems pretty pertinent as well to the OP.
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On Wed, Jan 18, 2012 at 09:03:18PM +0200, Timo Sirainen wrote: On 18.1.2012, at 20.51, Jan-Frode Myklebust wrote: What's the problem with director-based architecture? It hasn't been working reliably for lmtp in v2.0. Yes, besides that :) Besides that it's great! unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not... I'm not sure if that patch is useful or not. The important patch to fix it is http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c So with that oneliner on our directors, you expect lmtp proxying trough director to be better than lmtp to rr-dns towards backend servers? If so, I guess we should give it another try. -jf
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.1.2012, at 22.14, Jan-Frode Myklebust wrote: unfortunately I haven't tested that patch, so I have no idea if it fixed the issues or not... I'm not sure if that patch is useful or not. The important patch to fix it is http://hg.dovecot.org/dovecot-2.0/rev/71084b799a6c So with that oneliner on our directors, you expect lmtp proxying trough director to be better than lmtp to rr-dns towards backend servers? If so, I guess we should give it another try. It should fix the hangs that were common. I'm not sure if it fixes everything without the complexity reduction patch.
[Dovecot] Quota won't work
I tried to set a quota setting. I installed dovecot with newest
version, patched it and started it.
dovecot -n:
# 1.2.15: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-5-amd64 x86_64 Debian 6.0.3 ext4
log_timestamp: %Y-%m-%d %H:%M:%S
protocols: imap imaps pop3 pop3s
ssl_listen: 143
ssl_cipher_list: ALL:!LOW:!SSLv2
disable_plaintext_auth: no
login_dir: /var/run/dovecot/login
login_executable(default): /usr/lib/dovecot/imap-login
login_executable(imap): /usr/lib/dovecot/imap-login
login_executable(pop3): /usr/lib/dovecot/pop3-login
mail_privileged_group: mail
mail_location: maildir:/var/vmail/%d/%n/Maildir
mbox_write_locks: fcntl dotlock
mail_executable(default): /usr/lib/dovecot/imap
mail_executable(imap): /usr/lib/dovecot/imap
mail_executable(pop3): /usr/lib/dovecot/pop3
mail_plugins(default): quota imap_quota
mail_plugins(imap): quota imap_quota
mail_plugins(pop3): quota
mail_plugin_dir(default): /usr/lib/dovecot/modules/imap
mail_plugin_dir(imap): /usr/lib/dovecot/modules/imap
mail_plugin_dir(pop3): /usr/lib/dovecot/modules/pop3
namespace:
type: private
inbox: yes
list: yes
subscriptions: yes
lda:
postmaster_address: postmas...@opsys.de
mail_plugins: sieve quota
log_path:
auth default:
mechanisms: plain login
verbose: yes
passdb:
driver: sql
args: /etc/dovecot/dovecot-sql.conf
userdb:
driver: static
args: uid=5000 gid=5000 home=/var/vmail/%d/%n/Maildir
allow_all_users=yes
socket:
type: listen
client:
path: /var/spool/postfix/private/auth
mode: 432
user: postfix
group: postfix
master:
path: /var/run/dovecot/auth-master
mode: 384
user: vmail
/etc/dovecot/dovecot-sql.conf:
driver = mysql
connect = host=127.0.0.1 dbname=mailserver user=mailuser
password=**
default_pass_scheme = PLAIN-MD5
password_query = SELECT email as user, password FROM virtual_users
WHERE email='%u';
user_query = SELECT CONCAT('/var/mail/', maildir) AS home,
CONCAT('*:bytes=', quota) AS quota_rule \
FROM virtual_users WHERE email='%u'
virtual_users has this:
CREATE TABLE IF NOT EXISTS `virtual_users` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`domain_id` int(11) NOT NULL,
`password` varchar(32) NOT NULL,
`email` varchar(100) NOT NULL,
`quota` int(11) NOT NULL DEFAULT '629145600',
PRIMARY KEY (`id`),
UNIQUE KEY `email` (`email`),
KEY `domain_id` (`domain_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Also postfix is installed with this (not the hole cfg):
virtual_mailbox_domains =
mysql:/etc/postfix/mysql-virtual-mailbox-domains.cf
virtual_mailbox_limit_inbox = no
virtual_mailbox_limit_maps = mysql:/etc/postfix/mysql-quota.cf
virtual_mailbox_limit_override = yes
virtual_mailbox_maps = mysql:/etc/postfix/mysql-virtual-mailbox-maps.cf
virtual_maildir_extended = yes
virtual_maildir_limit_message = The user you are trying to reach is
over quota.
virtual_maildir_limit_message_maps = mail:/etc/postfix/mysql-quota.cf
virtual_overquota_bounce = yes
/etc/postfix/mysql-quota.cf:
user = mailuser
password = **
hosts = 127.0.0.1
dbname = mailserver
query = SELECT quota FROM virtual_users WHERE email='%s'
I changed the quota of my mail account to 40, so 40Byte should be the
maximum.
My account is at a size of 600KB now. I still recieve mails, also they
will be saved without errors.
/var/log/mail.log says nothing to quota, just normal recieve and store
entries.
What to fix?
--
Markus Fritz
Administration
opsys.de
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 18.1.2012, at 21.49, Mark Moseley wrote: What's the problem with director-based architecture? Nothing, per se. It's just that migrating to mdbox *and* to a director architecture is quite a bit more added complexity than simply migrating to mdbox alone. Yes, I agree it's safer to do one thing that a time. That's why I'd do a switch to director first. :)
Re: [Dovecot] Quota won't work
On 18.1.2012, at 22.41, Markus Fritz wrote:
passdb:
driver: sql
args: /etc/dovecot/dovecot-sql.conf
userdb:
driver: static
args: uid=5000 gid=5000 home=/var/vmail/%d/%n/Maildir allow_all_users=yes
You use sql as passdb, static as userdb.
password_query = SELECT email as user, password FROM virtual_users WHERE
email='%u';
passdb sql executes password_query.
user_query = SELECT CONCAT('/var/mail/', maildir) AS home, CONCAT('*:bytes=',
quota) AS quota_rule \
FROM virtual_users WHERE email='%u'
userdb sql executes user_query. But you're not using userdb sql, you're using
userdb static. This query never gets executed.
Also you don't have plugin { quota } setting.
Re: [Dovecot] Panic: file mbox-sync.c: line 1348: assertion failed
Am 10.01.2012 16:32, schrieb Jürgen Obermann: I have the following problem with doveadm: # gdb --args /opt/local/bin/doveadm -v mailbox status -u userxy/g029 'messages' Software-alle/AK-Software-Tagung GNU gdb 5.3 Copyright 2002 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type show copying to see the conditions. There is absolutely no warranty for GDB. Type show warranty for details. This GDB was configured as sparc-sun-solaris2.8... (gdb) run Starting program: /opt/local/bin/doveadm -v mailbox status -u g029 messages Software-alle/AK-Software-Tagung warning: Lowest section in /lib/libthread.so.1 is .dynamic at 0074 warning: Lowest section in /lib/libdl.so.1 is .hash at 00b4 doveadm(g029): Panic: file mbox-sync.c: line 1348: assertion failed: (file_size = sync_ctx-expunged_space + trailer_size) doveadm(g029): Error: Raw backtrace: 0xff1cbc30 - 0xff319544 - 0xff319fa8 - 0xff31add8 - 0xff31b278 - 0xff2a69b0 - 0xff2a6bac - 0x16808 - 0x1b8fc - 0x16ba0 - 0x177cc - 0x17944 - 0x17a50 - 0x204e8 - 0x165c8 Program received signal SIGABRT, Aborted. Hallo, the problem went away after I deleted the dovecot index files for the mailbox. Greetins, Jürgen Obermann Hochschulrechenzentrum der Justus-Liebig-Universität Gießen Heinrich-Buff-Ring 44 Tel. 0641-9913054
Re: [Dovecot] Performance of Maildir vs sdbox/mdbox
On 1/18/2012 7:54 AM, Timo Sirainen wrote:
On Wed, 2012-01-18 at 20:44 +0800, Lee Standen wrote:
* All mail storage presented via NFS over 10Gbps Ethernet (Jumbo Frames)
* Postfix will feed new email to Dovecot via LMTP
* Dovecot servers have been split based on their role
- Dovecot LDA Servers (running LMTP protocol)
- Dovecot POP/IMAP servers (running POP/IMAP protocols)
You're going to run into NFS caching troubles with the above split
setup. I don't recommend it. You will see error messages about index
corruption with it, and with dbox it can cause metadata loss.
http://wiki2.dovecot.org/NFS http://wiki2.dovecot.org/Director
Would it be possible to fix this NFS mdbox index corruption issue in
this split scenario by using a dual namespace and disabling indexing on
the INBOX? The goal being no index file collisions between LDA and imap
processes. Maybe something like:
namespace {
separator = /
prefix = #mbox/
location = mbox:~/mail:INBOX=/var/mail/%u:INDEX=MEMORY
inbox = yes
hidden = yes
list = no
}
namespace {
separator = /
prefix =
location = mdbox:~/mdbox
}
Client access to new mail might be a little slower, but if it eliminates
the index corruption issue and allows the split architecture, it may be
a viable option.
--
Stan
Re: [Dovecot] shared folder files not displaying in thunderbird
Can anyone help me figure out why email in a sub-folder (created using
Thunderbird) of a dovecot namespace will not display in Thunderbird?
...
Hello,
I have dovecot installed with the configuration below.
One of the subfolders created (using the email client) under the
'/home/vpopmail/domains/mydomain.com/shared/projects' share no longer
(it used to) displays the files located in it. There are about 150
folders under the '/home/vpopmail/domains/mydomain.com/shared/projects'
share all of which display the files located in them, the one mentioned
used to display the contents but no longer does.
What would be the reason that one folder would no longer display
existing files in the email client (Thunderbird) and the other folders
would? And, how do I fix this?
I've already tried unsubscribing and resubscribing the folder. This did
not work.
Would it now be simply a matter of unsubscribing the folder, deleting
the dovecot files, and resubscribing to the folder?
Eric
# 2.0.11: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.18-238.19.1.el5 i686 CentOS release 5.7 (Final)
auth_cache_size = 32 M
auth_mechanisms = plain login digest-md5 cram-md5
auth_username_format = %Lu
disable_plaintext_auth = no
first_valid_uid = 89
log_path = /var/log/dovecot.log
login_greeting = Dovecot toaster ready.
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
namespace {
location = maildir:/home/vpopmail/domains/mydomain.com/shared/projects
prefix = projects.
separator = .
type = public
}
passdb {
args = cache_key=%u webmail=127.0.0.1
driver = vpopmail
}
plugin/quota = maildir
protocols = imap
ssl_cert = /var/qmail/control/servercert.pem
ssl_key = /var/qmail/control/servercert.pem
userdb {
args = cache_key=%u quota_template=quota_rule=*:backend=%q
driver = vpopmail
}
protocol imap {
imap_client_workarounds = delay-newmail
}
