[Dovecot] empty emails with 2.2.13/pigeonhole
I recently upgraded from 2.2.10 to 2.2.13, and also upgraded pigeonhole. Ever since im seeing some empty emails appear in my inbox. Return-Path: cric...@stats2.xs4all.net Delivered-To: cor Received: from imapdirector1.xs4all.net ([194.109.26.173]) by userimap9.xs4all.nl (Dovecot) with LMTP id CHu0MUDDjlOBdwAAwvsqNQ for cor; Wed, 04 Jun 2014 08:57:04 +0200 Thats all this mail contains. This only happens using LMTP/Sieve, and with many different senders/recipients. Debug log shows nothing out of the ordinary, except that the mail seems to be corrupted by the time it hits sieve. It's very hard to diagnose. Our path is kind of lengthy, and in theory it could happen everywhere. Mail first arrives on incoming MX server which is sendmail. From there it's sent to a dovecot director using lmtp, which sends it to a server that does final delivery with sieve. Im not sure where the email gets corrupted. Any extra debugging I could turn on? Two extra things to note. So far it seems to only happen to emails that would have hit a matching sieve filter rule. I have asked a dozen people to verify this, and all emails would have matched a filter if all was well. Instead, it gets dropped into the inbox because pigeonhole seems to be working with the corrupt file and cant match any rules anymore. And second, 99% of our email is handled differently, from the sendmail server directly into a local mailer, and that never causes issues. We're in the process of switching to lmtp. Any insights? cor
Re: [Dovecot] possible nfs issue
On Oct 4, 2012, at 3:55 AM, Kelsey Cummings k...@corp.sonic.net wrote: On 10/2/2012 2:39 PM, Cor Bosman wrote: Anyone else with NFS mailspools seeing this? Yes, it is like 1999 all over again. I haven't had a chance to track them down or setup a cron job to rm them all. All of the ones I'm seeing are ex dovecot.index files but it looks like yours are ex messages? I figured this was a probably a regression in the RHEL6.3 (Sl6.3) (2.6.32-279.9.1.el6.x86_64) kernel. What are you running Cor? We're running debian with a 3.2.2 kernel. Just yesterday one of my colleagues had a few new ones in his mailspool. Definitely no server crash or anything. Something is creating these outside the 'normal' parameters for .nfs files. My colleague said these were emails he deleted that day. We've set up a cleaning run, and are probably going to ignore it for now. These things are near impossible to track down without a lot of debugging. Cor
[Dovecot] possible nfs issue
Hi all, we've started receiving complaints from users that seemingly use more quota than they actually have. We noticed that these users have (in some cases many) .nfs files in their mailspool. Some of our admins checked their own dirs, and noticed them there as well. This could of course be unrelated to dovecot (kernel issue, netapp issue) but maybe somehow has an idea about if dovecot could cause this. This has been going on for at least a year, not really enough to notice before now. root@userimap1# find . -type f|grep -i .nfs ./cur/.nfs003967ad003c0603 ./cur/.nfs0757b44b003be609 ./cur/.nfs035e89bd003be60b ./cur/.nfs0796251c003be60c ./cur/.nfs0796251f003be60e ./cur/.nfs0262f9a1003be33a ./cur/.nfs096513f3003be524 ./cur/.nfs07962525003be60f ./cur/.nfs03e7d8ab003be62b ./cur/.nfs026f4fad003be50d ./cur/.nfs00bdaeab003c0611 ./cur/.nfs05da42c7003be525 ./cur/.nfs03d74729003be526 ./cur/.nfs0229769e003be535 ./cur/.nfs0440969e003be516 With NFS these files are created when a file gets unlinked, but another process still has it open. It disappears as soon as the other process closes it. For some reason they dont disappear. As far as I can tell we've had no server crashes that could explain this. One possible theory is that a rename happens after an unlink. In that case the file remains. This could possibly be a dovecot issue. Anyone else with NFS mailspools seeing this? Cor
Re: [Dovecot] possible nfs issue
On Oct 3, 2012, at 12:35 AM, Timo Sirainen t...@iki.fi wrote: On 3.10.2012, at 0.45, Timo Sirainen wrote: On 3.10.2012, at 0.39, Cor Bosman wrote: With NFS these files are created when a file gets unlinked, but another process still has it open. It disappears as soon as the other process closes it. For some reason they dont disappear. As far as I can tell we've had no server crashes that could explain this. One possible theory is that a rename happens after an unlink. In that case the file remains. This could possibly be a dovecot issue. How can a rename happen after unlink? The rename should fail. (Unless doing rename(.nfs1234, something), but Dovecot definitely isn't doing that.) You could see if this old test program leaves .nfs files lying around: http://dovecot.org/tmp/readdir.c Just comment out the line: close(fd); I meant the .nfs1234 indeed, but it seemed very unlikely. Thanks for clarifying. The readdir program leaves no .nfs files. We'll have to explore other possibilities. Cor
Re: [Dovecot] v2.2 status update: IMAP NOTIFY extension and more
I'm also considering implementing an SMTP submission server, which works only as a proxy to the real SMTP server. The benefits of it would mainly be: What would be really cool is if you also kept statistics on certain metrics, like how many emails a specific sender has sent. If this is done right, it could become a centralised spam sender back-off system over multiple smtp servers. Maybe something for the future. We now pay for a commercial system that implements this. Cor
Re: [Dovecot] v2.2 status update: IMAP NOTIFY extension and more
What would be really cool is if you also kept statistics on certain metrics, like how many emails a specific sender has sent. If this is done right, it could become a centralised spam sender back-off system over multiple smtp servers. Maybe something for the future. We now pay for a commercial system that implements this. The submission servers would need to share state with each others to make this reliable. Some nosql database would work for that I guess (memcached?) Can you describe more fully what you would want it to do and/or what the current system does? Probably not something I'll implement soon, but would be good to know for future :) They would indeed need to share the state, this is what the commercial system does (Cloudmark Gateway). The idea is to prevent senders to send email beyond certain limits that are configurable. Lets say, 200 emails in a minute, or 500 an hour, or whatever. Of course allowing for whitelists. With ever increasing speeds of residential connections (we have hundreds of fiber connections now, growing rapidly), these become very fast spam sources without something to keep them in check. It's easy on 1 server, not so easy if you have dozens of smtp servers. But yes, something for a distant future :) And CM has other features as well, so it may not be so easy to replace :) Cor
[Dovecot] interesting stats pattern
Hey all, im experimenting with dovecot stats service, and graphing the result. My initial results are kind of interesting. Check out this graph showing connected sessions and users: http://grab.by/dReu At first I thought maybe one of our 35 imap servers was having issues sending data, but all individual servers show this patters. Here is a bunch of individual servers: http://grab.by/dReC Anyone have any idea what could cause such a pattern? Maybe dovecot does some cleaning up of idle sessions at specific intervals? Or maybe our loadbalancers do, or the imapdirectors. regards, Cor
Re: [Dovecot] interesting stats pattern
On May 29, 2012, at 2:21 PM, Timo Sirainen wrote: On 29.5.2012, at 13.23, Cor Bosman wrote: Hey all, im experimenting with dovecot stats service, and graphing the result. My initial results are kind of interesting. Check out this graph showing connected sessions and users: http://grab.by/dReu How do you get the list? Are you periodically just getting list of sessions/users with doveadm stats dump? Anyone have any idea what could cause such a pattern? Maybe dovecot does some cleaning up of idle sessions at specific intervals? Or maybe our loadbalancers do, or the imapdirectors. doveadm stats dump by default dumps a lot of historic data as well. If you want to see only the currently connected sessions/users, add connected parameter. Note that I'm not entirely sure what would be the best API for getting the stats. I'm also thinking that for best behavior the stats process should simply be dumping the data to some permanent database, and you'd do the lookups from there. Otherwise data is lost when Dovecot restarts. The dumping to db could already be done with a doveadm stats dump cronjob that runs e.g. once a minute. And/or perhaps stats process should be saving its state permanently to /var/lib/dovecot/ and loading it at startup. Still, a permanent DB would probably be better for some purposes. Yes, I am getting a list of sessions/users every 5 minutes through cron. Im already using doveadm stats dump session/user connected It's not a big deal or anything, just wondering about the weird patterns. If it's really dropping/gaining connections, id like to figure out why. Cor
Re: [Dovecot] interesting stats pattern
On 29.5.2012, at 21.03, Cor Bosman wrote: es, I am getting a list of sessions/users every 5 minutes through cron. Im already using doveadm stats dump session/user connected Actually that's not really correct behavior either, since it ignores all the connections that happened during the 5 minutes if they don't exist at the time when you're asking for them. I'm not sure what the most correct way to do this kind of a graph would be :) I dont really need to know how many total connections ive had. More an idea of amount of concurrent sessions/users. Wouldnt this be pretty accurate? It's not a big deal or anything, just wondering about the weird patterns. If it's really dropping/gaining connections, id like to figure out why. Are you only counting imap/pop3 sessions or also others? Anything that touches mailboxes are counted as sessions (lda, lmtp, doveadm, indexer, .) We only have imap connections for now, nothing else. Cor
Re: [Dovecot] index IO patterns
Alternatively, does anyone have any experience with other redundant storage options? Im thinking things like MooseFS, DRBD, etc? You seem to be interested in multi-site clustering/failover solutions, not simply redundant storage. These two are clustering software solutions but DRBD is not suitable for multi-site use, and MooseFS doesn't seem to be either. MooseFS is based heavily on FUSE, so performance will be far less than optimal. MooseFS is a distributed filesystem, and as with all other distributed/cluster filesystems its metadata performance will suffer, eliminating maildir as a mail store option. Can you provide more specifics on your actual storage architecture needs? There are some people in our company that like MooseFS, so i'll just include it in the tests and let that speak for itself :) We are not looking for multisite solutions. Then we may as well stay with the metrocluster. I dont even care if it has to be in the same rack. It's only for the indexes, not the mail store itself which will stay on the metrocluster. In the very worst case, when the whole site explodes, i can always tell dovecot to use memory for indexes temporarily :) The indexes are doing a lot of iops on the metrocluster, and it's a bit of an expensive option for something it's not even that good at. Im aiming for something with 2 servers, each with a 12 disk enclosure with SSD for fast random io with 10G network interfaces, 24 core, 48GB memory. I just want to test some io patterns on different hardware/software solutions, including the metrocluster itself, before we commit to a specific solution. Im slightly leaning towards DRBD right now. Cor
Re: [Dovecot] index IO patterns
Mail is always a random IO workload, unless your mailbox count is 1, whether accessing indexes or mail files. Regarding the other two questions, you'll likely need to take your own measurements. Wait, maybe there is a misunderstanding. I mean the IO inside one index file, not across the different mailboxes. So within 1 index file that covers a mailbox with say 10.000 emails, how does the IO occur. I would guess pretty random as well, but on the other hand i guess in some ways it could be pretty linear too. If dovecot keeps most changes in memory and writes it all back in 1 go. Cor
[Dovecot] index IO patterns
Hey all, we're in the process of checking out alternatives to our index storage. We're currently storing indexes on a NetApp Metrocluster which works fine, but is very expensive. We're planning a few different setups and doing some actual performance tests on them. Does anyone know some of the IO patterns of the indexes? For instance: - mostly random reads or linear reads/writes? - average size of reads and writes? - how many read/writes on average for a specific mailbox size? Anyone do any measurements of this kind? Alternatively, does anyone have any experience with other redundant storage options? Im thinking things like MooseFS, DRBD, etc? regards, Cor
Re: [Dovecot] index IO patterns
Hi javier, Indexes are very random, mostly read, some writes if using dovecot-lda (ej: dbox). The average size is rather small, maybe 5 KB in our setup. Bandwith is rather low, 20-30 MB/sec Even without LDA/LMTP dovecot-imap needs to write right? It would need to update the index every time an imap connect happens and new mails are found in the mail store. Cor
Re: [Dovecot] Log messages
My guess is that I need to recompile and reinstall dovecot pidgeon (dovecot-2.1-pigeonhole-0.3.0) as well as dovecot but cannot find any documentation on this. Is my gues correct? Is there anything else that is needed to upgrade from 2.1.1 to 2.1.5? Your guess is correct, always recompile sieve plugin when you upgrade dovecot. Cor
Re: [Dovecot] trash plugin bug?
Well, normally you shouldn't be over quota I guess.. :) Anyway, http://hg.dovecot.org/dovecot-2.1/rev/ec8564741aa8 http://hg.dovecot.org/dovecot-2.1/rev/dd3798681283 This indeed fixed the problem. Thank you, Cor
Re: [Dovecot] trash plugin bug?
Hey all, has anyone ever tried turning on the trash plugin when the directory is already over quota? I see some messages being deleted, but it seems it just deletes enough to fit the new email, not enough to go below quota. Well, normally you shouldn't be over quota I guess.. :) Anyway, :) It's hard to initialize new quota rules on a large userbase without some users falling outside the expected parameters :) Sucks when you have lots of users, you always always hit edge cases. Thanks! Cor
Re: [Dovecot] quota not being calculated
http://hg.dovecot.org/dovecot-2.1/rev/4c8f79d1f9f1 should fix it with dict quota. Thank you, this fixed it with dict quota. Cor
[Dovecot] trash plugin bug?
Hey all, has anyone ever tried turning on the trash plugin when the directory is already over quota? I see some messages being deleted, but it seems it just deletes enough to fit the new email, not enough to go below quota. Example: . getquotaroot Spam * QUOTAROOT Spam User quota Spam Quota * QUOTA User quota (STORAGE 591036 5242880) * QUOTA Spam Quota (STORAGE 21173 20480) I now move a 5K message into this folder, and I see: Apr 23 22:26:43 lmtp1 dovecot: imap(cor): expunge: box=Spam, uid=226337, msgid=3yvkiie4ii6r0064ba7im55c2q4md01vx12...@shedevils.com, size=12984 Apr 23 22:26:43 lmtp1 dovecot: imap(cor): expunge: box=Spam, uid=226338, msgid=aab6bffa3ad83767d4a7bc571c314...@bonus-news.com, size=5623 Apr 23 22:26:43 lmtp1 dovecot: imap(cor): save: box=Spam, uid=227759, msgid=20120423191509.f3b584a3...@signal.groundspeak.com, size=2694 And now: . getquotaroot Spam * QUOTAROOT Spam User quota Spam Quota * QUOTA User quota (STORAGE 591016 5242880) * QUOTA Spam Quota (STORAGE 21158 20480) . OK Getquotaroot completed. This seems like a possible bug. For one, it shouldnt be allowed to actually save a message into that folder, but it does. The email does end up in the Spam folder even though the folder is over quota. And of course it doesnt clear up enough space to have the new email fit, it just deleted enough to match the 5K. Without the trash plugin quota is being properly enforced. Everything works fine if you start out below the quota limit btw. Regards, Cor ps: this is 2.1.4, but with the recent trash and quota patches applied.
Re: [Dovecot] Dovecot upgrade
On Apr 22, 2012, at 10:03 AM, Jos Chrispijn wrote: Can someone tell me how I can upgrade from Dovecot 1.x to 2.x best? thanks for your reply, Jos Chrispijn Have you read this? http://wiki2.dovecot.org/Upgrading/2.0 Cor
Re: [Dovecot] Correction of home and mail directory config
Everything concerning sieve should be in the home dir. Why? It can be anywhere you want as long as it doesnt conflict with the names of your mailstore. Cor
Re: [Dovecot] quota not being calculated
It looks like my quota isnt being calculated properly after I started
setting quota to a specific folder. The quota in that folder is always
starting out at 0, and only new email is being added to the quota. If I
remove the maildirsize file, and recalculate, it still starts at 0. Once
email arrives in the folder, it gets added to maildirsize. This is with
maildirquota, in perhaps a bit of an exotic config. The idea is to set a
specific quota to Spam folder, using the trash plugin to keep it at a
certain size.
I'll try later, but try if it would work with dict quota file?
http://wiki2.dovecot.org/Quota/Dict
This almost works, but maybe there's a small bug. Config (2.1.4)
namespace spam {
type = private
separator = /
inbox = no
prefix = Spam/
location =
maildir:%h/.Spam:INDEX=/var/spool/mail/dovecot-control/indexes/%1u/%2u/%u/.Spam:CONTROL=/var/spool/mail/dovecot-control/%1u/%2u/%u/.Spam
subscriptions = no
list = yes
hidden = yes
}
plugin {
quota = fs:User quota
quota2 = dict:Spam Quota::ns=spam:file:%h/spam-quota
quota2_rule = *:storage=20M
}
Now I see:
. getquotaroot Spam
* QUOTAROOT Spam User quota Spam Quota
* QUOTA User quota (STORAGE 10128 1228800)
* QUOTA Spam Quota (STORAGE 8727 20480)
The problem is that the storage calculated for the spam ns is wrong. It is less
than 1MB. The 8727 seems to be the whole maildir including all folders.
This seems to be caused by:
Apr 21 10:00:11 lmtp1 dovecot: imap(cor): Error: quota: Unknown namespace: spam
Regards,
Cor
Re: [Dovecot] quota not being calculated
On Apr 21, 2012, at 12:32 PM, Timo Sirainen wrote: On 21.4.2012, at 11.01, Cor Bosman wrote: prefix = Spam/ . quota2 = dict:Spam Quota::ns=spam:file:%h/spam-quota . Apr 21 10:00:11 lmtp1 dovecot: imap(cor): Error: quota: Unknown namespace: spam Oh. It would make more sense to have ns=spam, but unfortunately I hadn't thought of namespace names before v2.1. You need to use ns=Spam/ Exact same issue with dict quota. It starts out at 0 (even though there are 116 messages in the Spam folder. Then when 1 email gets saved to the Spam folder, the counter is increased by the size of that 1 email. Cor
[Dovecot] quota not being calculated
It looks like my quota isnt being calculated properly after I started setting
quota to a specific folder. The quota in that folder is always starting out at
0, and only new email is being added to the quota. If I remove the maildirsize
file, and recalculate, it still starts at 0. Once email arrives in the folder,
it gets added to maildirsize. This is with maildirquota, in perhaps a bit of an
exotic config. The idea is to set a specific quota to Spam folder, using the
trash plugin to keep it at a certain size.
. getquotaroot Spam
* QUOTAROOT Spam User quota Spam quota
* QUOTA User quota (STORAGE 40848 1228800)
* QUOTA Spam quota (STORAGE 0 2)
. OK Getquotaroot completed.
Once 1 email gets added:
.Spam# cat maildirsize
2048S
0 0
2017 1
namespace {
hidden = no
ignore_on_failure = no
inbox = yes
list = yes
location =
maildir:%h:INDEX=/var/spool/mail/dovecot-control/indexes/%1u/%2u/%u:CONTROL=/var/spool/mail/dovecot-control/%1u/%2u/%u/INBOX
prefix =
separator = /
subscriptions = yes
type = private
}
namespace spam {
hidden = yes
ignore_on_failure = no
inbox = no
list = yes
location =
maildir:%h/.Spam:INDEX=/var/spool/mail/dovecot-control/indexes/%1u/%2u/%u/.Spam:CONTROL=/var/spool/mail/dovecot-control/%1u/%2u/%u/.Spam
prefix = Spam/
separator = /
subscriptions = no
type = private
}
plugin {
quota = fs:User quota
quota2 = maildir:Spam quota:ns=Spam/
quota2_rule = *:storage=2K
sieve = /var/spool/mail/dovecot-control/sieve/%1u/%2u/%u/dovecot.sieve
sieve_before = /etc/sieve/before
sieve_dir = /var/spool/mail/dovecot-control/sieve/%1u/%2u/%u/scripts
trash = /etc/dovecot/conf.d/dovecot-trash.conf.ext
}
(full config: http://pastebin.com/Mui4X7Zh)
[Dovecot] multiple Return-Path headers added using lmtp director
Emails arrive with 2 Return-Paths when using lmtp director. Is this something configurable in the director or is this a bug? Return-Path: xxx...@xs4all.net Delivered-To: +Spam Received: from lmtpdirector1.xs4all.net ([194.109.26.176]) by lmtp2.xs4all.net (Dovecot) with LMTP id gOe2E3YSkE/aEAAAYH/+MA for x+Spam; Thu, 19 Apr 2012 15:26:14 +0200 Return-Path: xx...@xs4all.net Received: from mxdrop-builder.xs4all.nl ([194.109.23.200]) by lmtpdirector1.xs4all.net (Dovecot) with LMTP id CnOvKTkzjU9sJQAAbddgHA ; Thu, 19 Apr 2012 15:26:14 +0200 Regards
[Dovecot] trash plugin
The trash plugin docs say:
Normally if a message can't be saved/copied because it would bring user over
quota, the save/copy fails with Quota exceeded error. The trash plugin can be
used to avoid such situations by making Dovecot automatically expunge oldest
messages from configured mailboxes until the message can be saved
This leads me to believe that if a message cant be stored, the plugin would
avoid returning a quota error, but instead delete/expunge messages until the
original message can fit, and then complete the command.
But I see something different, at least using the plugin in imap (havent tried
it on lmtp). When I drag a message to a folder thats almost full, and the
message doesnt fit, I get an actual quota error:
23 uid copy 454343 Spam
23 NO [OVERQUOTA] Quota exceeded (mailbox for user is full)
After my client sees this error, I see the trash plugin creating space. But
then it's too late right? The client already has a visible error. Am I just
misunderstanding how the trash plugin is supposed to work?
Dovecot 2.1.4, config is on a set of test servers:
namespace {
type = private
separator = /
inbox = yes
}
namespace {
type = private
separator = /
inbox = no
prefix = Spam/
location =
maildir:/var/spool/mail/%1u/%1.1u/%u/.Spam:INDEX=/var/spool/mail/dovecot-control/indexes/%1u/%2u/%u/.Spam:CONTROL=/var/spool/mail/dovecot-control/%1u/%2u/%u/.Spam
subscriptions = no
list = yes
hidden = yes
}
plugin {
quota = maildir:Spam quota:ns=Spam/
quota_rule = *:storage=5000K
}
My goal is to create a FIFO spam folder that never returns a quota error, but
instead just deletes the oldest emails as new spam comes in.
Regards,
Cor
[Dovecot] returning a namespace location from userdb?
hey all, is it possible to return the location of namespace from the userdb
lookup? The code is a bit unclear about it. There seems to be a part of the
docs saying:
If you want to override settings inside sections, you can separate the section
name and key with '/'. For example:
namespace default {
inbox = yes
}
The inbox setting can be overridden by returning namespace/default/inbox=no
extra field
I realize you can set 'location=' in the namespace, but we have mailbox
locations that cant be expressed by the normal variable substitution vars in
dovecot (dont ask). So we have a userdb that returns the correct mail location.
This works fine. But im currently trying to implement a second namespace, and
Im having trouble returning a mailbox location for this specific namespace from
our userdb. Is this possible? If so, what would the userdb have to return
exactly?
namespace spam {
type = private
separator = /
inbox = no
prefix = Spam/
subscriptions = no
list = yes
hidden = yes
}
Regards,
Cor
Re: [Dovecot] Configuration advices for a 50000 mailboxes server(s)
Here we have approx. 200K users with 4000 concurrent connections (90% POP3 users) All servers in virtual environment Vmware, supermicro servers and Netapp Metrocluster storage solutions (nfs storage with 10G ethernet network) POP3 sessions take betwen 40 and 300 milisecons at connect, auth and list. All accounts lives in LDAP, CentOS 5 and exim like a mta relay. Similar setup here. Maybe 15-20K concurrent connections, imap only (pop is not handled by dovecot yet) about 800K mailboxes. We have all bare metal servers, linux. Currently 35 servers handling the load easily. We could run on 1/3rd of the load probably. In front of the 35 servers are 3 directors, handling imap only, although im in the process of adding LMTP/sieve to the mix. Backend storage is NetApp Metrocluster over 2 datacenters. Cor
[Dovecot] LMTP auth problem
hey all, im getting the following error:
Apr 14 14:29:44 lmtpdirector1 dovecot: auth: Error: passdb(scorpio,127.0.0.1):
Auth client doesn't have permissions to do a PASS lookup:
/var/run/dovecot/auth-userdb mode=0666, but not owned by UID 112(dovecot)
Apr 14 14:29:44 lmtpdirector1 dovecot: lmtp(18298): Error: user scorpio: Auth
PASS lookup failed
My config. Director servers running both imap and lmtp with a matching set of
real servers accepting imap/lmtp. Imap is working fine, and has been working
fine for a while. Im trying to add lmtp to the director, but i cant seem to get
that working. We're passing passdb on to the real servers. How does this work
with lmtp?
protocols = imap lmtp
protocol lmtp {
auth_socket_path = director-userdb
}
lmtp_proxy = yes
# passdb check on real servers
passdb {
driver = static
args = proxy=y nopassword=y
}
Cor
Re: [Dovecot] LMTP auth problem
Of course the moment I post I seem to have figured it out..
service auth {
unix_listener auth-userdb {
mode = 0777
}
}
Is this safe if your servers are secure?
Cor
Re: [Dovecot] LMTP auth problem
Apr 14 14:29:44 lmtpdirector1 dovecot: auth: Error:
passdb(scorpio,127.0.0.1): Auth client doesn't have permissions to do a
PASS lookup: /var/run/dovecot/auth-userdb mode=0666, but not owned by UID
112(dovecot)
Apr 14 14:29:44 lmtpdirector1 dovecot: lmtp(18298): Error: user scorpio:
Auth PASS lookup failed
I'd just try 'user = dovecot' rather than making it wide open because that's
what the log basically says.
$ doveconf -d | grep 'unix_listener auth-userdb' -A 4
unix_listener auth-userdb {
group =
mode = 0666
user =
}
My config was the same as yours. That didnt work for me. But if I add
user = dovecot
mode = 0666
That does work. Of course, the difference between 777 and 666 is
minimal. I think 666 is handled as a special case in the code?
Cor
Re: [Dovecot] Hints for a NFS-Setup
On Wed, Apr 04, 2012 at 02:02:02PM +0200, Patrick Westenberg wrote: Timo Sirainen schrieb: Another director. They're meant to connect to each others and do LB/HA. But what about my MTAs? How can I tell my two postfix servers that there are two directors and it should/can use the other one if the first is down? We use hardware loadbalancers in front of the directors. If a director goes down, the hardware loadbalancers sees that. We also use the poolmon script to tell the directors if a certain imap server is missing. Cor
Re: [Dovecot] crashes on 2.0.16
This also is not the kernel list, since updating to a kernel released in the 21st century Cor's issue has gone away, so this thread is now rather entirely pointless on the Dovecot list. So I'll my participation in Actually, it hasn't. For the last few days we've been trying to pinpoint the problem by running half a dozen servers with different kernels and options. This 'kernel released in the 20th century' we have this problem with is the current stable kernel for debian. So instead of starting a distro war (and I thought canon/nikon, mac/pc were bad), lets be a bit more constructive. Plenty of people use the stable debian release. It's happening on about half of our imap servers (20 or so), but on none of our other debian servers (hundreds). So it's not so weird to suspect a dovecot link. But it looks like it may be a driver issue on a specific range of hardware we use. Once we find the issue, i'll report back. Until then, this doesn't look like a dovecot problem. Cor
Re: [Dovecot] crashes on 2.0.16
I installed a newer kernel on these boxes, and it's fixed. Seems to be a problem with the stock debian squeeze kernel. Not a dovecot issue, but others with a stable squeeze box might see similar problems so good to have it in the archive :) regards, Cor
[Dovecot] crashes on 2.0.16
Hey all, I upgraded some servers today from Debian Lenny to Debian Squeeze, and
after the upgrade I started getting dovecot crashes. I was on 2.0.13 but got
these there as well, and hoped upgrading to 2.0.16 would help. It didn't.
Anyone have an idea?
Cor
Dec 18 23:32:21 userimap1 kernel: [263492.245530] __ratelimit: 122 callbacks
suppressed
Dec 18 23:32:21 userimap1 kernel: [263492.245536] imap[19665]: segfault at
7f108efb7f60 ip 7f108ecd37e0 sp 7fff73c3d658 error 5 in
libc-2.11.2.so[7f108ec5d000+158000]
Dec 18 23:32:21 userimap1 kernel: [263492.245663] BUG: Bad page map in process
imap pte:0001 pmd:09767067
Dec 18 23:32:21 userimap1 kernel: [263492.252563] page:ea00
flags:(null) count:-84 mapcount:-84 mapping:(null) index:0
Dec 18 23:32:21 userimap1 kernel: [263492.261186] addr:7f108ef46000
vm_flags:0870 anon_vma:(null) mapping:88012b6e5238 index:2e9
Dec 18 23:32:21 userimap1 kernel: [263492.270804] vma-vm_ops-fault:
filemap_fault+0x0/0x460
Dec 18 23:32:21 userimap1 kernel: [263492.276172] vma-vm_file-f_op-mmap:
generic_file_mmap+0x0/0x60
Dec 18 23:32:21 userimap1 kernel: [263492.282424] Pid: 19665, comm: imap
Tainted: GB 2.6.32.36-xsserver #1
Dec 18 23:32:21 userimap1 kernel: [263492.282429] Call Trace:
Dec 18 23:32:21 userimap1 kernel: [263492.282441] [81103ee5]
print_bad_pte+0x1d5/0x280
Dec 18 23:32:21 userimap1 kernel: [263492.282448] [8110568a]
unmap_vmas+0xa1a/0xa20
Dec 18 23:32:21 userimap1 kernel: [263492.282458] [8137e501] ?
sch_direct_xmit+0x81/0x1d0
Dec 18 23:32:21 userimap1 kernel: [263492.282465] [8110b60d]
exit_mmap+0xbd/0x190
Dec 18 23:32:21 userimap1 kernel: [263492.282473] [81059cbd]
mmput+0x4d/0x120
Dec 18 23:32:21 userimap1 kernel: [263492.282481] [8105f282]
exit_mm+0x112/0x150
Dec 18 23:32:21 userimap1 kernel: [263492.282488] [8106104d]
do_exit+0x13d/0x800
Dec 18 23:32:21 userimap1 kernel: [263492.282496] [81072f31] ?
__dequeue_signal+0xe1/0x200
Dec 18 23:32:21 userimap1 kernel: [263492.282503] [81061765]
do_group_exit+0x55/0xd0
Dec 18 23:32:21 userimap1 kernel: [263492.282510] [81075251]
get_signal_to_deliver+0x1e1/0x3e0
Dec 18 23:32:21 userimap1 kernel: [263492.282517] [8100b315]
do_signal+0x75/0x7d0
Dec 18 23:32:21 userimap1 kernel: [263492.282523] [81103203] ?
print_vma_addr+0xb3/0x120
Dec 18 23:32:21 userimap1 kernel: [263492.282533] [8143856e] ?
printk+0x41/0x43
Dec 18 23:32:21 userimap1 kernel: [263492.282541] [81036436] ?
__bad_area_nosemaphore+0xf6/0x1e0
Dec 18 23:32:21 userimap1 kernel: [263492.282547] [810365ee] ?
bad_area_access_error+0x4e/0x60
Dec 18 23:32:21 userimap1 kernel: [263492.282553] [8100bac7]
do_notify_resume+0x57/0x60
Dec 18 23:32:21 userimap1 kernel: [263492.282559] [8100cadc]
retint_signal+0x48/0x8c
# 2.0.16: /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 2.6.32.36-xsserver x86_64 Debian 6.0.3
auth_cache_size = 1 M
auth_verbose = yes
base_dir = /var/run/dovecot/
default_vsz_limit = 2 G
disable_plaintext_auth = no
first_valid_uid = 20
lock_method = dotlock
login_greeting = User-IMAP ready.
login_trusted_networks = 194.109.26.128/27
mail_fsync = always
mail_nfs_index = yes
mail_nfs_storage = yes
mailbox_idle_check_interval = 1 mins
maildir_stat_dirs = yes
mmap_disable = yes
namespace {
inbox = yes
location =
maildir:%Nu:INDEX=/var/spool/mail/dovecot-control/indexes/%1u/%2u/%u:INBOX=%Nu:CONTROL=/var/spool/mail/dovecot-control/%1u/%2u/%u/INBOX
prefix =
separator = /
type = private
}
passdb {
args = /usr/local/etc/dovecot/dovecot.masterusers
driver = passwd-file
master = yes
}
passdb {
args = cache_key=%u%r dovecot
driver = pam
}
plugin {
quota = fs:User quota
}
protocols = imap
service auth {
client_limit = 5000
}
service imap-login {
process_min_avail = 8
service_count = 0
vsz_limit = 2 G
}
service imap {
process_limit = 4096
vsz_limit = 2 G
}
shutdown_clients = no
ssl_cert = /etc/ssl/certs/xx
ssl_key = /etc/ssl/private/xx
userdb {
driver = passwd
}
verbose_proctitle = yes
protocol imap {
imap_capability = IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT
SORT=DISPLAY THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN
NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT
SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS
imap_client_workarounds = delay-newmail
mail_max_userip_connections = 24
mail_plugins = quota imap_quota
}
[Dovecot] OATH/OTP?
Hey all, has anyone ever successfully implemented some form of OTP system with dovecot? Im looking at setting up an OATH/HOTP-TOTP based OTP for our services, but the webmail service (which uses dovecot) is a difficult one. Any info on implementations would be appreciated, Regards, Cor
Re: [Dovecot] director monitoring?
We use a setup as seen on http://grab.by/agCb for about 30.000 simultaneous(!) imap connections. We have 2 Foundry loadbalancers. They check the health of the directors. We have 3 directors, and each one runs Brandon's poolmon script (https://github.com/brandond/poolmon). This script removes real servers out of the director pool. The dovecot imap servers are monitored with nagios just to tell us when they're down. This setup has been absolutely rock solid for us. I have not touched the whole system since november and we have not seen any more corruption of meta data, which is the whole reason for the directors. Kudos to Timo for fixing this difficult problem. Cor
[Dovecot] slow squat fts index creation
Hi all, ive been playing with squat indexes. Up to about 300.000 emails in a single mailbox this was working flawlessly. The search index file is about 500MB at that time. Ive now added some more emails, and at 450.000 or so emails im seeing a serious problem with squat index creation. It takes...f o r e v e r . The .tmp file is being so slowly, it will probably take 2-3 hours to create. Upto this point it took maybe a minute. Im doing this in an openvz container, so theoretically i may be hitting some openvz resource limit. But ive upped all the limits and dont see any improvements. I dont see any resources starvation either. Could there be some dovecot issue when the search index reaches say 1GB? (im estimating that it's now trying to save about 1GB search index). Regards, Cor
Re: [Dovecot] fts squat index file not appearing
On May 20, 2011, at 12:46 AM, Timo Sirainen wrote:
On 19.5.2011, at 18.40, Cor Bosman wrote:
Hey all, im experimenting with squat for a small project but am not having
much luck. Debugging tells me squat is being loaded, but the index.search
files are not appearing after TEXT/BODY commands. Squat plugin was added to
config as well.
plugin {
fts = squat
}
No, it was something wholly unexpected to me. OSX Mail refuses to do remote
searches. I expected OSX Mail to issue a BODY/TEXT when doing an fts, but it
only ever searches local copies. So I cant even set up a remote archive under
OSX Mail without first copying all 1 million emails to a local copy. How
stupid.
When I issues the commands by hand everything worked as expected.
Cor
[Dovecot] fts squat index file not appearing
Hey all, im experimenting with squat for a small project but am not having much luck. Debugging tells me squat is being loaded, but the index.search files are not appearing after TEXT/BODY commands. Squat plugin was added to config as well. Anyone have an idea? Cor
[Dovecot] restarting director
Hi all, anyone having any problems with restarting the director? Every time I bring down 1 of the director servers, reboot it, or just restart it for whatever reason, im seeing all kinds of problems. Dovecot generally always gives me this error: Jan 20 22:49:55 imapdirector3 dovecot: director: Error: Director 194.109.26.173:444/right disconnected before handshake finished It seems the directors cant agree on forming a ring anymore, and this may be leading to problems with clients. I mostly have to resort to bringing down all directors, and restarting them all at once. Not really a workable solution. As an example, last night for a few hours we were getting complaints from customers about being disconnected, and the only obvious error in the log was the one above, after one of my colleagues had to restart a director because of some changes in the syslog daemon. After I restarted all directors withing a few seconds of each other, all complaints disappeared. Timo, i know ive asked similar questions before, but the answer just eludes me. If I have 3 director servers, and need to take one down and restart it, what is the proper method to reconnect the ring? In practice, I cant seem to work it out and I mostly end up with the above error until I just restart them all. Not fun with 20.000 clients connected. Cor
Re: [Dovecot] SSD drives are really fast running Dovecot
This discussion has been in the context of _storing_ user email. The assumption is that an OP is smart/talented enough to get his spam filters/appliances killing 99% before it reaches intermediate storage or mailboxes. Thus, in the context of this discussion, the average size of a spam message is irrelevant, because we're talking about what goes into the mail store. The fact is, we all live in different realities, so we're all arguing about apples and oranges. If you're managing a SOHO, small company, large company, university, or in our case, an ISP, the requirements are all different. We have about a million mailboxes, about 20K active at the same time, and people pay for it. Take for example Stan's spam quote above. In the real world of an ISP, killing 99% of all spam before it hits the storage is unthinkable. We only block spam that is guaranteed to be unwanted, mostly based on technical facts that can't ever happen in normal email. But email that our scanning system flags as probable spam, is just that, probable spam. We can not just throw that away, because in the real world, there are always, and I mean always, false positives. It is unthinkable to throw false positives away. So we have to put these emails in a spam folder in case the user wants to look at it. We block about 40% of all spam on technical grounds, our total spam percentage is 90%, so still about 80% of all customer email reaching the storage is spam. But in other environments throwing away all probable spam may be perfectly fine. For my SOHO id have no problem throwing probable spam away. I never look in my spam folder anyways, so cant be missing much. The same goes for SSD. We use SSD drives extensively in our company. Currently mostly in database servers, but our experiences have been good enough that we're slowly starting to add them to more systems as even boot drives. But we're not using them yet in email storage. Like Brad we're using Netapp filers because as far as I know they're one of the few commercially available HA filesystem companies. We've looked at EMC and Sun as well, but havent found a reason to move away from Netapp. In 12 years of netapp we've only had 1 major outage that lasted half a day (and made the front page of national news papers). So, understand that bit. Major outages make it to national news papers for us. HA, failover, etc are kind of important to us. So why not build something ourselves and use SSD? I suppose we could, but it's not as easy as it sounds for us. (your mileage may vary). It would take significant amounts of engineering time, testing, migrating, etc etc. And the benefits are uncertain. We dont know if an open source HA alternative can give us another 12 years of virtually faultless operation. It may. It may not. Email is not something to start gambling with. People get kind of upset when their email disappears. We know what we've got with Netapp. I did dabble in using SSD for indexes for a while, and it looked very promising. Certainly indexes are a prime target for SSD drives. But when the director matured, we started using the director and the netapp for indexes again. I may still build my own NFS server and use SSD drives just for indexes, simply to offload IOPS from the Netapp. Indexes are a little less scary to experiment with. So, if you're in the position to try out SSD drives for indexes or even for storage, go for it. Im sure it will perform much better than spinning drives. Cor
Re: [Dovecot] SSD drives are really fast running Dovecot
Btw, our average mailsize last we checked was 30KB. Thats a pretty good average as we're an ISP with a very wide user base. I think 4KB average is not a normal mail load. Cor
Re: [Dovecot] director in large(ish) environments
On Dec 28, 2010, at 3:55 AM, Henrique Fernandes wrote: Can i ask how are you storing your mail ? like, NFS, gfs, ocfs2 etc and with type, like mbox, maildir, sdbox etc.. In my system we are not usign director, using IPVS but having lots of IO wait problems! We're using NFS/maildir. Cor
Re: [Dovecot] Iphone all imap folders are visibles !
When I use my iphone to read my emails through IMAP I can see ALL folders that are in my email home directory not only those are listed in the .subscriptions file ... This should be a real security problem Anyone have the same problem ? This is not something dovecot can change. Apple has chosen to ignore subscription files, not just in the iphone, but in their desktop client as well. Why is it a security problem? They're all your folders right? Cor
[Dovecot] director in large(ish) environments
Hey all, just wondering who here is running the director service in a larger environment. I just switched our whole production setup to the director and am quite pleased with the result. We're doing a peak of about 25000 tot 3 concurrent sessions on 3 servers. But ive shut 1 server down a couple of days ago to see what would happen and 2 servers carried the load easily. (16 CPU, 24GB memory servers). If others are using the director on larger setups maybe we can all post when things do or dont work well. Now to see if the solution is better than the problem :) Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Oops, now I finally understand why Mail.app kept asking for my password for each mail I sent: it helpfully decided to start signing mails with the only client cert I had, without asking me.. Forget about those signatures in the last two mails :) Heh, is that the key you used to get into our test server for this problem? :) Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
It looks like Timo's patch fixes the problem. Context switches are now back to normal, and the load graph is smooth and lower. I saw someone also posted a patch to the LKML. Cor
Re: [Dovecot] adding servers to director_mail_servers
Timo, sorry to get back to this, but im still confused :) It seems every time I want to add a director, or other need to restart one or work on its config, im having a hard time making things work again without service disruption (or minimal service disruption). Lets say I have the following situation: director1 director2 director3 They form a ring. Now I shut down director2. Then director1 and 3 connect to each other to fix the ring right? Now I restart director2. What happens? Do I have to restart director1 and 3 as well? And if so, what order do I do this? And another situation. Lets say I have: director1 director2 director3 Now I want to add director4. So I change the config, add a new director server to the config. I put this new config on all director servers. Now what order should I start and/or restart everything? Regards, Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
For those interested, graph showing the difference before and after patch: context switches: http://grab.by/7W9u load:http://grab.by/7W9x Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Correct. Only Linux is affected. Anyway, Postfix's design has been like this forever, so no one would have ever noticed context switch counts increasing. But changing this might make them notice that context switches are dropping. Im a little surprised we havent seen more reports on dovecot as well. The impact was noticeable for us. Thanks for looking into the issue! Cor
[Dovecot] director errors
Hi Timo, im getting some director errors. Dec 13 10:11:25 imapdirector1 dovecot: director: Error: User hash 3608614717 is being redirected to two hosts: 194.109.26.171 and 194.109.26.141 Dec 13 10:11:25 imapdirector1 dovecot: director: Error: User hash 755231546 is being redirected to two hosts: 194.109.26.132 and 194.109.26.154 Dec 13 10:11:25 imapdirector1 dovecot: director: Error: User hash 2468924024 is being redirected to two hosts: 194.109.26.135 and 194.109.26.159 Dec 13 10:11:25 imapdirector1 dovecot: director: Error: User hash 3275820100 is being redirected to two hosts: 194.109.26.134 and 194.109.26.160 Dec 13 10:11:25 imapdirector1 dovecot: director: Error: User hash 1674197395 is being redirected to two hosts: 194.109.26.137 and 194.109.26.159 Dec 13 10:11:25 imapdirector1 dovecot: director: Error: User hash 1753370669 is being redirected to two hosts: 194.109.26.134 and 194.109.26.171 Dec 13 10:11:25 imapdirector1 dovecot: director: Error: User hash 3760447994 is being redirected to two hosts: 194.109.26.131 and 194.109.26.141 Dec 13 10:11:25 imapdirector1 dovecot: director: Error: User hash 3687955574 is being redirected to two hosts: 194.109.26.171 and 194.109.26.158 Dec 13 10:11:26 imapdirector1 dovecot: director: Error: director(194.109.26.174:444/right): Handshaking timed out Dec 13 10:11:26 imapdirector1 dovecot: director: Error: Director 194.109.26.174:444/right disconnected Ive got 37 director_mail_servers and 2 director_servers. Cor
[Dovecot] director_user_expire
Would it be ok to set director_user_expire to a very high value? Say 1 day? This would pretty much lock a user into a specific server right? What if I set it to 1 year? :) What happens when the server a user is locked into disappears (i remove it from the pool). Then a day later, I add the server again. Is the user now locked to the new server, or will he revert back to the old server? Cor
[Dovecot] adding servers to director_mail_servers
Is there a specific order in which one has to add servers to director_mail_servers, Let's say I have 20 loadbalanced director servers (I dont, but for arguments sake). I want to add 5 servers to director_mail_servers. If I do this by restarting each director server 1 by 1 with an updated conf, it seems the 5 new servers dont get added. Or at least I dont get what makes it work eventually. Only after restarting a few times they seem to all want to agree on the new list of servers. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Using gcc: gcc version 4.4.5 (Debian 4.4.5-8) We run gcc version 4.3.2 Im not using any configure options, except for a few that Timo had me try during debugging. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Are the process pids also logged in the messages, so it's clear which
messages belong to which imap process? If not, add %p to mail_log_prefix.
Done. It wasnt logging this, now it is.
As I said previously, im not longer running the imap server=0 patch because
it caused these errors:
Dec 9 06:54:32 userimap30 dovecot: imap(cor): Error: user cor:
Initialization failed: Namespace '':
mkdir(/var/spool/mail/.4a/L/0/14/corbosch) failed: Permission denied
(euid=1000(cor) egid=50(staff) missing +w perm: /var/spool/mail/.4a/L/0/14,
euid is not dir owner).
After I removed the patch these errors disappeared.
The patch was supposed to fix this.. Is uid 1000 wrong for this user? Didn't
you already exist, why is it complaining about mkdir()?
It's very strange. Im user 'cor'. Maybe it's logging my name because I started
the process (su-d to root obviously)? My uid is 1000. But im not the one
doing this login, these are customer logins. The mkdir fails because the target
dir is only writable by root. Normally dovecot creates a directory as root, and
chowns it to the owner it belongs to. At least, thats what I assume happens as
normally this all works fine.
It's like dovecot was unable to change the process to euid root but instead is
doing everything as euid cor.
Just for thoroughness ive started 2 servers with the logging patch. One with
service_count=0 and one with service_count=1
What, are you saying that without the patch service imap { service_count=0 }
works? Or are you talking about imap-login?
Im talking about imap-login. Currently im not running any servers with imap {
service_count=0 } due to the permission errors. Im more than happy to give it
another go if you feel i did something wrong.
Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Great. If the logs aren't too huge I could look at the raw ones, or you could try to write a script yourself to parse them. I'm basically interested in things like: 1. How large are the first volcs entries for processes? (= Is the initial process setup cost high?) 2. Find the last volcs entry for each process. Sum them up for some time period. Is the total about the same as the system's entire volcs during that time? 3. What kind of a volcs distribution do the processes have? A graph might be nice :) 4. Most importantly: Are there some commands that trigger a high increase in volcs numbers? If you want to have a quick look already, im mailing you the locations of 2 files, 1 with service_count=0 and one with service_count=1. Well, that clearly says it's trying to initialize user cor.. Is the corbosch directory you or someone else? corbosch is actually a non-existing user, and im suspecting a local bug. I mailed you about that in private. I'd have guessed that the above corbosch already existed. Yeah, it probably should be targetting 'cor' but for some reason chars get appended to the dir location. Not sure if its our local modification or something in dovecot. Normally dovecot creates a directory as root, and chowns it to the owner it belongs to. At least, thats what I assume happens as normally this all works fine. Uh. No.. Privileges are always dropped before Dovecot creates any directories. I don't know what creates the directories for you. I'd guess the authentication code? Something in dovecot does :) If a user without a maildir logs in for the first time, the dir is created. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
On Dec 9, 2010, at 10:41 AM, Timo Sirainen wrote: On 9.12.2010, at 9.13, Cor Bosman wrote: If you want to have a quick look already, im mailing you the locations of 2 files, 1 with service_count=0 and one with service_count=1. I see that about half the commands that do hundreds or thousands of volcses are IDLE. Wonder if that is the problem. But there are others .. I see for example that one LOGOUT command takes 2000 volcses, which I doubt is true. More likely it's counting something that happened before the LOGOUT, like probably doing a mailbox synchronization. Attached a new version of the patch that also logs volcs for syncs. Also STATUS and SELECT seem to have high volcses, which do a mailbox sync.. Hmm. So if it is about mailbox sync, where in the sync then is it slow? Added more debugging for this in the attached patch. Ok, i'll apply this patch to both running servers doing logging. Something in dovecot does :) If a user without a maildir logs in for the first time, the dir is created. Probably would be good to find out what exactly, in case that's caused by some security hole :) Sorry, my mistake. It's not dovecot, it's our LDA (not dovecot) thats making the dir (which means we now get an error if a user logs in and they have never had email.. but that never happens since we mail them on signup..interesting little problem :) Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Some preliminary findings..
Changing the kernel seems to have a positive effect on the load. I changed from
2.6.27.46 to 2.6.27.54 (sorry, im bound by locally available kernels due to a
kernel patch we created to fix some NFS problems in the linux kernel. Patch
should be available in the stock linux kernel as of 2.6.36, but we dont have
that kernel locally available yet), At similar user levels the load is now back
to what it was. It doesnt however influence the context switches. Still very
high. Here's a graph of a server where I changed the kernel and nothing else:
load: http://grab.by/7Odt (i rebooted the server with a new kernel at the
point where the high load drops to 0)
cs: http://grab.by/7Odw
Here's a server with the old kernel, but imap-login { service_count=0 }. It
seems this doesnt have much impact. cs are lower (but higher than 1.2),
probably because user levels havent gone up to normal levels yet due to
loadbalancer distribution issues after a reboot)
load: http://grab.by/7OdL
cs: http://grab.by/7OdN
Ive now also set up a server with new kernel and service_count = 0, just to
cover all the bases. But not enough data on that server yet. It takes a while
for user levels to creep up once a server gets rebooted as users are sticky to
a server once they're linked to it.
Cor
[Dovecot] load increase after upgrade to 2.0.8
I upgraded most of our servers from 1.2.x to 2.0.8 and am noticing a really big increase in server load. This is across the board, not any specific server. Check this screenshot of a load graph: http://grab.by/7N8V Is there anything i should be looking at that could cause such a drastic load increase? Conf file is available here: http://wa.ter.net/download/doveconf.txt Regards, Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Here's the doveconf -n output: http://wa.ter.net/download/doveconf-n.txt Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
It could be that you both are running a different Kernel from the Standard Lenny Kernel 2.6.26. (this could be a clue ..) It would be interesting to hear from people that aren't seeing a big load increase. My initial guess was some kind of NFS problem, but since Ralf isn't doing NFS, that's probably not it. Our CPUs are actually pretty much idle (these are 8 and 16 core machines). And memory is plentiful. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
On Dec 8, 2010, at 5:11 PM, David Ford wrote: what's your task switch HZ compiled at? CONFIG_HZ_1000? you would probably be better at 300 or 250. have you tried tickless? is your kernel compiled for precisely your cpu type and smp/hyper options set correctly? what about CONFIG_PREEMPT? definitely don't use realtime, server is appropriate. You're probably asking Ralf, but I might as well answer as well.. CONFIG_HZ_250=y CONFIG_PREEMPT_NONE=y Not using realtime, all cpu/smp options look fine. CPU is usually pretty idle. Hovering around 90-99% idle. But the load graphs clearly show a radical increase, where before things were pretty smooth, now they're erratic and much higher. Cor ps: we use linux, not freebsd.
Re: [Dovecot] load increase after upgrade to 2.0.8
Timo and I found excessive numbers of context switches, factor 20-30. But it's unclear why the IMAP process would do/cause this. Im seeing this as well...http://grab.by/7Nni Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
I see you're using userdb passwd. Do your users have unique UIDs? If they have, maybe it has to do with that.. Yes, we have about 1 million unique UIDs in the passwd file (actually NIS). I did upgrade 4 machines to the latest kernel, but it's hard to tell if that changed much as our usage peak is during the day, So I'll know more tomorrow. But so far it looks as though the context switches are still high. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
So you have tons of imap-login processes. Did you have that in v1.2 too? Try setting service_count=0 (and same for pop3-login) http://wiki2.dovecot.org/LoginProcess Yes, we have tons of imap-login processes. I'll set service_count=0 for imap (we dont do pop with dovecot) on a few servers. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Oh, and I dont know if we did in 1.2. I think so, but cant be positive. I tried making the config the same. I have the config still around if you want to see it. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
login_process_per_connection = yes So seems like we did have that set in 1.2. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
We're running on bare metal, no VM involved. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Hope im doing what you want :) Getting kinda confusing. Right now I have running: 3 servers with just a new kernel, no other changes 2 servers with service_count = 0, no other changes 1 server with service_count = 0, and src/imap/main.c patch Is that ok? I wont be seeing much impact until tomorrow morning, not many users on at night. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
1 server with service_count = 0, and src/imap/main.c patch By this you mean service_count=0 for both service imap-login and service imap blocks, right? Yes, on both imap-login and imap, The 2 servers without the patch only have it on imap-login. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
1 server with service_count = 0, and src/imap/main.c patch Is that ok? Looks good! I had to revert this patch because it's causes permission errors on our filesystem. Directories are being created for user X with the euid of user Y (which fails, so at least i didnt get corruption, just that service failed on that server). Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
lcs values for v1.2 and v2.0. Wonder if you get different values? If you don't mind some huge logs, you could also try the attached patch that logs the voluntary context switch count for every executed IMAP command. Maybe some command shows up that generates them much more than others. diff I have applied this patch to 1 server, and it's logging away. Let me know what kind of output you'd like to see. Or i can send you the raw logfile. As I said previously, im not longer running the imap server=0 patch because it caused these errors: Dec 9 06:54:32 userimap30 dovecot: imap(cor): Error: user cor: Initialization failed: Namespace '': mkdir(/var/spool/mail/.4a/L/0/14/corbosch) failed: Permission denied (euid=1000(cor) egid=50(staff) missing +w perm: /var/spool/mail/.4a/L/0/14, euid is not dir owner). After I removed the patch these errors disappeared. Cor
Re: [Dovecot] load increase after upgrade to 2.0.8
Just for thoroughness ive started 2 servers with the logging patch. One with service_count=0 and one with service_count=1
Re: [Dovecot] Fwd: NFS question
On 2010-09-29 5:52 PM, Timo Sirainen wrote: Anyway, I can't give you any guarantees. I only know that it's not 100% safe, but some people with POP3-only/mostly setups have been happy enough anyway. Timo, can you at least clarify this - my understanding is that the problem here is not inherent to dovecot, but to NFS caching, and that dovecot is no more or less prone to having problems than any other imap/pop3 server? That is correct. Except that many other imap/pop servers do not implement a message index to increase performance. The problem lies in the index, not in the messages themselves. You can turn off the index completely, and then you'll be just as slow as any other imap server and wont ever see this problem. Or you can turn off index updating in the LDA if you use the LDA, and that prevents a big part of the issue as well. Or you implement the dovecot director. Cor
Re: [Dovecot] nfs director
Hi Noel, I do take exception to be told this issue can be fixed, but NFS users are not worth it, which is essentially what he told us, I dare say if I guess he told you this in private? The way i understand it the index-over-nfs problem can not be fixed 100%. I know for a fact Timo tried very hard to fix it, but NFS is what it is. And even if you can fix it in one OS, say Linux, you'd have a dozen other OSs to cope with. It is not only a dovecot problem, it is actually more of an NFS protocol problem. installs :) Some NFS users here are of small size where they can see beneifits of using another program (director) to manage things, and dont care about it impacting their inbound servers, some do care, but yes when you get to large user numbers, well, we are engineers, not PR/Sales ppl, so we can by our nature see through the upselling of using it. We dont use the director on our inbound servers either. We dont even use dovecot-lda. In our experience this isnt really a problem. The imap processes keep the indexes working well enough. So to answer Edward's original question, No, i dont think you need to replace your Foundries. xs-for-all is not a welcome domain here for spam/idiots in the past it We have over 300,000 DSL users. There's always idiots. But we're very active in the anti-spam community. We even run multiple public mirrors of all kinds of blacklists. If you send us a complaint, it will get acted on. Scoring any largish ISP +7 in SA because you got some spam in the past is kinda silly :) I cant imagine blocking a whole ISP because you got spam from them. Wow. Our users would revolt. Im not selling the director at all. Quite the contrary. I think most people wont need it. And I wish we didnt need it. It adds complexity, and that sucks. I hate it :) I think we actually agree there! :) You're right, turning off indexing in the LDA has no impact whatsoever on the delivery side. (again, we also dont use it there). But it does have a slight impact on the frontend side. It means the imap servers need to update the indexes every time they're out of date (which is basically every time new email arrives). For us, this impact doesnt seem to be a problem. Not having indexes does impact imap at a noticable level, especially for large mailboxes. Ive seen 10+ second delays on large mailboxes while the imap server is sorting the mailbox without the index. Basically, you're damned if you do, and damned if you dont. Cor
Re: [Dovecot] minor director issues
Oops, not CRC32, but a 32bit hash based on MD5 of the username. Still, it's 32bit so with lots of users there will be a lot of collisions (which shouldn't be a problem, except doveadm director map will list users who haven't really logged in). Isn't it possible to keep a copy of the username with the md5 checksum? Or is that problematic because of the collisions? Cor
Re: [Dovecot] nfs director
For those that are worried about the md5 hash sending out most users to 1 server and thus loadbalancing badly. This doesnt seem to be the case. I just pointed one of our test webmail environments to a director cluster (2 director servers behind a foundry, pointing to 2 real imap servers), and this is what the load spread was after a few minutes: imapdirector1:/usr/home/cor# doveadm director status mail server ip vhosts users 194.109.26.171 100 698 194.109.26.172 100 701 It doesnt get any better. It looks like Timo's choice of hash is spot on. Regards, Cor
[Dovecot] minor director issues
I noticed 2 minor things with the director. 1) if I start 2 directors, but in 1 of them I dont add the other as a director_servers, they both start spamming the logfiles really really fast. It seems there is no backoff mechanism in the protocol between directors. 2) usermail server ip expire time cor 194.109.26.171 2010-09-02 14:32:13 unknown 194.109.26.172 2010-09-02 14:33:13 Why does it say 'unknown'? That session is active, im manually logged in with telnet, issued a login and a select inbox. The first one is a normal mail client login. Regards, Cor
Re: [Dovecot] nfs director
Hi, We use Dovecot only recent, but many I speak use for very many year, if director was really need, why it only come about now and not 5 or more year ago, all many mail network run broken for so many year? I no think so. It might compliment some situation, but not suitable or advantage for all. Imap not likely happen here for user, hardware no so cheap here,such is storage space not big deal, not until we buy hardware for near cheap as U.S., which 50-70% less than we pay. The director is not needed in most cases. If you feel you dont need the director, then just forget about its existence and ignore all emails about it. Just pretend it doesnt exist :) I'll try to explain why the director is needed in some situations. dovecot has an index file. If you look in your configuration file, you'll see something like: mail_location = maildir:%%h/Maildir:INDEX=~/Maildir/shared/%%u The index file holds an index so dovecot can search in your mailbox really fast. (sorry if you know all this). So, if for instance you want to sort your mailbox on subject, dovecot doesnt have to read through all your emails to find all the subjects. Instead, it can read the index file really fast, and return a sorted mailbox basically instantly. A lot of IMAP servers dont even have this feature! Imagine the alternative, where dovecout would have to read through 25.000 emails, just so it can sort your mailbox on subject. A nightmare for performance, but like I said, some imap servers still do that! Dovecot luckily uses an index, and is therefore faster than a lot of other imap servers. So, we have concluded that dovecot has a really cool performance feature in the form of an index file. Now imagine what happens if you have more than 1 imap server. We have 35 imap servers right now. Each server needs access to the index file. One popular way to do this is by using NFS. With NFS you have a central file server, and each imap server uses the remote fileserver for storage. So now, all 35 servers have access to a cool central index file. But, there is a slight snag. NFS has a problem where caching of the files can cause each imap server to see outdated information. Server 1 could have just written some updates into the index file, but Server 2 doesnt see this yet, and writes conflicting data into the index file. The index file can then end up being corrupted. This causes dovecot to give faulty data back to the client, which in practice ends up looking like the mailbox of the user is empty. Not good. This causes calls to support desks about why a user has just lost all his mail. It's not lost, it's just that the index file is corrupt, and now dovecot thinks the mail is gone. You may wonder rightly so, why this is a problem only now. Well, it hasnt been a problem only now. We've been dealing with this issue for years. We've had discussions with Timo over the years on how to solve this problem. But Timo has only so much time, and this problem was not important enough to fix. He had more important bugs to fix. So, we lived with the problem, or worked around it in ugly ways. Recently Timo has found the time (I believe an ISP paid Timo to solve this problem) to think about ways to solve it. It's really not an easy problem to fix. We've known for years that the best way to fix this would be to have a user always end up on the same server, because thats when that specific NFS problem doesnt occur. If 2 sessions of the same user both end up on Server 1, then both sessions will see the same data, and no problems can occur. But how to get all sessions of a user on the same server? Most loadbalancers dont support this feature. Our Foundries dont, and our previous Alteons didnt either. So Timo thought of a way how dovecot itself could make sure all sessions of a user end up on the same server. Basically by turning dovecot into a proxy, that proxies connections to another server. It maintains an internal state on what user belongs on what server, and redirects connections appropriately. And thats all the director is. A proxy with an internal state of where to proxy a connection to. So why dont you see this problem with your setup? Like I said, this is only a problem with specific setups. - you need to be using NFS for central index storage. If you dont use NFS for your indexes, you wont see this problem, and you wont need the director. carry on as you always have. - even if you use NFS, you may not run into this problem. Because you also need to have situations where multiple servers could be updating the indexes at the same time. There are only 2 situations where this could happen: 1) you're using dovecot-lda, and have index updating turned on. Index updating in the lda is cool, because it means dovecot will always have the latest information about your email in the index files. and/or 2) you allow users to have sessions to multiple servers. For instance, they connect with Outlook and
Re: [Dovecot] minor director issues
Director doesn't know what users are logged in, only CRC32 of their username. So doveadm first gets a list of all users, adds them to hash table, then gets the list of CRC32s of users from director and maps them to usernames via the hash table. unknown means that the user wasn't found from the hash table. So .. my guess is that doveadm didn't get a full list of usernames. And you probably don't even want it to read those 2 million (or whatever) user records from SQL? :) Ah ofcourse! cor is in the local password file, while the other login is in the NIS database. That explains it. And we dont allow processes to ask for the whole NIS table :) Thanks, Cor
Re: [Dovecot] nfs director
Hi Noel, if you dont need the director, then thats great right? Why does anyone need to justify anything? Just dont use it, end of discussion. Those of us that do have a need for it, can use it anyways. Even without your agreement? Is that such a big problem? we have a some total of 2 imap servers, only used for webmail, so it is not a problem for us, at all, period. Awesome. imap access is not common in this country, it is in fact extremely rare, common only for webmail servers. It's common here. We have tens of thousands of direct imap connections, and they're often from the same user. Seeing 3 or more connections from the same user on different remote IP numbers is not uncommon. not an issue as I previously stated (twice I believe) as we do not permit multi session logins Not for you perhaps, but it is for us, and it is for others. So im glad you dont have this issue, we do, and the director solves it. No need to bash the director service just because you dont like it. dovecot LDA, which also wants to update the index files). Because of issues funny how postfix doesnt have this issue, qmail doesnt have this issue nor does exim (never ran sendmail in this config only a crazy person would use mbox over NFS) We dont use mbox. We use maildir. I agree only a crazy person would use mbox in a large scale environment. Postfix has this issue as well. So does qmail. So does exim. It has nothing to do with the software being used. It is a problem in the NFS protocol. (it's not a locking issue as I mentioned, but a caching issue as timo corrected.. i knew that, but in my haste to respond had locking in my mind. same principle applies. Multiple servers accessing the indexes can cause corruption due to inherent problems in the NFS protocol). The reason you dont see this, is because you dont allow simultaneous logins and dont offer direct imap access. Lets have this discussion again once you offer that :) there certainly is a workaround, its called, in our case, use postfix's deliver, which under my instruction has already happened on half the servers since it is clear this issue will not ever be solved in dovecot. Good for you. Im happy it works for you. yes, this has been mentioned some 40 times already, you know, i do actually know this, however that will NOT solve the problem as outlined earlier, the director by Timos admission will NOT load balance users in its current state, nor will it load balance inbound connections. Huh? Guess I dont understand what you outlined earlier. The director does load balance users. It uses an MD5 hash to do this, which in theory could cause uneven loadbalancing, but I know for a fact that in practice it poses no problem as we've been using an MD5 hash to loadbalance our webmail frontend to our imap servers for years. Thats 20.000+ simultaneous users and they get very evenly spread. Maybe I just dont understand what your issues with other people using the director are. If you have a solid system without the director, more power to you. Thats great. Congrats. Regards, Cor
Re: [Dovecot] nfs director
Hi, Postfix has this issue as well. So does qmail. So does exim. It has nothing to do with the software being used. It is a problem in the NFS protocol. Just to be clear. Of course these programs wont have this issue when used with dovecot-imap, because obviously they wont be updating any dovecot index files. You can tell dovecot-lda to not update index files, and you'd have exactly the same behavior. Combine that with not allowing multiple logins, and indeed, you wont ever need the director. What I meant was, if qmail, or exim, or postfix, would use some kind of indexing like dovecot, and would allow that over NFS, they'd have the same underlying issue to deal with. It's actually quite amazing and cool that Timo has addressed this problem for those that are using dovecot-lda with indexing over NFS, and/or dovecot-imap with multiple logins by the same user over NFS. Cor
[Dovecot] mixing 1.2 and 2.0?
Hi all, anyone know if it's safe to mix a 1.2 environment with 2.0 servers? Im planning on adding some 2.0 servers for test purposes, but now im wondering if thats going to mess up index files or other files for users selecting the test server and then switching back to our normal servers. Regards, Cor
Re: [Dovecot] nfs director
Hi, If you don't mind random Dovecot errors about index corruption I guess you're fine with how it works now. I guess your mails are delivered to maildirs by qmail? If you ever switch to Dovecot LDA you'll probably start getting more errors. And if you ever plan to switch to dbox format then at latest you'll need director. Ah! This probably explains why we're not being hit as hard as some others. We dont use dovecot lda. So we only see problems when users have 2 or more clients open and happen to hit the exact same polling times. We do have plans to move to dovecot lda so good to know our problems would have increased a lot. Should start testing with the director this weekend. Got 4 servers to play with. Cor
Re: [Dovecot] nfs director
We might be a slightly larger install than you (60k users, mail on FAS 3170 Metrocluster), but we have noticed corruption issues and the director is definitely going to see use in our shop. We still use Sendmail+procmail for delivery, so no issue there... but we've got hordes of IMAP users that will leave a client running at home, at their desk, on their phone, and then will use Webmail on their laptop. Without the director, all of these sessions end up on different backend mailservers, and it's basically a crapshoot which Dovecot instance notices a new message first. NFS locking being what it is, odds are an index will get corrupted sooner or later, and when this happens the user's mail 'disappears' until Dovecot can reindex it. The users inevitably freak out and call the helpdesk, who tells them to close and reopen their mail client. Maybe you're small enough to not run into problems, or maybe your users just have lower expectations or a higher pain threshold than ours. Either way, it's unpleasant for everyone involved, and quite easy to solve with the director proxy. We are in the exact same position as Brad. We also use sendmail's LDA, we also use a metrocluster, and we also have hordes of imap and webmail users. We see the exact same thing Brad sees. And I see it myself about once a week as well. The index gets corrupted due to access by 2 different clients, and to the user it then looks like their mail disappears. The user totally freaks out, because they'll invariably have really really important mail that has to be recovered right now. Usually a law firm as well. They call the helpdesk, keeping a support person busy with something thats really just a known bug. It probably isnt much of an issue if you use POP. But in large scale IMAP setups, where people are getting used to having access to all their email server-side (and thus mailboxes growing, needing larger indexes, increasing the chances of problems) from a myriad of clients this WILL happen if you're using NFS. Ive even considered moving away from NFS again for indexes due to this problem. But it really is noticable if you have a lot of email that your index isnt up to date as you move across our dozens and dozens of imap servers. Cor
Re: [Dovecot] nfs director
Noel, I think you just dont quite understand the problem the director is solving. The issue is that NFS is not lock-safe over multiple servers. We have 35 imap servers accessing a central NFS cluster. (we have over a million mailboxes) We offer IMAP to end user clients, and through webmail. This means that users are more and more likely to have multiple mail clients open. 1) they have a mail client open at home, lets say Thunderbird or OSX Mail. When they go to work they leave it on, so the software keeps looking for mail. 2) At work, they open webmail, so they can access their private email through their companies firewall. They leave this webmail session open in a browser tab. 3) They also have an iphone, and it's continuously checking their mail as well. This is not farfetched. This is normal behavior when you start offering IMAP access. We of course have a hardware loadbalancer (foundry) that directs incoming connections. But this loadbalancer does not know the 3 connections above are from the same user. So each gets directed to a different imap server. All these 3 sessions want to update the index files. (im current not including dovecot LDA, which also wants to update the index files). Because of issues inherent in NFS, the 3 imap servers that handle these connections may all think they have an exclusive lock. They may end up writing to the index files at the same time because of this. So the end result is a corrupt index file, which causes problems in the clients. The chances of this happening get higher as you have more users and more servers. Now there is a workaround. NFS locking can be made to work better if all processes trying to access the NFS indexes are on the same machine. So if all 3 clients in the above example happen to end up on the same imap server, there wouldnt be a problem. That imap server can safely say 'sorry, you cant write to the index file right now, another process is already writing to it'. What the dovecot director is doing is ensuring that sessions from the same user all get directed to the same imap server, so NFS locking works safely. I wish my foundry could do this, so I wouldnt need the director, but alas, it cant. If you operate a situation like im describing above, you WILL need the director, or accept that your users may run into problems reading their mail. If you dont see this problem, you either are not running the same situation im describing, or you do have this problem but just dont know :) I am very happy that Timo implemented this, so those of us that run this setup and are experiencing this issue, have a way to work around it. Regards, Cor
[Dovecot] read-only dovecot-uidlist
Hi all, i just investigated a user complaint and found a read-only dovecot-uidlist. Since dovecot couldnt write it, the process failed. Users can not reach this file, so how this became readonly is beyond me. Must be something in dovecot. Maybe an older bug? Doing a search now for more readonly uidlist files, but that'll take a few hours. Cor
Re: [Dovecot] read-only dovecot-uidlist
On Jan 26, 2010, at 12:29 PM, Timo Sirainen wrote: On Tue, 2010-01-26 at 10:24 -0400, Cor Bosman wrote: Hi all, i just investigated a user complaint and found a read-only dovecot-uidlist. Since dovecot couldnt write it, the process failed. Users can not reach this file, so how this became readonly is beyond me. Must be something in dovecot. Maybe an older bug? What were the file's permissions (owner, group, mode)? I think it's unlikely it is/was a Dovecot bug.. The user was the owner, permissions were 0400, group was our normal group. Everything was the same as others except for the mode, 0400 instead of 0600. We have nothing operating on these files (well, except for a deleted user wipe script), so i dont get what it could be if it isnt dovecot. NFS bug? Could be I suppose. Cor
Re: [Dovecot] quick question
On Jan 22, 2010, at 1:19 PM, Timo Sirainen wrote: On Fri, 2010-01-22 at 19:16 +0200, Timo Sirainen wrote: 2) Long term solution will be for Dovecot to not use NFS server for inter-process communication, but instead connect to other Dovecot servers directly via network. Actually not NFS server, but filesystem. So this would be done even when not using NFS. Is this the situation we discussed once where a dovecot instance becomes a proxy if it detects that a user should be on a different server? The one thing I remember sorta missing from that idea at the time was a fallback to local spool if the other dovecot server isnt available. Cor
Re: [Dovecot] quick question
Wow, that's almost the exact same setup we use, except we have 10 IMAP/POP and a clustered pair of FAS920's with 10K drives which are getting replaced in a few weeks. We also have a pair of clustered 3050's, but they're not running dovecot (yet). Pretty much the same as us as well. 35 imap servers. 10 pop servers. clustered pair of 6080s, with about 250 15K disks. We're seeing some corruption as well. I myself am using imap extensively and regularly have problems with my inbox disappearing. Im not running the patch yet though. Is 1.2.10 imminent or should i just patch 1.2.9? Cor
Re: [Dovecot] quick question
You guys must serve a pretty heavy load. What's your peak connection count across all those machines? How's the load? We recently went through a hardware replacement cycle, and were targeting 25% utilization at peak load so we can lose one of our sites (half of our machines are in each site) without running into any capacity problems. We're actually at closer to 10% at peak, if that... Probably less now that we've disabled noac. Dovecot is fantastic :) I think the peak is around 1 concurrent connections, out of about 500,000 mailboxes. The servers are way overspecced, so we can lose half of them. The netapps are also being used for webservices. Cor
Re: [Dovecot] 1.2.9 imap crash with backtrace
We're seeing this same coredump on a few dozen customers as well. Also Netapp/NFS. cor
[Dovecot] problem with uidlist
In both 1.2.6 and 1.2.7 (probably also before that, but dont have logs) im seeing quite a few of these: Nov 17 12:45:12 userimap10.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 2650: 1257768394._smtp.mxdrop238.41874,S=17304:2,Sad (uid 34270 - 37166) I never really looked at it because it doesnt seem to be causing many problems, but i just noticed something that probably hints at the cause.. Nov 17 12:45:12 userimap10.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 2650: 1257768394._smtp.mxdrop238.41874,S=17304:2,Sad (uid 34270 - 37166) Nov 17 12:45:12 userimap10.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 2651: 1257769385._smtp.mxdrop155.53889,S=305996:2,Sad (uid 34271 - 37167) Nov 17 12:45:12 userimap10.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 2652: 1257769984._smtp.mxdrop107.23815,S=320781:2,Sabd (uid 34272 - 37168) Nov 17 12:45:12 userimap10.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 2670: 1257858287._smtp.mxdrop131.43563,S=26827:2,Sad (uid 34311 - 37186) Nov 17 12:45:12 userimap10.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 2671: 1257858303._smtp.mxdrop238.88513,S=2143:2,Sad (uid 34312 - 37187) Nov 17 12:45:12 userimap10.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 2672: 1257858868._smtp.mxdrop217.21298,S=8548:2,Sad (uid 34314 - 37188) Nov 17 12:45:12 userimap7.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 994: 1244800736._smtp.mxdrop237.59318,S=4573:2,Sad (uid 32467 - 35510) Nov 17 12:45:12 userimap7.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 995: 1244807927._smtp.mxdrop215.53275,S=439887:2,Sad (uid 32468 - 35511) Nov 17 12:45:12 userimap7.xs4all.nl dovecot: IMAP(xxx): /var/spool/mail/dovecot-control/c/c0/xxx/INBOX/.INBOX/dovecot-uidlist: Duplicate file entry at line 996: 1244828590._smtp.mxdrop111.81795,S=3606:2,RSabd (uid 32469 - 35512) This customer seems to be using our IMAP service from 2 machines at the same time. Possibly a webmail service and a desktop imap client. It may be that both machines are trying to manipulate the uidlist at the same time, causing who knows that kind of corruption. This could maybe explain some of the coredumps im seeing in 1.2.7? This is on Debian/Lenny, using NFS spool and NFS control/indexes. Timo, you asked in private mail a few days ago if we're sure the filenames are forever unique. Im sure. I double checked it in the source code. The above filenames are timestamp_hostname.pid, and that seems pretty unique right? Regards, Cor
[Dovecot] panic 1.2.7
Im getting a panic from a specific user: Nov 11 13:53:29 userimap24.xs4all.nl dovecot: IMAP(xx): Panic: file maildir-uidlist.c: line 1242 (maildir_uidlist_records_drop_expunges): assertion failed: (recs[i]- uid rec-uid) Nov 11 13:53:29 userimap24.xs4all.nl dovecot: IMAP(xx): Raw backtrace: imap [0x4a6c02] - imap [0x4a6c83] - imap [0x4a6346] - imap [0x444d6d] - imap(maildir_uidlist_sync_finish+0x2d3) [0x445d83] - imap [0x440d7e] - imap(maildir_storage_sync_init+0x148) [0x441338] - imap(mailbox_sync+0x30) [0x46d3b0] - imap(imap_status_get+0x88) [0x42a9d8] - imap(cmd_status+0x12f) [0x421e0f] - imap [0x423b7c] - imap [0x423c2d] - imap(client_handle_input+0x45) [0x423da5] - imap(client_input+0x5f) [0x42473f] - imap(io_loop_handler_run+0xcb) [0x4aedab] - imap(io_loop_run+0x18) [0x4ae228] - imap(main+0x537) [0x42c707] - /lib/libc.so.6(__libc_start_main+0xe6) [0x7f0cef0471a6] - imap [0x41c859] Nov 11 13:53:30 userimap24.xs4all.nl dovecot: dovecot: child 3594 (imap) killed with signal 6 (core dumps disabled) 1.2.7, Linux/Debian Lenny, amd64, NFS spool and index. Seems to be happing every few minutes, probably every time they check mail. The command they're sending: A002 STATUS INBOX (MESSAGES UNSEEN RECENT) Output doesnt show any result, only their login: A001 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE SORT THREAD=REFERENCES THREAD=REFS MULTIAPPEND UNSELECT IDLE CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH STARTTLS AUTH=PLAIN QUOTA] Logged in Regards, Cor
Re: [Dovecot] panic 1.2.7
Im seeing multiple users with this same panic now. Probably a few an hour (out of tens of thousands of users, so nothing overly dramatic), and a new panic as well: Nov 11 14:34:59 userimap20.xs4all.nl dovecot: IMAP(yy): Panic: file maildir-uidlist.c: line 403 (maildir_uidlist_records_array_delete): assertion failed: (pos != NULL) Nov 11 14:34:59 userimap20.xs4all.nl dovecot: IMAP(yy): Raw backtrace: imap [0x4a6c02] - imap [0x4a6c83] - imap [0x4a6346] - imap [0x44347c] - imap(maildir_uidlist_refresh+0x8f5) [0x25] - imap [0x440e60] - imap(maildir_storage_sync_init+0x148) [0x441338] - imap(imap_sync_init+0x68) [0x42bc88] - imap [0x41] - imap [0x45e37e] - imap(io_loop_handle_timeouts+0xfb) [0x4ae4ab] - imap(io_loop_handler_run+0x71) [0x4aed51] - imap(io_loop_run+0x18) [0x4ae228] - imap(main+0x537) [0x42c707] - /lib/libc.so. 6(__libc_start_main+0xe6) [0x7f913b3b11a6] - imap [0x41c859] Nov 11 14:34:59 userimap20.xs4all.nl dovecot: dovecot: child 11196 (imap) killed with signal 6 (core dumps disabled)
Re: [Dovecot] I've moved to US
Congratulations, sounds like a great place to be, Cor
Re: [Dovecot] unkillable imap process(es) with high CPU-usage
The other guy who also had a problem was using 2.6.27. If the 3rd guy also replies that he's using 2.6.27 then that's pretty clearly the problem. Might be worth asking about in Linux kernel mailing list. We've had something similar happen on 2.6.24. Over time processes would take more and more CPU until they're at 100%. I moved to 2.6.26 and for now I havent seen this happen on any of our 35 servers. It seemed to be NFS related in our case. Cor
Re: [Dovecot] Source patches from Apple
Very cool to see Apple contribute to dovecot! Cor
Re: [Dovecot] Folder aliases
May be Outlook is buggy, but your mentioned Thunderbird has it's own issue: When I want to use german names for a user I setup his mail account with a folder named Papierkorb as trash folder (german translation). This can be used with many email clients without problems. But a german Thunderbird wants to create a new folder Trash and displays it as Papierkorb. So what is the result? In Thunderbird I have two folders named as Papierkorb where I see in other clients a folder Trash and a folder Papierkorb. (The same applies to sent, draft and templates folders.) It's 'simply' a translation issue and one of the problems with translating folder names. The same happens with roundcube, and I removed the folder translation code from the source. It's just too confusing for people when they end up with 2 folders that seem to have the same name. Cor
Re: [Dovecot] v1.1.6 released
If there were no error messages logged with 1.1.5, there's nothing I can think of that could explain it. Im not sure when this started, but im seeing very high CPU use in dovecot. I recently swapped our systems to Linux from FreeBSD, and ive also moved versions up a few (now on 1.1.6). Here's an idea of what im seeing: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 28554 xxx 20 0 11108 1452 1136 R 17 0.0 0:00.76 imap 28542 xxx 20 0 11004 1456 1120 R 16 0.0 0:00.86 imap 28552 xxx 20 0 11108 1504 1136 R 16 0.0 0:00.64 imap 28550 xxx 20 0 11168 1556 1176 R 15 0.0 0:00.78 imap 28485 xxx 20 0 11192 1588 1152 R 14 0.0 0:01.58 imap 24161 xxx 20 0 11124 1684 1204 R 13 0.0 0:20.44 imap 28521 xxx 20 0 13388 1816 1308 R 13 0.0 0:01.60 imap 28545 xxx 20 0 11108 1448 1136 R 13 0.0 0:00.82 imap 28486 xxx 20 0 11232 1612 1152 R 12 0.0 0:01.38 imap 28565 xxx 20 0 11268 1644 1148 R 12 0.0 0:00.56 imap 32470 xxx 20 0 15972 2428 1400 R 11 0.1 0:55.74 imap 28520 xxx 20 0 11152 1600 1212 R 11 0.0 0:01.10 imap 25056 xxx 20 0 13584 2072 1396 R 10 0.1 0:36.64 imap 25677 xxx 20 0 11108 1612 1172 R 10 0.0 0:09.22 imap 28347 xxx 20 0 11328 1708 1220 R 10 0.0 0:03.54 imap At what stage does dovecot use a lot of CPU? What makes this more complicated is that im not seeing it on all servers. I currently have 35 in production and about 5 are showing this. A bit too many to be a server issue. Load is equally distributed so it's not that those 5 have more users on them. Cor
