Re: index corruption weirdness
On 10/10/18 7:26 AM, Aki Tuomi wrote: >> Are you saying that there is a bug in this version that affects RHEL 7.5 >> but not RHEL 6 or just use the newest version and maybe the problem goes >> away? > > We have very limited interest in figuring out problems with (very) old > dovecot versions. At minimum you need to show this problem with 2.2.36 > or 2.3.2.1. > > A thing you should make sure is that you are not accessing the user with > two different servers concurrently. The directors appear to be working fine so, no, users aren't hitting multiple back end servers. To be clear, we don't suspect Dovecot as much - our deployment had been stable for years - but rather behavior changes between the RHEL6 and RHLE7 environment, particularly with regards to NFSv3. But we've have been at a loss to find a smoking gun. For various reasons achieving stability (again) on the current version is very important while we continue to plan Dovecot and storage backend upgrades. Corruption leading to crashes is very infrequent percentage wise but it's enough to negatively impact performance and impact users -- out of 5+ million sessions/day we're seeing ~5 instances whereas on 6 it would have been one every few months. Has anyone else experienced any NFS/locking issues transitioning from RHEL6 to 7 with Netapp storage? Grasping at straws - perhaps compiler and/or system library issues interacting with Dovecot? -K
namespace alias issues?
The Courier IMAP backwards compatibility section in the Namespaces wiki suggests that hidden aliased namespace can cause problems for some clients but doesn't quantify the extent or likelihood of running into these problems. Does anyone with real world experience with similar configurations have any feedback? -- kelsey.cummi...@sonic.com sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Storage Design (Regligious War, Whatever)
We've been running dovecot director setup using MAILDIR++ spools on NFS served by 4 (aging) Netapp clustered filers and are considering modernizing our storage. There's nothing particularly wrong with the existing storage except the age, density and power consumption don't compare well when against new systems. Refreshing the Netapps is an option but they come at a substantial cost. The last time we were looking one of the recommended hot DIY setups was Linux NFS backed by XFS on N mirrors but this predated the availability of stable ZFS in FreeBSD or Linux as well as products like FreeNAS. A FreeBSD NFS/ZFS filer on commodity hardware with zil/l2arc on a PCIe SSD seems like an attractive, affordable and easily scaled out solution which also would allow us to leverage compression at the filesystem layer. Does anyone have any experience running ZFS spool storage? If so, how do you handle DR/HA for spool storage? Thoughts on how this might compare to using DRBD? Any pitfalls to watch out for or general pointers? Suggestions on pool configuration? -K
Re: [Dovecot] courier to dovecot
On Mon, Jul 15, 2013 at 03:13:54PM -0700, J Gao wrote: Now I want to build a new system on CentOS 6.4 64bit with postfix, dovecot and migrate all user accounts and their emails. I assume you've already reviewed the migration pages on the wiki? http://wiki2.dovecot.org/Migration/Courier My only advice would be to consider dropping the INBOX. prefix - this has some significant affects for the migration but will prevent headaches from clients that don't handle namespaces correctly (like all iOS devices.) Or, maybe add a second hidden namespace for INBOX. compatibility so existing clients that are hard coded hopefully wont trip up on themselves too bad. -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] avoid log message when using nagios check
On 2013-07-10 05:16, Daniel Parthey wrote: Just do a complete login/logout sequence. If you aren't doing a complete login/logout sequence and possibly even pulling down a message you aren't performing an accurate health check to begin with. We don't use nagios but I'd be surprised if those scripts don't optionally take a username and password. This is fine for the nagios checks, but we are facing similar problems with our loadbalancer, which is just doing TCP Healthchecks on the IMAP/POP3/SIEVE ports, so being able to disable the warning for trusted networks would be really helpful. Same here. We use LVS with surealived and LUA scripting to do a complete login/logout cycle as part of the health check. Even the ancient Alteon's had similar functionality (and supported common protocols like pop and imap out of the box.) -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] Idea: POP3 deletion as a flag
On 2013-05-03 19:13, Professa Dementia wrote: When I specify that an email be deleted from the server, I expect that it is *deleted*. While I see the point you're trying to make, I don't think it is valid. On our servers the deleted message could exist in filesystem snapshots, disk-disk backups and on tape. In many other places this may be a question of regulatory requirements that email be held on to for years no matter what the client thinks its status is.
Re: [Dovecot] Idea: POP3 deletion as a flag
On 2013-05-03 09:14, Timo Sirainen wrote: GMail doesn't delete mails when POP3 client issues a DELE command for it. Instead they just become invisible for future POP3 sessions, but they still exist for IMAP/webmail. The same could be implemented pretty easily for Dovecot: How does the usage case by your large customer differ from that allowed by the lazy_expunge plugin?
Re: [Dovecot] Idea: POP3 deletion as a flag
On 2013-05-03 15:44, Timo Sirainen wrote: I didn't ask what their main reason for this was, but for me it would be: Oops, I accidentally configured my new email client as POP3 instead of IMAP, and now it deleted everything from my INBOX. With lazy_expunge the user would have to explicitly go and undelete the mails, and it would also undelete those mails that were intentionally deleted. With this feature nothing at all would go wrong on IMAP/webmail side. Ah, that makes more sense now. We're only using lazy_expunge for POP3 but with a namespace visible to IMAP so a POP3 user can restore a message using webmail/IMAP if needed. -K
Re: [Dovecot] ios clients and namespace trouble
On 4/24/2013 12:05 PM, Kelsey Cummings wrote:
before or if they had tried and failed. Perhaps a hidden namespace with
folders linked to the real special folders or might that have unintended
consequences?
This seems to kinda work with the only oddity being that the ios client,
if not manually configured with the correct prefix, ends up creating new
folders in the hidden one and initially displaying them at the same
level as the inbox. Once the app is restarted it sees them in the
correct namespace as a folder under the inbox. However, looks like
some other clients might get confused, but maybe Windows Live Mail is
going to get confused anyway. ;)
namespace {
type=private
separator = .
prefix = INBOX.
inbox = yes
mailbox Trash {
auto = create
special_use = \Trash
}
...
}
namespace FAKE {
type=private
separator = .
hidden = yes
list = no
mailbox Trash {
special_use = \Trash
}
...
}
--
Kelsey Cummings - k...@corp.sonic.net sonic.net, inc.
System Architect 2260 Apollo Way
707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] ios clients and namespace trouble
On 2013-04-24 10:54, Robert Schetterer wrote: Apple has a long history with bugs on imap, they are focused in using their own services, so they might never fix outside stuff, but however why not ask Apple for help, you paid a lot to them I've never paid them a dime to them in my life but I probably have many thousands of their devices talking to my imap servers and this issue leads to a support burden on our end. I recognize that their client is apparently broken on this point and that asking apple to fix it is pointless. Perhaps ironically, the local apple store knows about this problem but views it as our problem and not theirs - they only know that they get a people in the store on a regular basis asking for help because they can't delete their mail on their sonic.net mail accounts. I was curious if anyone else had come up with a work around for this before or if they had tried and failed. Perhaps a hidden namespace with folders linked to the real special folders or might that have unintended consequences? -K
Re: [Dovecot] stats plugins causing dns lookup per connection
On 03/12/13 06:58, Axel Luttgens wrote: started for seemingly unrelated reasons, but ended with some form of cacheing of the results fetched with gethostbyname(). It is a bit odd that it would totally block unless DNS requests on your hosts weren't working at all - and even then, the requests would timeout eventually and unblock. Which version of Dovecot are you running? 2.1.13 I see that caching is in 2.2 now too. Timo, if there are any other 2.1.x releases it'd be nice to get this back ported to it as well. -K
[Dovecot] stats plugins causing dns lookup per connection
I noticed our imap servers were generating a lot of A record lookups for their own IP's the other day and just got around to tracking down the source. Seems like they are all being caused by guid_128_generate() - perhaps the lookup could be cached at start up or it could just use make use of the hostname rather than spending the effort to get the IP via gethostbyname() calls. The function is used in a few other places too, so this might help more than just the stats plugin. nscd and/or host entries mitigate the total time spent on the lookup of course, but it seems unnecessary. -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] memory allocation issues
On Fri, Nov 23, 2012 at 08:36:37AM +0200, Timo Sirainen wrote:
On 9.11.2012, at 2.49, Kelsey Cummings wrote:
One of our dovecot backend servers ran into a problem with it's auth
process a few days ago. This doesn't appear to be the error logged when
dovecot hits its internal limit so I'm not sure what is going on here.
auth: Error: malloc: 58012: Cannot allocate memory
auth: Error: Unable to allocate memory for mutexes from the region
auth: Error: PANIC: Cannot allocate memory
auth: passwd(test,1.1.1.1,8HTlNHzNIQBAjhKC): unknown user
It would have been nicer if libc would have just crashed the process instead
of silently converting it into unknown user error.. That's probably
actually a bug since the getpwuid_r() that Dovecot uses would have been able
to return an error message.
We saw two boxes do this over the weekend.
pop3: Error: Authenticated user not found from userdb, auth lookup
+id=2509111297 (client-pid=4781 client-id=1)
pop3-login: Internal login failure (pid=4781 id=1) (internal failure, 1
+succesful auths): user=test...
There was at least 10+GB free RAM on the server and no indication of a
system level issue at the same time. The server is running 2.1.9.
There were about 3,200 active sessions, with something like 12 new
sessions/sec. The other identical servers are/were handling virtually
identical load with the same service uptime and haven't had any issues
so far. (Crash happened 7 days ago.)
Memory leak maybe? service auth { vsz_limit } anyway was reached (default 256
MB).
It is currently set to 768M, I'll go ahead and raise it up to 1G.
Anything I can do to help see if it is a memory leak?
# dovecot -n
# 2.1.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.9.1.el6.x86_64 x86_64 Scientific Linux release 6.3
(Carbon)
auth_master_user_separator = *
auth_username_format = %Ln
auth_verbose = yes
auth_verbose_passwords = sha1
auth_worker_max_count = 64
login_log_format_elements = user=%u session=%{session} method=%m rip=%r
lip=%l mpid=%e %c
mail_fsync = always
mail_log_prefix = %s(%u): session=%{session}
mail_plugins = stats zlib
maildir_very_dirty_syncs = yes
mmap_disable = yes
namespace {
inbox = yes
location =
prefix = INBOX.
separator = .
type = private
}
passdb {
args = /etc/dovecot/master-users
driver = passwd-file
master = yes
}
passdb {
args = imap
driver = pam
}
plugin {
lazy_expunge = DELETED_MESSAGES.
mail_log_events = delete expunge flag_change
mail_log_fields = uid box msgid from flags size
quota = fs:User quota
stats_refresh = 30 secs
stats_track_cmds = yes
}
protocols = imap pop3
service anvil {
client_limit = 1
}
service auth {
client_limit = 1
vsz_limit = 768 M
}
service doveadm {
inet_listener {
port = 1842
}
unix_listener doveadm-server {
mode = 0666
}
}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_limit = 7000
process_min_avail = 32
vsz_limit = 256 M
}
service imap-postlogin {
executable = script-login -d /etc/dovecot/bin/sonic-imap-postlogin
user = $default_internal_user
}
service imap {
executable = imap imap-postlogin
process_limit = 4096
vsz_limit = 512 M
}
service pop3-login {
inet_listener pop3 {
port = 110
}
inet_listener pop3s {
port = 995
ssl = yes
}
process_limit = 2000
process_min_avail = 32
vsz_limit = 256 M
}
service pop3-postlogin {
executable = script-login -d /etc/dovecot/bin/sonic-pop3-postlogin
user = $default_internal_user
}
service pop3 {
executable = pop3 pop3-postlogin
process_limit = 4096
}
service stats {
fifo_listener stats-mail {
mode = 0666
}
}
shutdown_clients = no
ssl = required
ssl_parameters_regenerate = 1 days
syslog_facility = local0
userdb {
driver = passwd
}
verbose_proctitle = yes
protocol imap {
imap_id_send = support-url support-email
mail_max_userip_connections = 20
mail_plugins = stats zlib mwi_update mail_log notify imap_stats imap_zlib
}
protocol pop3 {
mail_plugins = stats zlib lazy_expunge
pop3_fast_size_lookups = yes
pop3_uidl_format = %f
}
--
Kelsey Cummings - k...@corp.sonic.net sonic.net, inc.
System Architect 2260 Apollo Way
707.522.1000 Santa Rosa, CA 95407
[Dovecot] memory allocation issues
One of our dovecot backend servers ran into a problem with it's auth process a few days ago. This doesn't appear to be the error logged when dovecot hits its internal limit so I'm not sure what is going on here. auth: Error: malloc: 58012: Cannot allocate memory auth: Error: Unable to allocate memory for mutexes from the region auth: Error: PANIC: Cannot allocate memory auth: passwd(test,1.1.1.1,8HTlNHzNIQBAjhKC): unknown user pop3: Error: Authenticated user not found from userdb, auth lookup +id=2509111297 (client-pid=4781 client-id=1) pop3-login: Internal login failure (pid=4781 id=1) (internal failure, 1 +succesful auths): user=test... There was at least 10+GB free RAM on the server and no indication of a system level issue at the same time. The server is running 2.1.9. There were about 3,200 active sessions, with something like 12 new sessions/sec. The other identical servers are/were handling virtually identical load with the same service uptime and haven't had any issues so far. (Crash happened 7 days ago.) -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] (new) director issues in 2.1.10
On Mon, Oct 22, 2012 at 03:39:34PM +0300, Timo Sirainen wrote: On 26.9.2012, at 21.06, Kelsey Cummings wrote: 09:25:21 .. User X host lookup failed: Timeout - queued for 30 secs (Ring synced for 5032 secs) 09:25:55 .. User X host lookup failed: Timeout - queued for 30 secs (Ring synced for 5066 secs, weak user, user refreshed 64 secs ago) 09:26:28 .. User X host lookup failed: Timeout - queued for 30 secs (Ring synced for 5099 secs, weak user, user refreshed 97 secs ago) Looks like I had broken this in v2.1.8. http://hg.dovecot.org/dovecot-2.1/rev/e4c337f38ed6 fixes this. I also added a bunch of other things to give better error messages and to try to fix any unexpected problems. Thanks Timo! -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] possible nfs issue
On 10/2/2012 2:39 PM, Cor Bosman wrote: Anyone else with NFS mailspools seeing this? Yes, it is like 1999 all over again. I haven't had a chance to track them down or setup a cron job to rm them all. All of the ones I'm seeing are ex dovecot.index files but it looks like yours are ex messages? I figured this was a probably a regression in the RHEL6.3 (Sl6.3) (2.6.32-279.9.1.el6.x86_64) kernel. What are you running Cor? -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
[Dovecot] (new) director issues in 2.1.10
Timo - I upgraded to 2.1.10 on our director servers two nights ago and apart from errors associated with the directors processes restarting everything looked great for ~24 hours until I failed our the real servers last night to update the nfs mount options for the spools. I followed the suggested procedure for each backend server, just run on one of the directors, which seemed to work as expected. doveadm director add x.x.x.x 0 doveadm director flush x.x.x.x The following errors on the directors that started after this went unnoticed until this AM. director: User bb host lookup failed: Timeout - queued for 30 secs (Ring synced for 36 secs) director: User cc host lookup failed: Timeout - queued for 48 secs (Ring synced for 66 secs, user refreshed 12 secs ago) director: User dd host lookup failed: Timeout - queued for 124 secs (Ring synced for 119 secs, weak user, user refreshed 155 secs ago) director: User ee host lookup failed: Timeout - queued for 79 secs (Ring synced for 119 secs, weak user, user refreshed 113 secs ago) ... User ff host lookup failed: Timeout - queued for 30 secs (Ring synced for 7427 secs, weak user, user refreshed 620 secs ago) This continued, combined with occasional login timeouts (as reported by some internal imap clients.) The login delays/timeouts got bad enough that our load balancers dropped both the servers while I was investigating. They seem to be okay after being restarted. -K
Re: [Dovecot] (new) director issues in 2.1.10
On Wed, Sep 26, 2012 at 08:57:58PM +0300, Timo Sirainen wrote: On 26.9.2012, at 20.34, Kelsey Cummings wrote: The following errors on the directors that started after this went unnoticed until this AM. director: User bb host lookup failed: Timeout - queued for 30 secs (Ring synced for 36 secs) director: User cc host lookup failed: Timeout - queued for 48 secs (Ring synced for 66 secs, user refreshed 12 secs ago) director: User dd host lookup failed: Timeout - queued for 124 secs (Ring synced for 119 secs, weak user, user refreshed 155 secs ago) director: User ee host lookup failed: Timeout - queued for 79 secs (Ring synced for 119 secs, weak user, user refreshed 113 secs ago) ... User ff host lookup failed: Timeout - queued for 30 secs (Ring synced for 7427 secs, weak user, user refreshed 620 secs ago) This continued, combined with occasional login timeouts (as reported by some internal imap clients.) The login delays/timeouts got bad enough that our load balancers dropped both the servers while I was investigating. They seem to be okay after being restarted. After the first few minutes, did all the rest of the error messages contain weak user string? Did this happen to a lot of different users (few/some/most)? director_user_expire setting is the default 15 minutes? No, there continued to be a mix of both. The pattern seems to look like this. I'll run some stats later but it looks like a pretty significant number of users where affected. 09:25:21 .. User X host lookup failed: Timeout - queued for 30 secs (Ring synced for 5032 secs) 09:25:55 .. User X host lookup failed: Timeout - queued for 30 secs (Ring synced for 5066 secs, weak user, user refreshed 64 secs ago) 09:26:28 .. User X host lookup failed: Timeout - queued for 30 secs (Ring synced for 5099 secs, weak user, user refreshed 97 secs ago) -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] (new) director issues in 2.1.10
On 09/26/12 11:06, Kelsey Cummings wrote: No, there continued to be a mix of both. The pattern seems to look like this. I'll run some stats later but it looks like a pretty significant number of users where affected. Timo, it looks like the total number of affected users was only about 250 and that most of their erred connections were surrounded by successful sessions. -K
[Dovecot] Proxy connection timeouts
We are seeing a few (0-15) proxy failures like the following out of ~3m
successful proxied connections a day. Average session creation load over
our peak hour is about 47/sec. The backend servers aren't logging
anything that would suggest any internal problem like insufficient
processes to handle the load. It doesn't seem to happen when
utilization is lowest at night.
dovecot: imap-login: Error: proxy(foo): connect(1.1.1.1, 143) failed:
Connection timed out (after 63 secs)
I'm curious if anyone else has seen any similar problems or has any
suggestions.
# dovecot -n
# 2.1.9: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-279.5.2.el6.x86_64 x86_64 Scientific Linux release
6.3 (Carbon)
auth_master_user_separator = *
auth_username_format = %Ln
auth_verbose = yes
auth_verbose_passwords = sha1
auth_worker_max_count = 64
mail_fsync = always
mail_log_prefix = %s(%u): session=%{session}
mail_plugins = stats zlib
maildir_very_dirty_syncs = yes
mmap_disable = yes
passdb {
args = /etc/dovecot/master-users
driver = passwd-file
master = yes
}
passdb {
args = imap
driver = pam
}
plugin {
lazy_expunge = DELETED_MESSAGES.
stats_refresh = 30 secs
stats_track_cmds = yes
}
protocols = imap pop3
service anvil {
client_limit = 1
}
service auth {
client_limit = 1
vsz_limit = 512 M
}
service doveadm {
inet_listener {
port = 1842
}
unix_listener doveadm-server {
mode = 0666
}
}
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
process_limit = 7000
process_min_avail = 32
}
service imap-postlogin {
executable = script-login -d /etc/dovecot/bin/sonic-imap-postlogin
user = $default_internal_user
}
service imap {
executable = imap imap-postlogin
process_limit = 4096
vsz_limit = 512 M
}
service pop3-login {
inet_listener pop3 {
port = 110
}
inet_listener pop3s {
port = 995
ssl = yes
}
process_limit = 2000
process_min_avail = 32
}
service pop3-postlogin {
executable = script-login -d /etc/dovecot/bin/sonic-pop3-postlogin
user = $default_internal_user
}
service pop3 {
executable = pop3 pop3-postlogin
process_limit = 4096
}
service stats {
fifo_listener stats-mail {
mode = 0666
}
}
shutdown_clients = no
ssl = required
ssl_ca = /etc/dovecot/ssl/gd_bundle.crt
ssl_cert = /etc/dovecot/ssl/imap.sonic.net.crt
ssl_key = /etc/dovecot/ssl/imap.sonic.net.key
ssl_parameters_regenerate = 1 days
syslog_facility = local0
userdb {
driver = passwd
}
verbose_proctitle = yes
protocol imap {
imap_id_send = support-url support-email
mail_max_userip_connections = 20
mail_plugins = stats zlib mwi_update mail_log notify imap_stats imap_zlib
ssl_ca = /etc/dovecot/ssl/gd_bundle.crt
ssl_cert = /etc/dovecot/ssl/imap.sonic.net.crt
ssl_key = /etc/dovecot/ssl/imap.sonic.net.key
}
protocol pop3 {
mail_plugins = stats zlib lazy_expunge
pop3_fast_size_lookups = yes
pop3_uidl_format = %f
ssl_ca = /etc/dovecot/ssl/pop.sonic.net.ca-bundle
ssl_cert = /etc/dovecot/ssl/pop.sonic.net.crt
ssl_key = /etc/dovecot/ssl/pop.sonic.net.key
}
Re: [Dovecot] dovecot stats: useful data to gather
On 06/02/12 17:10, Daniel Parthey wrote: Patrick Ben Koetter wrote: following our discussion on dovecot stats at the LinuxTag 2012 my team and I sat down and put together a list of stat items we think to be useful in daily dovecot usage. Besides pulling together all the data we also think it would be useful to have an SNMP interface to access the stats. Our offer to create and contribute a standalone web interface for dovecot stats stands. This should be done via SNMP subagent, but how could you differentiate different dovecot instances on the same machine, different snmp ports for the subagent, or different snmp trees? I'd suggest some additional performance metrics like min/max/avg time to authenicate, establish a proxy session and perhaps include auth failure causes counters as well. I personally wouldn't want to see this implemented as an SNMP subagent but so long as the stats would be available off a local socket directly I think everyone would be happy. -K
Re: [Dovecot] TIMO HELP! director ring wont stay connected
On 09/03/12 12:06, Timo Sirainen wrote: On 3.9.2012, at 21.26, Kelsey Cummings wrote: I've had 2x director ring up and running with production load on 2.1.8 with around 10,000 active connections for two weeks and everything has been working great - until this morning. There isn't anything obvious in the logs beyond the fact that the director connections started bouncing. It was not resolved by reloads or restarts or an upgrade to 2.1.9 (only the directors.) Did you try stopping both and then starting them again? That clears up all the state they have. I stopped both directors last night and they were able to stay in sync after they were restarted. Could corruption of the in memory state lead to the connections being dropped? If this happens again I'll try to get a tcpdump and an strace so the bug can get squashed. -K
Re: [Dovecot] TIMO HELP! director ring wont stay connected
On 9/4/2012 5:58 PM, Timo Sirainen wrote:
On 3.9.2012, at 21.26, Kelsey Cummings wrote:
passdb {
args = proxy=y nopassword=y
driver = static
}
I wonder if someone was doing a ton of logins for different usernames? This
kind of setup where director doesn't verify the username can be attacked that
way.
It doesn't look like there was a higher than normal number of failed
logins leading up to the connection issues. I'm going to write some
more stats collection tools to track state on the directors and see what
comes of it.
Can the director proxy validate the username via a unix pw lookup but
not check the password?
--
Kelsey Cummings - k...@corp.sonic.net sonic.net, inc.
System Architect 2260 Apollo Way
707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] TIMO HELP! director ring wont stay connected
On 9/3/2012 12:06 PM, Timo Sirainen wrote: Did you try stopping both and then starting them again? That clears up all the state they have. I'm not sure that they were both down when restarting them and will try this tonight. If the state clearing doesn't help, maybe this has something to do with the OS or the network is really having some issues. I can't rule that out but there are not any signs that there are any hardware, OS or network related issues. Thanks for gettting the ring status into doveadm by the way. At least our monitoring caught this quickly. -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] dovecot stats error
On 6/22/2012 6:34 AM, Timo Sirainen wrote: Which Dovecot version? I thought I fixed this already.. I'm seeing these errors running 2.1.8 -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] dovecot stats error
On 8/25/2012 12:14 PM, Kelsey Cummings wrote: On 6/22/2012 6:34 AM, Timo Sirainen wrote: Which Dovecot version? I thought I fixed this already.. I'm seeing these errors running 2.1.8 Examples below, let me know if I can provide any other info Timo. In other news, we're finally migrated to dovecot from courier. WHOOO H Aug 25 12:53:37 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mcache 331 332 Aug 25 12:53:37 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mrbytes 180435729 204849088 Aug 25 12:53:38 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mrbytes 50757363 62351358 Aug 25 12:53:38 d dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mlpath 17451 20067 Aug 25 12:53:41 d dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mrbytes 40483661 42086237 Aug 25 12:53:42 b dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mrbytes 65540465 67974537 Aug 25 12:53:42 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mlpath 811 946 Aug 25 12:53:43 b dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mrbytes 220133763 221888538 Aug 25 12:53:47 a dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mcache 13 14 Aug 25 12:53:48 c dovecot: stats: Error: Mail server input error: UPDATE-SESSION: stats shrank: mrbytes 118702153 121714865 -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] RAID1+md concat+XFS as mailstorage
On 06/28/12 05:56, Ed W wrote: So given the statistics show us that 2 disk failures are much more common than we expect, and that silent corruption is likely occurring within (larger) real world file stores, there really aren't many battle tested options that can protect against this - really only RAID6 right now and that has significant limitations... Has anyone tried or benchmarked ZFS, perhaps ZFS+NFS as backing store for spools? Sorry if I've missed it and this has already come up. We're using Netapp/NFS, and are likely to continue to do so but still curious. -K
Re: [Dovecot] High level of pop3 popping causing server to become unresponsive
On 5/18/2012 6:21 AM, Root Kev wrote: During the last time that the load went up, it became unable to login / su to root for the entire period that dovecot was running, we had to kill This sounds more like you are getting I/O bound or swapping heavily. What does iostat -x, etc, show when this is happening? -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
[Dovecot] POP3 dele to Trash?
To simplify recovery of accidentally deleted messages we'd like to send all messages that are deleted by pop clients to their Trash folder. (Which is auto expired already.) This allows a POP client who deletes an important email to restore it themselves by logging into our webmail client and move the lost message from Trash to the Inbox where it would be fetched by their client again. Has anyone already done this? Should this be possible via a plugin? I see the deleted-to-trash imap plugin. We are using Maildir if it makes a difference. -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] POP3 dele to Trash?
On 04/06/12 16:40, Kelsey Cummings wrote: Has anyone already done this? Should this be possible via a plugin? I see the deleted-to-trash imap plugin. We are using Maildir if it makes a difference. Of course, this is exactly what the Lazy Expunge plugin does, isn't it? -K
Re: [Dovecot] POP3 Performance
On 03/16/12 06:07, Timo Sirainen wrote: Maildir isn't very good for POP3, especially if the POP3 clients delete the mails after download. With Dovecot you could look into switching to multi-dbox format, which would have much better performance. Timo, can you explain why Maildir isn't a good for POP3 in this context? Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. -K
Re: [Dovecot] Just in time AV scanning
On 03/16/12 08:30, Ed W wrote: 2) Extremely racey, but if you were on maildir you could use some kind of pre-login scripting to kick off a scan on login. Touch some lock file so that you can tell when last scanned and only scan if the definitions have been updated since you last scanned? I think this is actually the best solution to match our existing POP behavior. This was a lot cooler back when 90% of our users were on POP and on average had a couple of hours between checks - it may be a feature that has outlived its usefulness. Still need to take a look at Timo's patch set. -K
Re: [Dovecot] POP3 Performance
On 03/16/12 10:54, Timo Sirainen wrote: Another thing our existing POP3 servers did was batch all of the deletes until after the +OK... was returned from quit. This doesn't reduce server load but has the impression of creating faster response times to the clients. You mean deleting the messages after +OK, instead of before? Does it really make a difference? In the context of a clients send and receive phase taking a (small) fraction of a second less time, perhaps, but it is a small difference in any case. It was one of many small changes we made to try to improve interactive performance. .. Dovecot can reply with -ERR to QUIT if deletions failed for some reason. True, we decided that loosing that ability didn't really matter. (Like not counting newlines as two bytes in the message size.) -K
[Dovecot] Just in time AV scanning
I'm curious if anyone has any plugins for AV integration directly into dovecot. Our old pop servers have been scanning messges as they're moved from new-cur in the inbox and, at least where user's aren't poping every few seconds, there is occasionally enough time between scanning through the MXs to message retreval to snag a few more virues with updated definitions before they reach customers. Anyone doing anything similar? -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] Master Users
On 03/05/12 17:33, Kelsey Cummings wrote: I have a setup where I need to use a Master User account to login on behalf of users normally authed via PAM. Is there any existing mechanism that will allow master users to be wired down to specific ip address rather Ah, found it. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/AllowNets -K
[Dovecot] Master Users
I have a setup where I need to use a Master User account to login on behalf of users normally authed via PAM. Is there any existing mechanism that will allow master users to be wired down to specific ip address rather than having these very magic user/pass combos be valid from any random host? It would be totally acceptable to be able to say that master logins were only valid from a specific list of hosts rather than wiring specific master users to specific hosts. -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] mail spool filesystem
On Fri, Aug 19, 2011 at 03:48:00AM -0500, Stan Hoeppner wrote: On 8/17/2011 9:42 AM, Adrian Ulrich wrote: I read that XFS is a good choice, but is not too reliable... Are you using Maildir or MBOX? In any case: XFS would be my last choice: XFS is nice if you are working with large files ( 2GB), but for E-Mail i'd stick with ext3 (or maybe even reiser3) as it works very well with small files. XFS was designed for parallelism, whether with large files or small, ... Anyone been using ZFS on FreeBSD for mail spool storage? -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] director monitoring?
On Fri, Aug 05, 2011 at 11:12:03AM +0200, Jan-Frode Myklebust wrote: On Thu, Jun 02, 2011 at 12:29:10PM -0700, Kelsey Cummings wrote: I'm using a hacked up version of poolmon. The only important changes are that it actually logs into the real server rather than just making a connection to it and that has heuristics to prevent the real servers from flapping and added a timeout to scan_host so if a real server blocks after the connection is established it won't hang indefinitely. Could you share your hacks ? :-) Sure. You'll probably want to change the regex at line 194 to match whatever your server says after the login is complete. My postlogin script puts out some extra info that I'm looking for instead of the deafult. Otherwise, YMMV, works for me so far. http://kgc.users.sonic.net/imapdmon -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] director monitoring?
On Thu, Jun 02, 2011 at 10:37:23AM +0200, Cor Bosman wrote: We use a setup as seen on http://grab.by/agCb for about 30.000 simultaneous(!) imap connections. This might as well be a diagram of my network, although, if I remember, you're running quite a few more netapps clusters than I am. ;) We have 2 Foundry loadbalancers. They check the health of the directors. We have 3 directors, and each one runs Brandon's poolmon script (https://github.com/brandond/poolmon). This script removes real servers out of the director pool. The dovecot imap servers are monitored with nagios just to tell us when they're down. I'm using a hacked up version of poolmon. The only important changes are that it actually logs into the real server rather than just making a connection to it and that has heuristics to prevent the real servers from flapping and added a timeout to scan_host so if a real server blocks after the connection is established it won't hang indefinitely. This setup has been absolutely rock solid for us. I have not touched the whole system since november and we have not seen any more corruption of meta data, which is the whole reason for the directors. Kudos to Timo for fixing this difficult problem. That is always good to hear! I'd be a lot happier if I was able to monitor the directors and make sure that they were connected and correctly synced with eachother - even as a protection from human error rather than anticipated software failure. -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] doveadm -S socket_path option enabled in 2.0.8?
On Thu, Dec 30, 2010 at 01:52:53PM +0200, Timo Sirainen wrote: On Tue, 2010-12-21 at 14:55 -0500, David Warden wrote: But when I try to get my quota on a different local IP using the -S flag to doveadm: doveadm quota get -u warden -S 137.238.2.244:143 The purpose of -S is completely different. It's about connecting to another doveadm instance. There's currently no way to set IP for doveadm queries. Just to bump an old thread. Timo, I think this would be a great feature to add. It would allow people to pull stats (and system health) directly out of dovecot rather than having to run agents on the dovecot servers to get the info into some other transport like SNMP. -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
[Dovecot] director monitoring?
I'm working the kinks of a new director based setup for the eventual migration away from courier. At this point, with everything basically working I'm trying to ensure that things are properly monitored and I've run into an issue. There doesn't appear to be a way to get dovecot to tell if it is (or is not) connected and properly synced with the other director servers in the ring apart from the logs. It seems like this is an important piece of information -- without it, it isn't apparent how you would be able to tell if your director servers have lost track of each other. I'm also curious what people are doing to health check their director servers when they are running load balancing upstream of them as well. It doesn't seem like it is a good idea to let the load balancers check all the way through to the real servers since a failure on the target real server could end up leading to a director being dropped from the pool (if so, it is most likely that the other directors would be dropped as well.) Otherwise, the health check failure tolerance at the load balancer must be greater than the tolerance for failure of the real servers on the director- a dead director could end up in the pool for longer than desired, or anyway, long enough to be sure that it isn't a transient failure on the real server behind it. A better method would seem to be for the load balancers to query the director for the number of active back-end servers and, so long as it was over a given threshold, to assume that the director is otherwise able to do its job and rely on external monitoring to pickup internal failures where dovecot isn't able to successfully proxy the connection to one of the real servers. So, how are people doing this in the real world? -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
[Dovecot] Client IP log variable for proxy/director setups?
Timo - it would be very convenient if we could get a variable that held the real client ip for use in logging with director/proxy setups. It's clearly already passed around in client-user-remote_ip but at this point, I don't see anyway to get it into the logs and it would take me ages to figure out the Right place to add it since client-user doesn't seem to be available in client-common.c: get_var_expand_table(..). Perhaps I'm missing something obvious? -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] restarting director
On Fri, Jan 21, 2011 at 08:00:08PM +0200, Timo Sirainen wrote: On Fri, 2011-01-21 at 19:59 +0200, Timo Sirainen wrote: I can take a look at it, but it would help if you were able to reproduce the problem. More clearly: Reliably reproduce this in a test setup :) Timo Cor, did you guys ever nail this down? We're looking at migration to a director config soon but I'd like to see this resolved first. Anything we can do do help? -K
[Dovecot] fts, solr, and client support
I've been playing around with the fts plugins and currently have solr up and running. While the initial indexing time is pretty rough, the search performance is impressive. I've run into a problem though- so far as I can tell, thunderbird (and most other popular clients?) won't actually take advantage of it. Even using thunderbird's advanced search and checking the 'run serach on server' box doesn't push the search to the server. If most popular clients don't support it, there doesn't seem much point in setting it up; which is a shame, solr beats the snot out of the built in search for thunderbird. That said, is anyone running any of the fts plugins in a large scale environment? If so, what kind of scaling issues do you see? How do the search servers compare hardware wise to users count/spool size? -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] Questions, Issues with 2.0
On Mon, May 03, 2010 at 05:33:12PM -0700, Kelsey Cummings wrote: But I'm still curious for any tips or pointers on the other issue w/regards adding an exterior maildir into a user's namespace. Anyone? (I'll freely admit to missing something obvious...) I'd rather not accomplish this with symlinks although that wouldn't be a horrible solution if it was the only easy option. -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
[Dovecot] Questions, Issues with 2.0
I'm trying to use an external program to set the location of user's
maildirs to match an existing hashing schema but the docs for how to do
this in 1.x do not seem to apply to 2.0 or I'm misinterpreting them.
protocol imap {
mail_executable = /opt/dovecot-test/sonic-imap
...
This correctly executes the and attempts to sets the MAIL variable but
it appears that mail_executable is run before the user is logged in so
the $USER variable isn't set and the process' uid is still 0.
We also need to have dovecot include an additional maildir for each user
that is currently stored outside of the user's main maildir directory into
the same namespace as a specific folder name. Is there a way to do this
in dovecot directly or will it need to be hacked (as it is currently with
courier) or could it be done with a plugin?
I've also noticed that while doveconf complains about mail_executable:
May 3 13:44:10 a dovecot: config: Obsolete setting in
/opt/dovecot-test/etc/dovecot/conf.d/imap.conf:7: mail_executable has
been replaced by service { executable }
However, when replaced as suggested by doveconf:
protocol_imap {
executable = /opt/dovecot-test/sonic-imap
...
It fails altogether:
doveconf: Fatal: Error in configuration file
/opt/dovecot-test/etc/dovecot/conf.d/imap.conf line 7: Unknown setting:
executable
Similarly, it also suggests 'drop_priv_before_exec = no' which doesn't
appear to be valid either.
--
Kelsey Cummings - k...@corp.sonic.net sonic.net, inc.
System Architect 2260 Apollo Way
707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] Questions, Issues with 2.0
On Mon, May 03, 2010 at 03:25:48PM -0700, Kelsey Cummings wrote: I'm trying to use an external program to set the location of user's maildirs to match an existing hashing schema but the docs for how to do this in 1.x do not seem to apply to 2.0 or I'm misinterpreting them. ... I've confirmed that things work as expected when running 1.x. But I'm still curious for any tips or pointers on the other issue w/regards adding an exterior maildir into a user's namespace. -- Kelsey Cummings - k...@corp.sonic.net sonic.net, inc. System Architect 2260 Apollo Way 707.522.1000 Santa Rosa, CA 95407
Re: [Dovecot] Questions, Issues with 2.0
On Tue, May 04, 2010 at 03:41:04AM +0300, Timo Sirainen wrote:
On 4.5.2010, at 1.25, Kelsey Cummings wrote:
I'm trying to use an external program to set the location of user's
maildirs to match an existing hashing schema but the docs for how to do
this in 1.x do not seem to apply to 2.0 or I'm misinterpreting them.
protocol imap {
mail_executable = /opt/dovecot-test/sonic-imap
http://dovecot.org/list/dovecot/2009-December/045139.html should help.
Indeed! Thanks.
--
Kelsey Cummings - k...@corp.sonic.net sonic.net, inc.
System Architect 2260 Apollo Way
707.522.1000 Santa Rosa, CA 95407
