Re: [Dovecot] imap-login processes
On Mon, 2008-05-19 at 12:06 -0400, Bryan Polk wrote: On Fri, 16 May 2008, Evaggelos Balaskas wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I hope something like this could help you : # Authentication Cache auth_cache_size = 10240 auth_cache_ttl = 18000 This may have done the trick! The number of imap-login processes has been holding around 60 all morning. So I guess most of the login processes were just waiting for authentication processes to respond? I hadn't thought about that. How many dovecot-auth processes do you have? Increasing auth_worker_max_count might be a good idea. signature.asc Description: This is a digitally signed message part
Re: [Dovecot] imap-login processes
On Sun, 2008-05-25 at 05:08 +0300, Timo Sirainen wrote:
On Mon, 2008-05-19 at 12:06 -0400, Bryan Polk wrote:
On Fri, 16 May 2008, Evaggelos Balaskas wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I hope something like this could help you :
# Authentication Cache
auth_cache_size = 10240
auth_cache_ttl = 18000
This may have done the trick! The number of imap-login processes has been
holding around 60 all morning.
So I guess most of the login processes were just waiting for
authentication processes to respond? I hadn't thought about that. How
many dovecot-auth processes do you have? Increasing
auth_worker_max_count might be a good idea.
Or it should have actually logged an error in that case: Couldn't
create new auth worker.
Maybe it's the passwd lookups that are slow? What NSS backend are you
using? It's probably a good idea to make them use auth workers also:
userdb passwd {
args = blocking=yes
}
signature.asc
Description: This is a digitally signed message part
Re: [Dovecot] imap-login processes
On Sun, 25 May 2008, Timo Sirainen wrote:
On Sun, 2008-05-25 at 05:08 +0300, Timo Sirainen wrote:
On Mon, 2008-05-19 at 12:06 -0400, Bryan Polk wrote:
On Fri, 16 May 2008, Evaggelos Balaskas wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I hope something like this could help you :
# Authentication Cache
auth_cache_size = 10240
auth_cache_ttl = 18000
This may have done the trick! The number of imap-login processes has been
holding around 60 all morning.
So I guess most of the login processes were just waiting for
authentication processes to respond? I hadn't thought about that. How
many dovecot-auth processes do you have? Increasing
auth_worker_max_count might be a good idea.
Or it should have actually logged an error in that case: Couldn't
create new auth worker.
Maybe it's the passwd lookups that are slow? What NSS backend are you
using? It's probably a good idea to make them use auth workers also:
userdb passwd {
args = blocking=yes
}
Right now (which is very low usage) there's only one dovecot-auth process
and I remember looking at this at the time and not being worried about the
number of auth processes, or I probably would have changed the default
from 30.. Also, I never saw anything in the logs. I turned on
auth_verbose = yes but not auth_debug.
We're using a rather old NIS server at the moment, planning to move to
LDAP this summer.
Dovecot has been running smoothly this week since I turned on the
auth_cache stuff, at times going up around 110 imap-login processes but
never going higher. It's always responsive which makes the users happy.
Thanks again for the great product!
Re: [Dovecot] imap-login processes
On Fri, 16 May 2008, Evaggelos Balaskas wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
I hope something like this could help you :
# Authentication Cache
auth_cache_size = 10240
auth_cache_ttl = 18000
This may have done the trick! The number of imap-login processes has been
holding around 60 all morning. In case there is a problem in the future I
tested the following script to truss the imap-login processes on Solaris:
for pid in `ps -ef | grep imap-login | grep -v grep | awk '{print $2}'`;
do
truss -o log.$pid -p $pid
done
sleep 20
kill `ps -ef | grep truss | grep -v grep | awk '{print $2}'`
During testing I didn't see anything out of the ordinary, mostly just
sleeping processes.
Thanks for the help everyone!
Bryan Polk
Unix Systems Administrator
Communication and Multimedia Services
FAMU-FSU College of Engineering
[EMAIL PROTECTED]
Re: [Dovecot] imap-login processes
On Fri, 2008-05-16 at 14:47 -0400, Bryan Polk wrote:
It would help to know what these extra processes are doing.
Unfortunately
there's no simple way to do that.. Maybe writing a script that trusses
the
processes for a few seconds and then seeing what it shows?
To truss each imap-login I would need to write the script to execute
truss imap-login and put that in place of imap-login in the config file?
That would probably work too, but I was thinking about getting a few
second snapshot using truss -p. So something like (untested):
for pid in `ps -ef|grep imap$|awk '{print $2}'`; do
truss -o log.$pid -p $pid
done
sleep 5
killall truss
signature.asc
Description: This is a digitally signed message part
[Dovecot] imap-login processes
First, thanks for this great product. We migrated from Courier last week with only small bumps along the way. We have a few hundred active users with a mix of pop3 and imap clients. The server is running on sparc Solaris 9. We're seeing a problem now with the number of imap-login processes rising slowly to it's limit and then (understandably) not letting any more logins happen. Legitimate users seem to have no problem logging in until the limit is hit. Another odd thing is that this is not a problem for pop3-login processes, just imap... I've tried changing some obvious things including the passdb from pam to passwd and back again, but here's the current dovecot -n output: # 1.0.13: /opt/dovecot/etc/dovecot.conf protocols: imap imaps pop3 pop3s ssl_cert_file: /opt/dovecot/etc/ssl/certs/dovecot.pem ssl_key_file: /opt/dovecot/etc/ssl/certs/key.pem disable_plaintext_auth: no login_dir: /opt/dovecot/var/run/dovecot/login login_executable(default): /opt/dovecot/libexec/dovecot/imap-login login_executable(imap): /opt/dovecot/libexec/dovecot/imap-login login_executable(pop3): /opt/dovecot/libexec/dovecot/pop3-login login_processes_count: 12 login_max_processes_count: 512 verbose_proctitle: yes mail_location: maildir:~/Maildir mmap_disable: yes mail_executable(default): /opt/dovecot/libexec/dovecot/imap mail_executable(imap): /opt/dovecot/libexec/dovecot/imap mail_executable(pop3): /opt/dovecot/libexec/dovecot/pop3 mail_plugin_dir(default): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(imap): /opt/dovecot/lib/dovecot/imap mail_plugin_dir(pop3): /opt/dovecot/lib/dovecot/pop3 pop3_uidl_format(default): pop3_uidl_format(imap): pop3_uidl_format(pop3): %08Xu%08Xv namespace: type: private prefix: INBOX. inbox: yes auth default: verbose: yes passdb: driver: pam userdb: driver: passwd Any ideas or advice? Thanks, Bryan Polk Unix Systems Administrator Communication and Multimedia Services FAMU-FSU College of Engineering [EMAIL PROTECTED]
Re: [Dovecot] imap-login processes
On Fri, 2008-05-16 at 11:38 -0400, Bryan Polk wrote: First, thanks for this great product. We migrated from Courier last week with only small bumps along the way. We have a few hundred active users with a mix of pop3 and imap clients. The server is running on sparc Solaris 9. We're seeing a problem now with the number of imap-login processes rising slowly to it's limit and then (understandably) not letting any more logins happen. Legitimate users seem to have no problem logging in until the limit is hit. Another odd thing is that this is not a problem for pop3-login processes, just imap... How many imap processes do you have at that time? Each SSL connection uses up one imap-login process. One sure way to fix this would be to change to high-performance mode as described by http://wiki.dovecot.org/LoginProcess signature.asc Description: This is a digitally signed message part
Re: [Dovecot] imap-login processes
On Fri, 16 May 2008, Timo Sirainen wrote: How many imap processes do you have at that time? Each SSL connection uses up one imap-login process. One sure way to fix this would be to change to high-performance mode as described by http://wiki.dovecot.org/LoginProcess Currently 74 imap processes, 336 imap-login processes. I tried switching to high-performance mode yesterday.. It seemed to end up with sort of the same results, though harder to diagnose. Each process I guess ended up hitting the login_process_size = 64 max and then stopped responding. So imap would work for some people and not others. I'd rather have it be broken for all or none, so I switched it back instead of tweaking the performance options. I think there's something in our environment causing this but I have no idea what.. -bryan
Re: [Dovecot] imap-login processes
On May 16, 2008, at 6:56 PM, Bryan Polk wrote: On Fri, 16 May 2008, Timo Sirainen wrote: How many imap processes do you have at that time? Each SSL connection uses up one imap-login process. One sure way to fix this would be to change to high-performance mode as described by http://wiki.dovecot.org/LoginProcess Currently 74 imap processes, 336 imap-login processes. It would help to know what these extra processes are doing. Unfortunately there's no simple way to do that.. Maybe writing a script that trusses the processes for a few seconds and then seeing what it shows? Another way would be to try if LINUX_PROCTITLE_HACK works also with Solaris. You can remove the comments around #define from src/lib/ process-title.c and then set verbose_proctitle=yes. This should first be tested though, because if it doesn't work it could break badly. If it does work, you should at least see IP addresses for each process that has a connected client and also TLS if SSL/TLS is being used. It could probably also include username for SSL proxies. I tried switching to high-performance mode yesterday.. It seemed to end up with sort of the same results, though harder to diagnose. Each process I guess ended up hitting the login_process_size = 64 max and then stopped responding. If it hits that limit, it gets killed by signal 9 by the kernel (and that gets logged). It's also a good idea to then increase it to something like 256. But I can't see why it would stop responding then. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] imap-login processes
I tried switching to high-performance mode yesterday.. It seemed to end up with sort of the same results, though harder to diagnose. Each process I guess ended up hitting the login_process_size = 64 max and then stopped responding. We had this same problem, and switching to high-performance mode helps. However, finding out which user or application is logging in, and controlling the end-user is the only way to fix it. In our case it was a multi-threaded application that used IMAP (excessively). Monitoring the maillog should help point you in the right direction. Dozens of these per minute should throw up a red-flag: May 16 17:36:16 hostname imap-login: Login: user [:::IP_Address] Joe
Re: [Dovecot] imap-login processes
On May 16, 2008, at 8:39 PM, Joe Allesi wrote: I tried switching to high-performance mode yesterday.. It seemed to end up with sort of the same results, though harder to diagnose. Each process I guess ended up hitting the login_process_size = 64 max and then stopped responding. We had this same problem, and switching to high-performance mode helps. However, finding out which user or application is logging in, and controlling the end-user is the only way to fix it. In our case it was a multi-threaded application that used IMAP (excessively). Monitoring the maillog should help point you in the right direction. Dozens of these per minute should throw up a red-flag: May 16 17:36:16 hostname imap-login: Login: user [:::IP_Address] v1.1 probably helps with this, since it limits the number of simultaneous connections from a user+IP combination. PGP.sig Description: This is a digitally signed message part
Re: [Dovecot] imap-login processes
On Fri, 16 May 2008, Joe Allesi wrote: We had this same problem, and switching to high-performance mode helps. However, finding out which user or application is logging in, and controlling the end-user is the only way to fix it. In our case it was a multi-threaded application that used IMAP (excessively). Monitoring the maillog should help point you in the right direction. Dozens of these per minute should throw up a red-flag: Looking back through the log for today we only have about 7-35 imap-logins happening per minute, from an assortment of users. There doesn't appear to be one user that's doing more than others. One thing I did notice was entries like this: imap-login: Login: user=faizalmi, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Is there a reason the rip/lip would say 127.0.0.1 for some small number of users and not others? It would help to know what these extra processes are doing. Unfortunately there's no simple way to do that.. Maybe writing a script that trusses the processes for a few seconds and then seeing what it shows? To truss each imap-login I would need to write the script to execute truss imap-login and put that in place of imap-login in the config file? I think I might try the source code re-compile first.. -bryan
Re: [Dovecot] imap-login processes
On Fri, 16 May 2008 14:47:05 -0400 (EDT) Bryan Polk wrote: On Fri, 16 May 2008, Joe Allesi wrote: We had this same problem, and switching to high-performance mode helps. However, finding out which user or application is logging in, and controlling the end-user is the only way to fix it. In our case it was a multi-threaded application that used IMAP (excessively). Monitoring the maillog should help point you in the right direction. Dozens of these per minute should throw up a red-flag: Looking back through the log for today we only have about 7-35 imap-logins happening per minute, from an assortment of users. There doesn't appear to be one user that's doing more than others. One thing I did notice was entries like this: imap-login: Login: user=faizalmi, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, secured Is there a reason the rip/lip would say 127.0.0.1 for some small number of users and not others? Do you have a Web-Interface like SquirrelMail or Horde on the same host? Our SquirrelMail shows up in this fashion. --Frank Elsner
Re: [Dovecot] imap-login processes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I hope something like this could help you : # Authentication Cache auth_cache_size = 10240 auth_cache_ttl = 18000 Bryan Polk wrote: On Fri, 16 May 2008, Frank Elsner wrote: Do you have a Web-Interface like SquirrelMail or Horde on the same host? Our SquirrelMail shows up in this fashion. Oh, yeah that would be it. Thanks :) Evaggelos Balaskas - http://ebalaskas.gr Unix System Engineer Informatics Engineer Technological Education -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFILfJ0WIK+Pe9twhoRAtP4AJ9tH5h7M+kPPCDQFgt3wiF0lvGBPwCgz3gj WFU4bDg/Y0SPWlBQ1Md78aI= =BcMN -END PGP SIGNATURE-
