Re: concerning dovecot settings for high volume server

[Tony Morehen]

You may be running up against Linux system/user limits.  Run
$ cat /proc/sys/kernel/pid_max
and
$ ulimit -a
That should give some insight into your problem.

On 12/09/2015 2:53 PM, Rajesh M wrote:

hi

centos 6 64 bit

hex core processor with hyperthreading ie display shows 12 cores
16 gb ram
600 gb 15000 rpm drive

we are having around 4000 users on a server


i wish to allow 1500 pop3 and 1500 imap connections simultaneously.

need help regarding the settings to handle the above

imap-login, pop3-login
imap pop3 service settings

i recently i got an error
imap-login: Error: read(imap) failed: Remote closed connection (process_limit 
reached?)


my current dovecot config file

# 2.2.7: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-431.23.3.el6.x86_64 x86_64 CentOS release 6.5 (Final)
auth_cache_negative_ttl = 0
auth_cache_ttl = 0
auth_mechanisms = plain login digest-md5 cram-md5
default_login_user = vpopmail
disable_plaintext_auth = no
first_valid_gid = 89
first_valid_uid = 89
log_path = /var/log/dovecot.log
login_greeting = ready.
mail_max_userip_connections = 50
mail_plugins = " quota"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date ihave
namespace {
   inbox = yes
   location =
   prefix =
   separator = .
   type = private
}
passdb {
   args = cache_key=%u webmail=127.0.0.1
   driver = vpopmail
}
plugin {
   quota = maildir:ignore=Trash
   quota_rule = ?:storage=0
}
protocols = imap pop3
service imap-login {
   client_limit = 256
   process_limit = 400
   process_min_avail = 4
   service_count = 0
   vsz_limit = 512 M
}
service pop3-login {
   client_limit = 1000
   process_limit = 400
   process_min_avail = 12
   service_count = 0
   vsz_limit = 512 M
}
ssl_cert = 

concerning dovecot settings for high volume server

[Rajesh M]

hi

centos 6 64 bit

hex core processor with hyperthreading ie display shows 12 cores
16 gb ram
600 gb 15000 rpm drive

we are having around 4000 users on a server


i wish to allow 1500 pop3 and 1500 imap connections simultaneously.

need help regarding the settings to handle the above

imap-login, pop3-login
imap pop3 service settings

i recently i got an error
imap-login: Error: read(imap) failed: Remote closed connection (process_limit 
reached?)


my current dovecot config file

# 2.2.7: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-431.23.3.el6.x86_64 x86_64 CentOS release 6.5 (Final)
auth_cache_negative_ttl = 0
auth_cache_ttl = 0
auth_mechanisms = plain login digest-md5 cram-md5
default_login_user = vpopmail
disable_plaintext_auth = no
first_valid_gid = 89
first_valid_uid = 89
log_path = /var/log/dovecot.log
login_greeting = ready.
mail_max_userip_connections = 50
mail_plugins = " quota"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date ihave
namespace {
  inbox = yes
  location =
  prefix =
  separator = .
  type = private
}
passdb {
  args = cache_key=%u webmail=127.0.0.1
  driver = vpopmail
}
plugin {
  quota = maildir:ignore=Trash
  quota_rule = ?:storage=0
}
protocols = imap pop3
service imap-login {
  client_limit = 256
  process_limit = 400
  process_min_avail = 4
  service_count = 0
  vsz_limit = 512 M
}
service pop3-login {
  client_limit = 1000
  process_limit = 400
  process_min_avail = 12
  service_count = 0
  vsz_limit = 512 M
}
ssl_cert = 

Re: concerning dovecot settings for high volume server

[Tony Morehen]

On 9/12/2015 10:51 PM, Rajesh M wrote:

- Original Message -
From: Tony Morehen [mailto:tmore...@ajmconsulting.ca]
To: dovecot@dovecot.org
Sent: Sat, 12 Sep 2015 17:57:27 -0400
Subject: Re: concerning dovecot settings for high volume server

You may be running up against Linux system/user limits.  Run
$ cat /proc/sys/kernel/pid_max
and
$ ulimit -a
That should give some insight into your problem.

On 12/09/2015 2:53 PM, Rajesh M wrote:

hi

centos 6 64 bit

hex core processor with hyperthreading ie display shows 12 cores
16 gb ram
600 gb 15000 rpm drive

we are having around 4000 users on a server


i wish to allow 1500 pop3 and 1500 imap connections simultaneously.

need help regarding the settings to handle the above

imap-login, pop3-login
imap pop3 service settings

i recently i got an error
imap-login: Error: read(imap) failed: Remote closed connection (process_limit 
reached?)


my current dovecot config file

# 2.2.7: /etc/dovecot/dovecot.conf
# OS: Linux 2.6.32-431.23.3.el6.x86_64 x86_64 CentOS release 6.5 (Final)
auth_cache_negative_ttl = 0
auth_cache_ttl = 0
auth_mechanisms = plain login digest-md5 cram-md5
default_login_user = vpopmail
disable_plaintext_auth = no
first_valid_gid = 89
first_valid_uid = 89
log_path = /var/log/dovecot.log
login_greeting = ready.
mail_max_userip_connections = 50
mail_plugins = " quota"
managesieve_notify_capability = mailto
managesieve_sieve_capability = fileinto reject envelope encoded-character 
vacation subaddress comparator-i;ascii-numeric relational regex imap4flags copy 
include variables body enotify environment mailbox date ihave
namespace {
inbox = yes
location =
prefix =
separator = .
type = private
}
passdb {
args = cache_key=%u webmail=127.0.0.1
driver = vpopmail
}
plugin {
quota = maildir:ignore=Trash
quota_rule = ?:storage=0
}
protocols = imap pop3
service imap-login {
client_limit = 256
process_limit = 400
process_min_avail = 4
service_count = 0
vsz_limit = 512 M
}
service pop3-login {
client_limit = 1000
process_limit = 400
process_min_avail = 12
service_count = 0
vsz_limit = 512 M
}
ssl_cert = 


this is the first time i got this message since past over an year.
the error went away as soon as i restarted dovecot..
surprisingly this happened in the night ie off-office hours.

here are the values i got

also could you provide me the settings for the limits if wish to handle around 
1500 simultaneous connections of pop3 and imap each ?

the current values are as follows

[root@ns1 log]# cat /proc/sys/kernel/pid_max
49152

[root@ns1 log]# ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127047
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 127047
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited


thanks
rajesh


I'm no expert but it is my understanding that the binding limit would be 
the lower of pid_max or max user processes ie 49152.  From that you 
would subtract 300 (reserved system pids) and the number of non-dovecot 
processes.


Dovecot itself uses about 6 base processes (/usr/sbin/dovecot -F; 
dovecot/anvil; dovecot/log; dovecot/config; dovecot/auth; 
dovecot/ssl-params) plus one processes for each logged-in pop3 and imap 
mailbox.  That's one process for each pop3 account but potentially 
multiple imap mailboxes per imap account if the imap client is using 
idle to monitor multiple imap mailboxes for changes.


Assuming 10 mailboxes per simultaneous imap account, that would be 15000 
processes, well below max_pid.  I'd say that your problem does not 
appear to arise from system limits.  Perhaps someone else could point 
you in a different direction.

Re: concerning dovecot settings for high volume server

[Rajesh M]

- Original Message -
From: Tony Morehen [mailto:tmore...@ajmconsulting.ca]
To: dovecot@dovecot.org
Sent: Sat, 12 Sep 2015 17:57:27 -0400
Subject: Re: concerning dovecot settings for high volume server

You may be running up against Linux system/user limits.  Run
$ cat /proc/sys/kernel/pid_max
and
$ ulimit -a
That should give some insight into your problem.

On 12/09/2015 2:53 PM, Rajesh M wrote:
> hi
>
> centos 6 64 bit
>
> hex core processor with hyperthreading ie display shows 12 cores
> 16 gb ram
> 600 gb 15000 rpm drive
>
> we are having around 4000 users on a server
>
>
> i wish to allow 1500 pop3 and 1500 imap connections simultaneously.
>
> need help regarding the settings to handle the above
>
> imap-login, pop3-login
> imap pop3 service settings
>
> i recently i got an error
> imap-login: Error: read(imap) failed: Remote closed connection (process_limit 
> reached?)
>
>
> my current dovecot config file
>
> # 2.2.7: /etc/dovecot/dovecot.conf
> # OS: Linux 2.6.32-431.23.3.el6.x86_64 x86_64 CentOS release 6.5 (Final)
> auth_cache_negative_ttl = 0
> auth_cache_ttl = 0
> auth_mechanisms = plain login digest-md5 cram-md5
> default_login_user = vpopmail
> disable_plaintext_auth = no
> first_valid_gid = 89
> first_valid_uid = 89
> log_path = /var/log/dovecot.log
> login_greeting = ready.
> mail_max_userip_connections = 50
> mail_plugins = " quota"
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope encoded-character 
> vacation subaddress comparator-i;ascii-numeric relational regex imap4flags 
> copy include variables body enotify environment mailbox date ihave
> namespace {
>inbox = yes
>location =
>prefix =
>separator = .
>type = private
> }
> passdb {
>args = cache_key=%u webmail=127.0.0.1
>driver = vpopmail
> }
> plugin {
>quota = maildir:ignore=Trash
>quota_rule = ?:storage=0
> }
> protocols = imap pop3
> service imap-login {
>client_limit = 256
>process_limit = 400
>process_min_avail = 4
>service_count = 0
>vsz_limit = 512 M
> }
> service pop3-login {
>client_limit = 1000
>process_limit = 400
>process_min_avail = 12
>service_count = 0
>vsz_limit = 512 M
> }
> ssl_cert =  ssl_dh_parameters_length = 2048
> ssl_key =  userdb {
>args = cache_key=%u quota_template=quota_rule=*:backend=%q
>driver = vpopmail
> }
> protocol imap {
>imap_client_workarounds = delay-newmail
>mail_plugins = " quota imap_quota"
> }
> protocol pop3 {
>pop3_client_workarounds = outlook-no-nuls oe-ns-eoh
>pop3_fast_size_lookups = yes
>pop3_lock_session = no
>pop3_no_flag_updates = yes
> }
>
>
> thanks very much,
>
> rajesh



this is the first time i got this message since past over an year.
the error went away as soon as i restarted dovecot..
surprisingly this happened in the night ie off-office hours.

here are the values i got

also could you provide me the settings for the limits if wish to handle around 
1500 simultaneous connections of pop3 and imap each ?

the current values are as follows

[root@ns1 log]# cat /proc/sys/kernel/pid_max
49152

[root@ns1 log]# ulimit -a
core file size  (blocks, -c) 0
data seg size   (kbytes, -d) unlimited
scheduling priority (-e) 0
file size   (blocks, -f) unlimited
pending signals (-i) 127047
max locked memory   (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files  (-n) 1024
pipe size(512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority  (-r) 0
stack size  (kbytes, -s) 10240
cpu time   (seconds, -t) unlimited
max user processes  (-u) 127047
virtual memory  (kbytes, -v) unlimited
file locks  (-x) unlimited


thanks
rajesh

Re: How to "Windows Authenticate"

[Mark Foley]

I am running Dovecot 2.2.15 on Linux Slackware 14.1 and Samba 4.1.17 as the
Active Directory/Domain Controller on the same host as Dovecot.
Sendmail/procmail delivers mail to users' $HOME/Maildir. MS Outlook/IMAP is the
client MTU used to connect with Dovecot to read mail on the Users' WIN7
workstations.

I believe I have confirmed that MS Outlook will either ...

1) send the userid and password configured in the Outlook settings to Dovecot
for authorizing. This mechanism has been working fine for months.

or ...

2) Use NTML authorization if "Require login using Secure Password Authentication
(SPA)" is checked: https://en.wikipedia.org/wiki/Secure_Password_Authentication

Those, I believe, are the only two choices with Outlook (other than Exchange). 
Therefore, in order not to configure a Domain-distinct password in Outlook, I
need to use the NTLM auth_mechanism for AD "Windows Authentication" with
Dovecot.  I've tried the settings below (just trying one user at the moment):

$ doveconf -n
# 2.2.15: /usr/local/etc/dovecot/dovecot.conf
# OS: Linux 3.10.17 x86_64 Slackware 14.1
auth_debug_passwords = yes
auth_mechanisms = plain ntlm
auth_use_winbind = yes
auth_verbose = yes
auth_verbose_passwords = plain
disable_plaintext_auth = no
info_log_path = /var/log/dovecot_info
mail_location = maildir:~/Maildir
protocols = imap
ssl_cert = , rip=192.168.0.58, 
lip=98.102.63.107, session=<2PnkuZkfqADAqAA6>

Can someone tell me what this means and how to fix it?

Note that I have read http://wiki2.dovecot.org/HowTo/ActiveDirectoryNtlm over 
and
over, so simply referring me to that link will not help.

Thanks, Mark

Re: Need help on checkpassword userdb/passdb

[Mark Foley]

I figured out how to make checkpassword work. There is a problem with the
documentation. http://wiki2.dovecot.org/AuthDatabase/CheckPassword, under
'Security' says, "a. If possible, change the checkpassword to return userdb_uid
and userdb_gid extra fields instead of using setuid() and setgid(). This also
improves the performance." And, under 'Checkpassword Interface' it says,
"Return the user's UNIX UID and GID using userdb_uid and userdb_gid
environments and add them to the EXTRA environment ..."

I did all of this and it didn't work. However, when I added the userdb_home
environment variable and added that to the EXTRA environment variable, it
worked. I tried this because I happened upon
http://wiki2.dovecot.org/UserDatabase/Prefetch which mentioned userdb_home. The
http://wiki2.dovecot.org/AuthDatabase/CheckPassword needs to have this bit of
information added in the appropriate place(s) or the developer/hackster will
waste days trying to get checkpassword working until he/she stumbles across the
userdb_home comment elsewhere.

Nevertheless, checkpassword turns out not to be the solution to my original
problem, so I will keep on keepin' on ... 

--Mark

-Original Message-
From: Mark Foley 
Date: Fri, 11 Sep 2015 21:57:40 -0400
To: dovecot@dovecot.org
Subject: Re: Need help on checkpassword userdb/passdb

[grumpy bit deleted]

To follow up on my previous posting in this thread, I'm trying to get
checkpassword to work. I have confirmed that it is setting the environment
variables as described in (http://wiki2.dovecot.org/AuthDatabase/CheckPassword).
My debug output of env variables sent to checkpassword-reply:

$USER=mark
userdb_uid=326
userdb_gid=100
INSECURE_SETUID=1
EXTRA=userdb_uid userdb_gid

I have confirmed that my checkpassword program returns 0 authenticating the user
with the AD:

fork pid = 4239, ntlm_auth status: 0

The pid listed above is the pid of the forked 
/usr/local/libexec/dovecot/checkpassword-reply 
program. For testing purposes, I've replaced that with a stub of my own that
shows the set environment variables so I know checkpassword-reply is getting
them (listed above).

Notice in the log messages below that everything looks correct. It has the
correct username, UID, GID, client passdb out: OK. No error in the log that I
can see.

I believe I've done everything exactly as documented in the wiki, but it doesn't
work I get the Outlook message "Your IMAP server closed the connection ... 
Error Code: 0x800CCCDD".  Finally, I tried setting:

chgrp dovecot /usr/local/libexec/dovecot/checkpassword-reply
chmod g+s /usr/local/libexec/dovecot/checkpassword-reply

As the wiki suggested and setting the env variable INSECURE_SETUID=1. Same
error. 

Googling the 0x800CCCDD code simply says to turn of scheduled send/received, but
that makes no different. Same error.

I believe I've done everything exactly according to the documentation.  Does
checkpassword actually work with Dovecot version 2.2.15? If not, could someone
please tell me so I can stop wasting my time.  If it does work, can someone
please help me figure out why it does not for me?

Thanks -- Mark

My dovecot log:

Sep 11 21:18:22 auth: Debug: Loading modules from directory: 
/usr/local/lib/dovecot/auth
Sep 11 21:18:22 auth: Debug: Read auth token secret from 
/usr/local/var/run/dovecot/auth-token-secret.dat
Sep 11 21:18:22 auth: Debug: auth client connected (pid=4234)
Sep 11 21:18:22 auth: Debug: client in: AUTH1   PLAIN   service=imap
session=tHPCm4IftgDAqAA6  lip=192.168.0.2  rip=192.168.0.58
lport=143   rport=50614 resp=AG1hcmsAZ2xhY29uXzk= (previous base64 data 
may contain sensitive data)
Sep 11 21:18:22 auth: Debug: 
checkpassword(mark,192.168.0.58,): execute: 
/user/util/bin/checkpassword /usr/local/libexec/dovecot/checkpassword-reply
Sep 11 21:18:22 auth: Debug: 
checkpassword(mark,192.168.0.58,): exit_status=0
Sep 11 21:18:22 auth: Debug: 
checkpassword(mark,192.168.0.58,): Received input: 
userdb_uid=326   userdb_gid=100
Sep 11 21:18:22 auth: Debug: client passdb out: OK  1   user=mark
Sep 11 21:18:22 auth: Debug: master in: REQUEST 1794375681  42341   
c2551b70ccf5e2f8e022869663bf6a70   session_pid=4240
request_auth_token
Sep 11 21:18:22 auth: Debug: prefetch(mark,192.168.0.58,): 
success
Sep 11 21:18:22 auth: Debug: master userdb out: USER1794375681  mark
uid=326 gid=100 auth_token=008ebf0ebd9c1654085de247f10cdf0a746555d4
Sep 11 21:18:22 imap-login: Info: Login: user=, method=PLAIN, 
rip=192.168.0.58, lip=192.168.0.2, mpid=4240, session=


-Original Message-
From: Mark Foley 
Date: Thu, 10 Sep 2015 23:05:18 -0400
To: dovecot@dovecot.org
Subject: Need help on checkpassword userdb/passdb

I'm experimenting with checkpassword as an auth method for usedb and passdb 
(http://wiki2.dovecot.org/AuthDatabase/CheckPassword). I've set up the userdb
and passdb *exactly* as the wiki suggests 

Re: My dovecot works fine against Active Directory 2003, but not against AD2008

[Mark Foley]

Fran - thanks for your reply. I'm cc'ing you directly on this as well as posting
to the list as I'm not sure how often you check the list and I'm down to hanging
by my last fingernail on this project.

I have some preliminary questions interspersed below.

Thanks, --Mark

-Original Message-
> Subject: Re: My dovecot works fine against Active Directory 2003, but not
>   against AD2008
> To: dovecot@dovecot.org
> From: Fran 
> Date: Thu, 10 Sep 2015 13:26:21 +0200
>
> Hi Mark,
>
> when I say AD 2003/8 I mean Active Directory 2003/8.

Hmmm, I've not heard of "Active Directory 2003" or 2008.  The year numbers
indicated to me you might be talking about Windows Small Business Server 2003 or
2008.  Is your AD Server Windows? Linux? Something else? I'm using Samba4 AD/DC
on Linux. 

>
> My configuration is attached.

Thank you very much for that. If I make some headway, I'll likely have more
questions on specifics.

>
> I based my installation (dovecot+postfix) in the guides of this site:
> http://www.linuxmail.info
>
> The LDAP part is this:
> http://www.linuxmail.info/postfix-dovecot-ldap-centos-5/

If you were able to make sense out of these sites' tiny screen-shots and 
one-line
descriptions my hat's off to you. "Your a better man that I am Gunga-Din!" If
there was more detailed narrative somewhere I couldn't find it. Also, I don't
have jXplorer on my system, so probably I couldn't get too far anyway.

BIG QUESTIONS:

1. Are you using MS Outlook IMAP clients in your environment? If so, how are you
making them connect with LDAP? By checking the SPA checkbox?

2.  The mail_gid/mail_uid as vmail confuses me.  I see that setting a lot,
including in your config.  http://wiki2.dovecot.org/VirtualUsers says, "You can
create, for example, one vmail user which owns all the mails, or you can assign
a separate UID for each user." I have assigned a separte UID for each based on
the UID returned by `wbinfo -u `.  Does assigning separate UIDs mess
up my ability to adapt your configuration?

little questions:

3. I'm not planning on using quotas. Can I safely omit your mail_plugins = " 
quota"
setting and all your plugin { quota_...} settings? I want to be as simple as
possible to start.

4. Likewise, dovecot seems to be able to find users' mailboxes just fine. Can I
omit the namespace inbox {} setting?

These may seem like amaturish questions, but little details have foiled me a lot
on this Dovecot project. 

If I feel confident with the answers you provide here, I'll move on to trying
some things.

Thanks a lot for your help!!!

--Mark

>
> You can also use PAM to connect to AD
> (http://www.linuxmail.info/active-directory-dovecot-pam-authentication/)
> but that way doesn't allow to retrieve custom fields from the AD (ex. a
> field to set quota per user), so I'm using the standard LDAP method.
>
> Regards
>
> El 10/09/2015 a las 4:51, Mark Foley escribió:
> > Fran and/or Matthias,
> >
> > Could you publish your doveconf -n? I can't get dovecot to authenticate 
> > with my
> > AD. Maybe you have a solution I could try.
> >
> > What mail client(s) are you using? I assume by "AD 2003/8" You mean 
> > SBS2003/8
> > and are therefore using Outlook?
> >
> > --Mark
> >
> > -Original Message-
[deleted]