Re: maildirfolder file created in maildir root during auto-creation with 2.3.4.1 but not 2.2.27
Hello,
thanks for the reply.
On Thu, 2 Sep 2021 12:47:43 +0300 (EEST) Aki Tuomi wrote:
> Would it be possible to workaround this with:
>
> mail_location = maildir:~/Mail/
>
Maybe, but that is not feasible in our deployment, which is LDAP driven
and thus looks like this:
mail_location = maildir:%h
Changing this in-situ by attaching a "/Mail/" to the location for literally
hundreds of thousands mailboxes clearly is a no-go, nor would I look
forward to go fix up all the other places and scripts that assume a
certain directory structure.
Regards,
Christian
> Aki
>
> > On 02/09/2021 11:21 Christian Balzer wrote:
> >
> >
> > Hello,
> >
> > it is now nearly 2 years later and we are running 2.3.13 with this bug
> > still present.
> > Would be nice if it were acknowledged at least if not even fixed.
> > And it was confirmed by other people who contacted me directly after
> > seeing the original report here.
> >
> > Regards,
> >
> > Christian
> >
> > On Wed, 5 Feb 2020 16:13:37 +0900 Christian Balzer wrote:
> >
> > > Hello,
> > >
> > > On Wed, 5 Feb 2020 08:58:29 +0200 Aki Tuomi wrote:
> > >
> > > > Can you provide full doveconf -n output? Also how are you delivering
> > > > mail?
> > > >
> > > As pretty much implied, Exim is delivering mails, w/o problems.
> > > And if it gets to create the home directory, everything is fine
> > > and maildirsize gets put there.
> > >
> > > But if the first access is via the newer dovecot the bogus maildirfolder
> > > file gets created in the home directory and prevents Exim (and itself?)
> > > from putting a maildirsize there.
> > >
> > > My bet is that that something in the auto-create logic changed or the
> > > "mail_home" needing to be set explicitly instead of defaulting to
> > > mail_location if unset, etc.
> > >
> > > Redacted and relevant parts only:
> > > ---
> > > # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf
> > > # Pigeonhole version 0.5.4 ()
> > > # OS: Linux 4.19.0-6-amd64 x86_64 Debian 10.2
> > > # Hostname: testbox.gol.com
> > > auth_default_realm = gol.com
> > > default_client_limit = 16384
> > > default_process_limit = 1024
> > > first_valid_uid = 8
> > > imap_hibernate_timeout = 30 secs
> > > imap_idle_notify_interval = 8 mins
> > > imap_logout_format = in=%i out=%o head=<%{fetch_hdr_count}>
> > > del=<%{deleted}> exp=<%{expunged}> trash=<%{trashed}> session=<%{session}>
> > > login_trusted_networks = some.net.work
> > > mail_gid = 8
> > > mail_location = maildir:%h
> > > mail_privileged_group = mail
> > > mail_uid = 8
> > > mailbox_idle_check_interval = 1 mins
> > > maildir_very_dirty_syncs = yes
> > >
> > > passdb {
> > > args = /etc/dovecot/dovecot-ldap.conf.ext
> > > driver = ldap
> > > }
> > > plugin {
> > > quota = maildir:User
> > > quota_rule = ?:storage=200M
> > > quota_rule2 = Trash:storage=+50M
> > > sieve = file:~/sieve;active=~/.dovecot.sieve
> > > }
> > >
> > > userdb {
> > > args = /etc/dovecot/dovecot-ldap.conf.ext
> > > driver = ldap
> > > }
> > > verbose_proctitle = yes
> > > protocol imap {
> > > mail_max_userip_connections = 40
> > > mail_plugins = quota imap_quota
> > > }
> > > protocol pop3 {
> > > mail_plugins = quota
> > > }
> > > ---
> > >
> > > Regards,
> > >
> > > Christian
> > > > Aki
> > > >
> > > > On 5.2.2020 4.24, Christian Balzer wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > as the tin says.
> > > > > I have several servers running 2.2.27 (Debian stretch) and am adding
> > > > > new
> > > > > ones with 2.3.4.1 (Debian buster).
> > > > > The configs were upgraded where needed but neither 10-mail.conf nor
> > > > > 15-mailboxes.conf were changed.
> > > > > 15-mailboxes is all commented out (I guess the default is auto-create,
> > > > > which isn't documented anywhere I could find) and the only
> > > > > non-comments in
> > > > > 10-mail.conf are
> > > > > ---
> > > > > mail_location = maildir:%h
> > > > > mail_privileged_group = mail
> > > > > ---
> > > > >
> > > > > So yes, no namespaces are explicitly defined/declared.
> > > > >
> > > > >
> > > > > The 2.3.4.1 version wrongly creates a maildirfolder file in the home
> > > > > directory (maildir root), preventing exim from correctly
> > > > > creating/using
> > > > > maildirsize.
> > > > >
> > > > > a) Is this expected behavior and can it be changed?
> > > > > b) How can I disable inbox auto-creation if a) doesn't pan out?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Christian
> > > >
> > >
> > >
> > > --
> > > Christian BalzerNetwork/Systems Engineer
> > > ch...@gol.com Rakuten Mobile Inc.
> > >
> >
> >
> > --
> > Christian BalzerNetwork/Systems Engineer
> > ch...@gol.com Rakuten Communications
>
--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com Rakuten Communications
Re: maildirfolder file created in maildir root during auto-creation with 2.3.4.1 but not 2.2.27
Hello,
it is now nearly 2 years later and we are running 2.3.13 with this bug
still present.
Would be nice if it were acknowledged at least if not even fixed.
And it was confirmed by other people who contacted me directly after
seeing the original report here.
Regards,
Christian
On Wed, 5 Feb 2020 16:13:37 +0900 Christian Balzer wrote:
> Hello,
>
> On Wed, 5 Feb 2020 08:58:29 +0200 Aki Tuomi wrote:
>
> > Can you provide full doveconf -n output? Also how are you delivering mail?
> >
> As pretty much implied, Exim is delivering mails, w/o problems.
> And if it gets to create the home directory, everything is fine
> and maildirsize gets put there.
>
> But if the first access is via the newer dovecot the bogus maildirfolder
> file gets created in the home directory and prevents Exim (and itself?)
> from putting a maildirsize there.
>
> My bet is that that something in the auto-create logic changed or the
> "mail_home" needing to be set explicitly instead of defaulting to
> mail_location if unset, etc.
>
> Redacted and relevant parts only:
> ---
> # 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.5.4 ()
> # OS: Linux 4.19.0-6-amd64 x86_64 Debian 10.2
> # Hostname: testbox.gol.com
> auth_default_realm = gol.com
> default_client_limit = 16384
> default_process_limit = 1024
> first_valid_uid = 8
> imap_hibernate_timeout = 30 secs
> imap_idle_notify_interval = 8 mins
> imap_logout_format = in=%i out=%o head=<%{fetch_hdr_count}> del=<%{deleted}>
> exp=<%{expunged}> trash=<%{trashed}> session=<%{session}>
> login_trusted_networks = some.net.work
> mail_gid = 8
> mail_location = maildir:%h
> mail_privileged_group = mail
> mail_uid = 8
> mailbox_idle_check_interval = 1 mins
> maildir_very_dirty_syncs = yes
>
> passdb {
> args = /etc/dovecot/dovecot-ldap.conf.ext
> driver = ldap
> }
> plugin {
> quota = maildir:User
> quota_rule = ?:storage=200M
> quota_rule2 = Trash:storage=+50M
> sieve = file:~/sieve;active=~/.dovecot.sieve
> }
>
> userdb {
> args = /etc/dovecot/dovecot-ldap.conf.ext
> driver = ldap
> }
> verbose_proctitle = yes
> protocol imap {
> mail_max_userip_connections = 40
> mail_plugins = quota imap_quota
> }
> protocol pop3 {
> mail_plugins = quota
> }
> ---
>
> Regards,
>
> Christian
> > Aki
> >
> > On 5.2.2020 4.24, Christian Balzer wrote:
> > >
> > > Hello,
> > >
> > > as the tin says.
> > > I have several servers running 2.2.27 (Debian stretch) and am adding new
> > > ones with 2.3.4.1 (Debian buster).
> > > The configs were upgraded where needed but neither 10-mail.conf nor
> > > 15-mailboxes.conf were changed.
> > > 15-mailboxes is all commented out (I guess the default is auto-create,
> > > which isn't documented anywhere I could find) and the only non-comments in
> > > 10-mail.conf are
> > > ---
> > > mail_location = maildir:%h
> > > mail_privileged_group = mail
> > > ---
> > >
> > > So yes, no namespaces are explicitly defined/declared.
> > >
> > >
> > > The 2.3.4.1 version wrongly creates a maildirfolder file in the home
> > > directory (maildir root), preventing exim from correctly creating/using
> > > maildirsize.
> > >
> > > a) Is this expected behavior and can it be changed?
> > > b) How can I disable inbox auto-creation if a) doesn't pan out?
> > >
> > > Thanks,
> > >
> > > Christian
> >
>
>
> --
> Christian BalzerNetwork/Systems Engineer
> ch...@gol.com Rakuten Mobile Inc.
>
--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com Rakuten Communications
Re: maildirfolder file created in maildir root during auto-creation with 2.3.4.1 but not 2.2.27
Hello,
On Wed, 5 Feb 2020 08:58:29 +0200 Aki Tuomi wrote:
> Can you provide full doveconf -n output? Also how are you delivering mail?
>
As pretty much implied, Exim is delivering mails, w/o problems.
And if it gets to create the home directory, everything is fine
and maildirsize gets put there.
But if the first access is via the newer dovecot the bogus maildirfolder
file gets created in the home directory and prevents Exim (and itself?)
from putting a maildirsize there.
My bet is that that something in the auto-create logic changed or the
"mail_home" needing to be set explicitly instead of defaulting to
mail_location if unset, etc.
Redacted and relevant parts only:
---
# 2.3.4.1 (f79e8e7e4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.4 ()
# OS: Linux 4.19.0-6-amd64 x86_64 Debian 10.2
# Hostname: testbox.gol.com
auth_default_realm = gol.com
default_client_limit = 16384
default_process_limit = 1024
first_valid_uid = 8
imap_hibernate_timeout = 30 secs
imap_idle_notify_interval = 8 mins
imap_logout_format = in=%i out=%o head=<%{fetch_hdr_count}> del=<%{deleted}>
exp=<%{expunged}> trash=<%{trashed}> session=<%{session}>
login_trusted_networks = some.net.work
mail_gid = 8
mail_location = maildir:%h
mail_privileged_group = mail
mail_uid = 8
mailbox_idle_check_interval = 1 mins
maildir_very_dirty_syncs = yes
passdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
plugin {
quota = maildir:User
quota_rule = ?:storage=200M
quota_rule2 = Trash:storage=+50M
sieve = file:~/sieve;active=~/.dovecot.sieve
}
userdb {
args = /etc/dovecot/dovecot-ldap.conf.ext
driver = ldap
}
verbose_proctitle = yes
protocol imap {
mail_max_userip_connections = 40
mail_plugins = quota imap_quota
}
protocol pop3 {
mail_plugins = quota
}
---
Regards,
Christian
> Aki
>
> On 5.2.2020 4.24, Christian Balzer wrote:
> >
> > Hello,
> >
> > as the tin says.
> > I have several servers running 2.2.27 (Debian stretch) and am adding new
> > ones with 2.3.4.1 (Debian buster).
> > The configs were upgraded where needed but neither 10-mail.conf nor
> > 15-mailboxes.conf were changed.
> > 15-mailboxes is all commented out (I guess the default is auto-create,
> > which isn't documented anywhere I could find) and the only non-comments in
> > 10-mail.conf are
> > ---
> > mail_location = maildir:%h
> > mail_privileged_group = mail
> > ---
> >
> > So yes, no namespaces are explicitly defined/declared.
> >
> >
> > The 2.3.4.1 version wrongly creates a maildirfolder file in the home
> > directory (maildir root), preventing exim from correctly creating/using
> > maildirsize.
> >
> > a) Is this expected behavior and can it be changed?
> > b) How can I disable inbox auto-creation if a) doesn't pan out?
> >
> > Thanks,
> >
> > Christian
>
--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com Rakuten Mobile Inc.
maildirfolder file created in maildir root during auto-creation with 2.3.4.1 but not 2.2.27
Hello, as the tin says. I have several servers running 2.2.27 (Debian stretch) and am adding new ones with 2.3.4.1 (Debian buster). The configs were upgraded where needed but neither 10-mail.conf nor 15-mailboxes.conf were changed. 15-mailboxes is all commented out (I guess the default is auto-create, which isn't documented anywhere I could find) and the only non-comments in 10-mail.conf are --- mail_location = maildir:%h mail_privileged_group = mail --- So yes, no namespaces are explicitly defined/declared. The 2.3.4.1 version wrongly creates a maildirfolder file in the home directory (maildir root), preventing exim from correctly creating/using maildirsize. a) Is this expected behavior and can it be changed? b) How can I disable inbox auto-creation if a) doesn't pan out? Thanks, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Rakuten Mobile Inc.
Re: CVE-2019-11500: Critical vulnerability in Dovecot and Pigeonhole
Hello, Debian Stretch impact free security upgrade. This is for the default version/unit which has a Type=forking service, not the backport and buster one which switched to simple and has dovecot running in the foreground. However it should work just the same, obviously try this on a test machine. Create /etc/systemd/system/dovecot.service.d/override.conf: --- [Service] ExecStart= ExecStart=/bin/true ExecStop= ExecStop=/bin/true KillMode=none --- Make it active with: systemctl daemon-reload Do a "systemctl status dovecot.service" and note the PID, the "Main PID: pidnum (dovecot)" one from the status output. A stop action will have systemd delete the pid file (why, oh why?) and thus during the course of an upgrade we will have to re-establish it in another session with: echo pidnum >/var/run/dovecot/master.pid I had to do this 3 times, every time there was a setting up or processing triggers for dovecot-core. You can see it by things stalling out or watching "systemctl status dovecot.service" which will have it "starting". Since no daemons actually get stopped or started this is harmless, as in there won't be any impact to existing or new connections. And after the upgrade is finished a: systemctl status dovecot.service should have the status back to active. At this point any new IMAP sessions will be using the new code, but imap-login will NOT. A "systemctl reload dovecot.service" will have the master spawn off new daemons while NOT killing off the old ones, so existing proxy connections won't be killed either. Afterwards you will likely want to comment out the "Exec" lines and do another "systemctl daemon-reload", to re-establish normal behavior. The Killmode bit depends on if you want the dovecot option "shutdown_clients = no" to work as expected. That is, on stretch backports dovecot or buster, with the default stretch version a "systemctl restart" will still kill off existing IMAP sessions for reasons that are currently beyond me. On a stretch-backport install we get initially: --- ps faxv|grep dove 3342 ?Ss 0:00 078 18449 3164 0.0 /usr/sbin/dovecot -F 3344 ?S 0:00 020 23471 3460 0.0 \_ dovecot/pop3-login 3345 ?S 0:00 030 23469 3740 0.0 \_ dovecot/imap-login 3346 ?S 0:00 016 9867 1096 0.0 \_ dovecot/anvil [17 connections] 3347 ?S 0:00 017 9998 2668 0.0 \_ dovecot/log [etc] 3362 ?S 0:00 0 117 26438 4452 0.1 \_ dovecot/config 3363 ?S 0:00 019 10120 2988 0.0 \_ dovecot/stats [19 connections] 3365 ?S 0:00 0 356 87703 8032 0.1 \_ dovecot/auth [0 wait, 0 passdb, 0 userdb] 3408 ?S 0:00 025 9866 1108 0.0 \_ dovecot/imap-hibernate [0 connections] 4690 ?S 0:00 0 235 26372 5312 0.1 \_ dovecot/imap [t...@goltest.com 203.216.99.99 IDLE] --- and after a restart as expected: --- 3347 ?S 0:00 017 9998 2668 0.0 dovecot/log 3363 ?S 0:00 019 10120 2988 0.0 dovecot/stats [1 connections] 4690 ?S 0:00 0 235 26372 5312 0.1 \_ dovecot/imap [t...@goltest.com 203.216.99.99 IDLE] 4701 ?Ss 0:00 078 18449 3096 0.0 /usr/sbin/dovecot -F 4704 ?S 0:00 020 23471 3460 0.0 \_ dovecot/pop3-login 4705 ?S 0:00 030 23469 3468 0.0 \_ dovecot/imap-login 4706 ?S 0:00 016 9867 1096 0.0 \_ dovecot/anvil [17 connections] 4707 ?S 0:00 017 9998 2424 0.0 \_ dovecot/log [etc] 4722 ?S 0:00 0 117 26306 4448 0.1 \_ dovecot/config 4723 ?S 0:00 019 9996 2540 0.0 \_ dovecot/stats [17 connections] 4725 ?S 0:00 0 356 87703 8048 0.1 \_ dovecot/auth [0 wait, 0 passdb, 0 userdb] --- Regards, Christian On Sat, 31 Aug 2019 11:30:12 +0900 Christian Balzer via dovecot wrote: > Daniel, > > thanks so much for the detailed pointers. > > So it turns out to be both the evil that is systemd and an overzealous > upgrade script. > > Apollon, should I raise a Debian bug for this? > > As for reasons, how do 50k proxy session on the proxy servers and 25k imap > processes on the mailbox servers sound? > > Even on a server with just 6k users and 7k imap processes that causes a > massive load spike and a far longer service interruption (about 50 > seconds) than I'm happy with. > > Penultimately if people do set "shutdown_clients = no" they hopefully know > what they are doing and do expect that to work. > > Regards, > > Christian > > On Fri, 30 Aug 2019 17:44:23 +0200 Daniel Lange via dovecot wrote: > > > Am 30.08.19 um 1
Re: CVE-2019-11500: Critical vulnerability in Dovecot and Pigeonhole
Daniel, thanks so much for the detailed pointers. So it turns out to be both the evil that is systemd and an overzealous upgrade script. Apollon, should I raise a Debian bug for this? As for reasons, how do 50k proxy session on the proxy servers and 25k imap processes on the mailbox servers sound? Even on a server with just 6k users and 7k imap processes that causes a massive load spike and a far longer service interruption (about 50 seconds) than I'm happy with. Penultimately if people do set "shutdown_clients = no" they hopefully know what they are doing and do expect that to work. Regards, Christian On Fri, 30 Aug 2019 17:44:23 +0200 Daniel Lange via dovecot wrote: > Am 30.08.19 um 17:38 schrieb Daniel Lange via dovecot: > > Am 30.08.19 um 10:00 schrieb Christian Balzer via dovecot: > >> When upgrading on Debian Stretch with the security fix packages all > >> dovecot processes get killed and then restarted despite having > >> "shutdown_clients = no" set. > > > > This is systemd doing its "magic" (kill all control group processes), > > see https://dovecot.org/pipermail/dovecot/2016-June/104546.html > > for a potential fix. > > Actually that will not be enough in the upgrade case as the maintainer > script calls > deb-systemd-invoke stop dovecot.socket dovecot.service > > I personally think re-connecting clients are normal operations so I > wouldn't bother. But you could override the stop action in the systemd > unit if you have local reasons that warrant such a hack. > -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Rakuten Mobile Inc.
Re: CVE-2019-11500: Critical vulnerability in Dovecot and Pigeonhole
Hello, Cc'ing Apollon in hopes he might have some insight here. When upgrading on Debian Stretch with the security fix packages all dovecot processes get killed and then restarted despite having "shutdown_clients = no" set. My guess would be a flaw in the upgrade procedure and/or unit files doing a stop and start when the new imapd package is installed. Can anybody think of a quick workaround or fix for this, as it's clearly not intended behavior (nor needed for this issue). Thanks, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Rakuten Mobile Inc.
Re: Anyone using object storage ceph?
No, not for lack of interest though. The former feels abandoned and has design issues IMHO. The later is a Dovecot Pro thing and when I talked with Timo 2(?) years ago he felt that Ceph with S3 or Swift wasn't a good fit and we didn't really want to go with Scality and simply can't do out of house/country stuff. The thing I said back then and I still think is valid is that I feel object storage with lots of small objects is going to require a fast and studly storage system below and due to the nature of the beast still loose against something like DRBD when it comes to both latency and overall costs. It also becomes more of a black box than maildir on a file system, but that's another and lesser issue. Christian On Tue, 14 May 2019 14:58:10 +0200 Marc Roos via dovecot wrote: > Just curious if there are already people actively using object storage? > > https://github.com/ceph-dovecot/dovecot-ceph-plugin > https://docplayer.net/docs-images/40/9935441/images/page_13.jpg > -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Rakuten Communications
Re: Push Notification clarification (MessageNew w/o LMTP or LDA)
Hello, On Thu, 9 May 2019 09:14:13 +0300 Aki Tuomi via dovecot wrote: > On 9.5.2019 9.01, Christian Balzer via dovecot wrote: > > Hello, > > > > Both the examples on the Push Notification wiki page and the XAPS plugin > > docs seem to suggest or state that LMTP/LDA is required. > > It is required. > Thanks for the quick, but unwelcome reply. There's a number of things that would simply break (in the sense of not working as people expect) if we were to change to LMTP. Not sure if that's worth it to make IOS users happy who willingly bought into a restrictive environment. > > > However IMAP IDLE notifications work without either of these (please no > > religious discussion of why Dovecot LMTP is the best thing since sliced > > bread and that everybody should use it). > > Without any religion, the IMAP IDLE notifications work because they > notify about different thing. The IDLE notifies about *changes* to > mailbox (detected or caused), while push notifications work when message > is actually delivered or saved. > > Also push notifications cannot happen if mail is delivered outside > dovecot, because it won't see them until the client logs in. > Yeah, the same client that would see the IMAP notification anyway if it hadn't been zombie'd by the (I)OS. Oh well, thanks again. Christian > > The LUA part of the push notification docs however states that events other > > than MessageNew are supported. > > > > So my question is, when not using LMTP/LDA and a LUA script and of course > > the correct mail_plugins definition (either global or for IMAP), will a > > message delivery trigger MessageNew? > > > > Regards, > > > > Christian > > No. Only LDA or LMTP based delivery will trigger MessageNew. > Copying/moving message, using IMAP APPEND or doveadm save will trigger > MessageAppend. > > Aki > > > -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Rakuten Communications
Push Notification clarification (MessageNew w/o LMTP or LDA)
Hello, Both the examples on the Push Notification wiki page and the XAPS plugin docs seem to suggest or state that LMTP/LDA is required. However IMAP IDLE notifications work without either of these (please no religious discussion of why Dovecot LMTP is the best thing since sliced bread and that everybody should use it). The LUA part of the push notification docs however states that events other than MessageNew are supported. So my question is, when not using LMTP/LDA and a LUA script and of course the correct mail_plugins definition (either global or for IMAP), will a message delivery trigger MessageNew? Regards, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Rakuten Communications
Re: v2.2.31 release candidate released
Hello, > - imap: NOTIFY command has been almost completely broken since the >beginning. I guess nobody has been trying to use it. I wonder with which client you would test that, I'm aware of none. We're pondering (commissioning) a NOTIFY capable client, based on the assumption that the Dovecot end actually works. Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Rakuten Communications
Re: [BUG] doveadm kick doesn't play well with hibernate
On Sun, 14 May 2017 23:34:39 +0300 Timo Sirainen wrote: > On 12 May 2017, at 5.14, Christian Balzer <ch...@gol.com> wrote: > > > > > > Hello, > > > > Dovecot 2.2.27 (Debian Jessie backports). > > > > When issuing a "doveadm kick" for a specific user I was greeted by: > > --- > > "warning: other connections would also be kicked from following users:" > > --- > > and a list of 22odd thousand users. > > > > As it turns out, kick wants to smack the hibernation proces(ses) that > > have hibernated sessions for this user, obviously the wrong approach here. > > Kicking in backends works by sending a signal to the mail processes. It would > be nice to support other ways also, but that would require larger changes. > It's probably something you'll need to address sooner or later, given that I'm pretty sure that kick isn't the only thing assuming (wrongly) that there's just one user per mail process. > > If there's doveadm command to wake up and thus un-hibernate sessions for a > > specific user this could be used as a workaround but a quick glance > > suggest no such thing. > > If you change the dovecot.index.log of any of the folders where the user is > IDLEing, that'd do it. I think just find /user 'dovecot.index.log' | xargs > touch would work. > That doesn't work for me at all. Before and after touching the dovecot.index.log files: --- # ps aux |grep chibi mail 884727 0.0 0.0 24360 9176 ?SMay14 0:00 dovecot/imap [ch...@notgol.com xxx.xxx.7.126 IDLE] # doveadm who |grep chibi username # proto (pids) (ips) ch...@notgol.com 8 imap (884727 877825 119157) (xxx.xxx.7.126 xxx.xxx.0.144 xxx.xxx.244.134) # ps aux |grep hibernate dovecot 119157 0.3 0.0 63448 56372 ?SApr01 247:20 dovecot/imap-hibernate [16361 connections] dovecot 877825 0.3 0.0 61776 54628 ?SApr23 115:39 dovecot/imap-hibernate [10098 connections] --- Note that I have "maildir_very_dirty_syncs = yes" set, but that shouldn't cause this based on the comment in the config blob. > > This functionality is absolutely crucial for us as we need this for our > > custom mailbox migration functionality. > > > > If there's no workaround or fix within 2 months I'll be forced to disable > > hibernation again (with a heavy heart). > > How about doveadm director kick / doveadm proxy kick? > Just proxy here. While that would work, it would require of course new code connecting to the proxies (maintain a list as well) in sequence. Something that doesn't scale too well, since we aim to keep the mailboxes in a locked state for as short as possible. OTOH if the "waking up" part above can't be made to work, I guess this is at least a way forward for the time being. Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
[BUG] doveadm kick doesn't play well with hibernate
Hello, Dovecot 2.2.27 (Debian Jessie backports). When issuing a "doveadm kick" for a specific user I was greeted by: --- "warning: other connections would also be kicked from following users:" --- and a list of 22odd thousand users. As it turns out, kick wants to smack the hibernation proces(ses) that have hibernated sessions for this user, obviously the wrong approach here. If there's doveadm command to wake up and thus un-hibernate sessions for a specific user this could be used as a workaround but a quick glance suggest no such thing. This functionality is absolutely crucial for us as we need this for our custom mailbox migration functionality. If there's no workaround or fix within 2 months I'll be forced to disable hibernation again (with a heavy heart). Regards, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: V1 Dovecot POP3 index migrate to V3
Hello, On Mon, 24 Apr 2017 14:48:01 +0300 Umut Arus wrote: > Hi, > > I've very old Dovecot version so I'm trying to migrate new disk and current > version of dovecot. But POP3 accounts have "version 1" > .INBOX/dovecot-uidlist files so clients download the messages again. > > How can achieve not to download to mailbox after copy to new disk? Is there > any script or something to migrate index v1 to v3? > > thanks for your suggestions. > Have you actually looked at the configuration file in question (20-pop3.conf)? Setting "pop3_uidl_format = %v.%u" should do the trick and if you're using maildir (we do) you can in addition set "pop3_save_uidl = yes" and in the future change to different UID format. Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: [BUG] client state / Message count mismatch with imap-hibernate and mixed POP3/IMAP access
Hello, On Thu, 6 Apr 2017 20:04:38 +0900 Christian Balzer wrote: > Hello Aki, Timo, > > according to git this fix should be in 2.2.27, which I'm running, so I > guess this isn't it or something else is missing. > See: > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859700 > Any update on this? Christian > Regards, > > Christian > > On Thu, 6 Apr 2017 15:37:33 +0900 Christian Balzer wrote: > > > Hello, > > > > On Thu, 6 Apr 2017 09:24:23 +0300 Aki Tuomi wrote: > > > > > On 06.04.2017 07:02, Christian Balzer wrote: > > > > > > > > Hello, > > > > > > > > this is on Debian Jessie box, dovecot 2.2.27 from backports, > > > > imap-hibernate obviously enabled. > > > > > > > > I've been seeing a few of these since starting this cluster (see > > > > previous > > > > mail), they all follow the same pattern, a user who accesses their > > > > mailbox > > > > with both POP3 and IMAP deletes mails with POP3 and the IMAP > > > > (imap-hibernate really) is getting confused and upset about this: > > > > > > > > --- > > > > > > > > Apr 6 09:55:49 mbx11 dovecot: imap-login: Login: > > > > user=<redac...@gol.com>, method=PLAIN, rip=xxx.xxx.x.46, > > > > lip=xxx.xxx.x.113, mpid=951561, secured, session=<2jBV+HRM1Pbc9w8u> > > > > Apr 6 10:01:06 mbx11 dovecot: pop3-login: Login: > > > > user=<redac...@gol.com>, method=PLAIN, rip=xxx.xxx.x.46, > > > > lip=xxx.xxx.x.41, mpid=35447, secured, session= > > > > Apr 6 10:01:07 mbx11 dovecot: pop3(redac...@gol.com): Disconnected: > > > > Logged out top=0/0, retr=0/0, del=1/1, size=20674 > > > > session= > > > > Apr 6 10:01:07 mbx11 dovecot: imap(redac...@gol.com): Error: > > > > imap-master: Failed to import client state: Message count mismatch > > > > after handling expunges (0 != 1) > > > > Apr 6 10:01:07 mbx11 dovecot: imap(redac...@gol.com): Client state > > > > initialization failed in=0 out=0 head=<0> del=<0> exp=<0> trash=<0> > > > > session=<2jBV+HRM1Pbc9w8u> > > > > Apr 6 10:01:15 mbx11 dovecot: imap-login: Login: > > > > user=<redac...@gol.com>, method=PLAIN, rip=xxx.xxx.x.46, > > > > lip=xxx.xxx.x.113, mpid=993303, secured, session=<6QC6C3VMF/jc9w8u> > > > > Apr 6 10:07:42 mbx11 dovecot: imap-hibernate(redac...@gol.com): > > > > Connection closed in=85 out=1066 head=<0> del=<0> exp=<0> trash=<0> > > > > session=<6QC6C3VMF/jc9w8u> > > > > > > > > --- > > > > > > > > Didn't see these errors before introducing imap-hibernate, but then > > > > again > > > > this _could_ be something introduced between 2.2.13 (previous generation > > > > of servers) and .27, but I highly doubt it. > > > > > > > > My reading of the log is that the original IMAP session > > > > (<2jBV+HRM1Pbc9w8u>) fell over and terminated, resulting in the client > > > > to > > > > start up a new session. > > > > If so and with no false data/state transmitted to the client it would be > > > > not ideal, but not a critical problem either. > > > > > > > > Would be delighted if Aki or Timo could comment on this. > > > > > > > > > > > > If you need any further data, let me know. > > > > > > > > Christian > > > > > > Hi! > > > > > > You could try updating to 2.2.28, if possible. I believe this bug is > > > fixed in 2.2.28, with > > > https://github.com/dovecot/core/commit/1fd44e0634ac312d0960f39f9518b71e08248b65 > > > > > Ah yes, that looks like the culprit indeed. > > > > I shall poke the powers that be over at Debian bugs to expedite the 2.2.28 > > release and backport. > > > > Christian > > -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: IMAP hibernate and scalability in general
Hello, Just to follow up on this, we've hit over 16k (default client limit here) hibernated sessions: --- dovecot 119157 0.1 0.0 63404 56140 ?SApr01 62:05 dovecot/imap-hibernate [11291 connections] dovecot 877825 0.2 0.0 28512 21224 ?SApr23 1:34 dovecot/imap-hibernate [5420 connections] --- No issues other than the minor bug I reported, CPU usage is slight (at most 2% of a CPU core), memory savings are immense, so I'm a happy camper. Just out of curiosity, how does dovecot decide to split and spread out sessions between hibernate processes? It's clearly something more involved than "fill up one and then fill up the next" or we would see 16k on the old one and a few on the new one. Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: System load spike on dovecot reload
Hello,
On Fri, 21 Apr 2017 10:43:47 +0200 d...@evilcigi.eu wrote:
> Hi everyone,
>
> I'm running dovecot with quite a lot of users and lots of active imap
> connections (like 20'000). I'm using different user IDs for users, so I
> need to have imap {service_count=1} - i.e. I have a lots of imap
> processes running.
>
We peaked out at 65k imap processes before upgrading to a version where
imap-hibernate more or less works, but we're using a common ID.
---
dovecot 119157 0.1 0.0 59364 52216 ?SApr01 48:25
dovecot/imap-hibernate [15137 connections]
---
The service_count parameter in this context is not doing what you think it
does, I have it at 200 these days and that will allow imap (or pop3)
processes to be recycled (they are labeled with "idling" when waiting for a
new client), not having one imap process serve multiple clients.
---
mail 591307 0.0 0.0 29876 4712 ?SApr20 0:00 dovecot/imap
[idling]
mail 735323 0.0 0.0 27396 4196 ?S13:20 0:00 dovecot/pop3
[idling]
---
The advantage (for me at least) is that the dovecot master process doesn't
have to to spin up a new mail processes each time during logins.
Since this process is quite single-threaded, it becomes a bottleneck
eventually.
> Everything works fine, until I reload dovecot configuration. When that
> happen, every client is forced to relogin in the same time and that
> causes a huge system load spike (2-3000 5 min load).
>
Unless you're making a change that affects the dovecot master process,
restarting everything isn't needed and you should set
"shutdown_clients = no".
You could still kick users with "dovecot kick" at a leisurely pace, but
security problems with the mail processes are rare.
> I was thinking that it would be great, if dovecot wouldn't kick all the
> users in the same time during reload, but somehow gradually, during
> specified interval. I'm aware of the shutdown_clients directive that
> could help, but I don't like it -
I've very much gotten to like it, once things got huge and busy.
> I do want the clients get disconnected
> on dovecot shutdown and also I want them to relogin in reasonably short
> time after reload.
>
> Is something like that possible with dovecot or does it make sense to
> implement that in the future versions?
>
Run a dovecot proxy (if you have single box with all these users on it,
Mr. Murphy would like a word with you) and set
"login_proxy_max_disconnect_delay" to something that suits you.
Christian
--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com Global OnLine Japan/Rakuten Communications
http://www.gol.com/
Re: IMAP hibernate and scalability in general
On Mon, 10 Apr 2017 23:11:24 +0300 Timo Sirainen wrote: > On 10 Apr 2017, at 21.49, Mark Moseley <moseleym...@gmail.com> wrote: > > > > Timo, any sense on where (if any) the point is where there are so many > > connections on a given login process that it would get too busy to keep up? > > I.e. where the sheer amount of stuff the login process has to do outweighs > > the CPU savings of not having to context switch so much? > > There might be some unexpected bottleneck somewhere, but I haven't heard of > anyone hitting one. > I haven't, OTOH context switching isn't _that_ bad and unless you're having a very dedicated box just doing this one thing it might happen anyway. Never mind that NUMA and device IRQ adjacency also factor into this. So you want to probably start with some reasonable (and that means larger than what you expect to be needed ^o^) value and if it grows beyond that no worries. > > I realize that's a terribly subjective question, so perhaps you might have > > a guess in very very round numbers? Given a typical IMAP userbase > > (moderately busy, most people sitting in IDLE, etc), I woud've thought 10k > > connections on a single process would've been past that tipping point. > > > > With the understood caveat of being totally subjective and dependent on > > local conditions, should 20k be ok? 50k? 100k? > > I only remember seeing a few thousand connections per process, but the CPU > usage there was almost nothing. So I'd expect it to scale well past 10k > connections. I think it's mainly limited by Linux, and a quick google shows > 500k, but I guess that's per server and not per process. Still, that's likely > not all that many CPUs/processes. > http://stackoverflow.com/questions/9899532/maximum-socket-connection-with-epoll > > <http://stackoverflow.com/questions/9899532/maximum-socket-connection-with-epoll> As I wrote and from my substantial experience, 8k connections per process are no issue at all, I'd expect it to go easily up to 50k. But w/o any pressing reason I'd personally keep it below 20k, too many eggs in one basket and all that. And in the original context of this thread, an imap-hibernate process with 2.5k connections uses about 10MB RAM and 0.5% of a CPU core, so 16k per process as configured here should be a breeze. > > Maybe a better question is, is there anywhere in the login process that is > > possible to block? > > Shouldn't be. Well, logging, but all the login processes are sharing the same > log pipe so if one blocks the others would block too. > > > If not, I'd figure that a login process that isn't using > > up 100% of a core can be assumed to *not* be falling behind. Does that seem > > accurate? > > Should be. In general I haven't heard of installations hitting CPU limits in > proxies. The problem so far has always been related to getting enough > outgoing sockets without errors, which is a server-wide problem. 2.2.29 has > one tweak that hopefully helps with that. > Which would be? The delayed connection bit? Anyway, with a properly sized login_source_ips pool this shouldn't be an issue, I have 80k sessions (that's 160k connections total) per proxy server now and they are bored. Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: IMAP hibernate and scalability in general
On Thu, 6 Apr 2017 13:10:03 +0300 Timo Sirainen wrote: > On 6 Apr 2017, at 9.56, Christian Balzer <ch...@gol.com> wrote: > > > >> For no particular reason besides wanting to start conservatively, we've got > >> client_limit set to 50 on the hibernate procs (with 1100 total hibernated > >> connections on the box I'm looking at). At only a little over a meg each, > >> I'm fine with those extra processes. > >> > > Yeah, but 50 would be a tad too conservative for our purposes here. > > I'll keep an eye on it and see how it goes, first checkpoint would be at > > 1k hibernated sessions. ^_^ > > imap-hibernate processes are similar to imap-login processes in that they > should be able to handle thousands or even tens of thousands of connections > per process. > I assume the config processes are in the same category, they are happy with 16k clients and using 169MB each, without any issues. ^.^ Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: [BUG] client state / Message count mismatch with imap-hibernate and mixed POP3/IMAP access
Hello Aki, Timo, according to git this fix should be in 2.2.27, which I'm running, so I guess this isn't it or something else is missing. See: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=859700 Regards, Christian On Thu, 6 Apr 2017 15:37:33 +0900 Christian Balzer wrote: > Hello, > > On Thu, 6 Apr 2017 09:24:23 +0300 Aki Tuomi wrote: > > > On 06.04.2017 07:02, Christian Balzer wrote: > > > > > > Hello, > > > > > > this is on Debian Jessie box, dovecot 2.2.27 from backports, > > > imap-hibernate obviously enabled. > > > > > > I've been seeing a few of these since starting this cluster (see previous > > > mail), they all follow the same pattern, a user who accesses their mailbox > > > with both POP3 and IMAP deletes mails with POP3 and the IMAP > > > (imap-hibernate really) is getting confused and upset about this: > > > > > > --- > > > > > > Apr 6 09:55:49 mbx11 dovecot: imap-login: Login: > > > user=<redac...@gol.com>, method=PLAIN, rip=xxx.xxx.x.46, > > > lip=xxx.xxx.x.113, mpid=951561, secured, session=<2jBV+HRM1Pbc9w8u> > > > Apr 6 10:01:06 mbx11 dovecot: pop3-login: Login: > > > user=<redac...@gol.com>, method=PLAIN, rip=xxx.xxx.x.46, > > > lip=xxx.xxx.x.41, mpid=35447, secured, session= > > > Apr 6 10:01:07 mbx11 dovecot: pop3(redac...@gol.com): Disconnected: > > > Logged out top=0/0, retr=0/0, del=1/1, size=20674 > > > session= > > > Apr 6 10:01:07 mbx11 dovecot: imap(redac...@gol.com): Error: > > > imap-master: Failed to import client state: Message count mismatch after > > > handling expunges (0 != 1) > > > Apr 6 10:01:07 mbx11 dovecot: imap(redac...@gol.com): Client state > > > initialization failed in=0 out=0 head=<0> del=<0> exp=<0> trash=<0> > > > session=<2jBV+HRM1Pbc9w8u> > > > Apr 6 10:01:15 mbx11 dovecot: imap-login: Login: > > > user=<redac...@gol.com>, method=PLAIN, rip=xxx.xxx.x.46, > > > lip=xxx.xxx.x.113, mpid=993303, secured, session=<6QC6C3VMF/jc9w8u> > > > Apr 6 10:07:42 mbx11 dovecot: imap-hibernate(redac...@gol.com): > > > Connection closed in=85 out=1066 head=<0> del=<0> exp=<0> trash=<0> > > > session=<6QC6C3VMF/jc9w8u> > > > > > > --- > > > > > > Didn't see these errors before introducing imap-hibernate, but then again > > > this _could_ be something introduced between 2.2.13 (previous generation > > > of servers) and .27, but I highly doubt it. > > > > > > My reading of the log is that the original IMAP session > > > (<2jBV+HRM1Pbc9w8u>) fell over and terminated, resulting in the client to > > > start up a new session. > > > If so and with no false data/state transmitted to the client it would be > > > not ideal, but not a critical problem either. > > > > > > Would be delighted if Aki or Timo could comment on this. > > > > > > > > > If you need any further data, let me know. > > > > > > Christian > > > > Hi! > > > > You could try updating to 2.2.28, if possible. I believe this bug is > > fixed in 2.2.28, with > > https://github.com/dovecot/core/commit/1fd44e0634ac312d0960f39f9518b71e08248b65 > > > Ah yes, that looks like the culprit indeed. > > I shall poke the powers that be over at Debian bugs to expedite the 2.2.28 > release and backport. > > Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: IMAP hibernate and scalability in general
Hello, On Wed, 5 Apr 2017 23:45:33 -0700 Mark Moseley wrote: > We've been using hibernate for about half a year with no ill effects. There > were various logged errors in earlier versions of dovecot, but even with > those, we never heard a reported customer-side error (almost always when > transitioning from hibernate back to regular imap; in the case of those > errors, presumably the mail client just reconnected silently). > This is my impression as well (silent reconnect w/o anything really bad happening) with regard to the bug I just reported/saw. > For no particular reason besides wanting to start conservatively, we've got > client_limit set to 50 on the hibernate procs (with 1100 total hibernated > connections on the box I'm looking at). At only a little over a meg each, > I'm fine with those extra processes. > Yeah, but 50 would be a tad too conservative for our purposes here. I'll keep an eye on it and see how it goes, first checkpoint would be at 1k hibernated sessions. ^_^ Christian > On Wed, Apr 5, 2017 at 11:17 PM, Aki Tuomi <aki.tu...@dovecot.fi> wrote: > > > > > > > On 06.04.2017 06:15, Christian Balzer wrote: > > > Hello, > > > > > > as some may remember, we're running very dense IMAP cluster here, in > > > excess of 50k IMAP sessions per node (current record holder is 68k, > > design > > > is for 200k+). > > > > > > The first issue we ran into was that the dovecot master process (which is > > > single thread and thus a bottleneck) was approaching 100% CPU usage (aka > > > using a full core) when trying to spawn off new IMAP processes. > > > > > > This was rectified by giving IMAP a service count of 200 to create a pool > > > of "idling" processes eventually, reducing the strain for the master > > > process dramatically. That of course required generous cranking up > > > ulimits, FDs in particular. > > > > > > The next issues of course is (as mentioned before) the memory usage of > > all > > > those IMAP processes and the fact that quite a few things outside of > > > dovecote (ps, etc) tend to get quite sedate when dealing with tens of > > > thousands of processes. > > > > > > We just started to deploy a new mailbox cluster pair with 2.2.27 and > > > having IMAP hibernate configured. > > > Getting this work is a PITA though with regards to ownership and access > > > rights to the various sockets, this part could definitely do with some > > > better (I know, difficult) defaults or at least more documentation (none > > > besides the source and this ML). > > > > > > Initial results are very promising, depending on what your clients are > > > doing (are they well behaved, are your users constantly looking at other > > > folders, etc) the vast majority of IDLE processes will be in hibernated > > > at any given time and thus not only using a fraction of the RAM otherwise > > > needed but also freeing up process space. > > > > > > Real life example: > > > 240 users, 86 imap processes (80% of those not IDLE) and: > > > dovecot 119157 0.0 0.0 10452 3236 ?SApr01 0:21 > > dovecot/imap-hibernate [237 connections] > > > That's 237 hibernated connections and thus less processes than otherwise. > > > > > > > > > I assume that given the silence on the ML what we are going to be the > > > first hibernate users where the term "large scale" does apply. > > > Despite that I have some questions, clarifications/confirmations: > > > > > > Our current default_client_limit is 16k, which can be seen by having 5 > > > config processes on our 65k+ session node. ^_- > > > > > > This would also apply to imap-hibernate, one wonders if that's fine > > > (config certainly has no issues) or if something smaller would be > > > appropriate here? > > > > > > Since we have idling IMAP processes around most of the time, the strain > > of > > > re-spawning proper processes from imap-hibernate should be just as > > reduced > > > as for the dovecot master, correct? > > > > > > > > > I'll keep reporting our experiences here, that is if something blows up > > > spectacularly. ^o^ > > > > > > Christian > > > > Hi! > > > > We have customers using it in larger deployments. A good idea is to have > > as much of your clients hibernating as possible, as the hibernation > > process is much smaller than actual IMAP process. > > > > You should probably also look at reusing the processes, as this will > > probably help your performance, > > https://wiki.dovecot.org/PerformanceTuning and > > https://wiki.dovecot.org/LoginProcess are probably a good starting > > point, although I suspect you've read these already. > > > > If you are running a dense server, cranking up various limits is rather > > expected. > > > > Aki > > > -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: [BUG] client state / Message count mismatch with imap-hibernate and mixed POP3/IMAP access
Hello, On Thu, 6 Apr 2017 09:24:23 +0300 Aki Tuomi wrote: > On 06.04.2017 07:02, Christian Balzer wrote: > > > > Hello, > > > > this is on Debian Jessie box, dovecot 2.2.27 from backports, > > imap-hibernate obviously enabled. > > > > I've been seeing a few of these since starting this cluster (see previous > > mail), they all follow the same pattern, a user who accesses their mailbox > > with both POP3 and IMAP deletes mails with POP3 and the IMAP > > (imap-hibernate really) is getting confused and upset about this: > > > > --- > > > > Apr 6 09:55:49 mbx11 dovecot: imap-login: Login: user=<redac...@gol.com>, > > method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.113, mpid=951561, secured, > > session=<2jBV+HRM1Pbc9w8u> > > Apr 6 10:01:06 mbx11 dovecot: pop3-login: Login: user=<redac...@gol.com>, > > method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.41, mpid=35447, secured, > > session= > > Apr 6 10:01:07 mbx11 dovecot: pop3(redac...@gol.com): Disconnected: Logged > > out top=0/0, retr=0/0, del=1/1, size=20674 session= > > Apr 6 10:01:07 mbx11 dovecot: imap(redac...@gol.com): Error: imap-master: > > Failed to import client state: Message count mismatch after handling > > expunges (0 != 1) > > Apr 6 10:01:07 mbx11 dovecot: imap(redac...@gol.com): Client state > > initialization failed in=0 out=0 head=<0> del=<0> exp=<0> trash=<0> > > session=<2jBV+HRM1Pbc9w8u> > > Apr 6 10:01:15 mbx11 dovecot: imap-login: Login: user=<redac...@gol.com>, > > method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.113, mpid=993303, secured, > > session=<6QC6C3VMF/jc9w8u> > > Apr 6 10:07:42 mbx11 dovecot: imap-hibernate(redac...@gol.com): Connection > > closed in=85 out=1066 head=<0> del=<0> exp=<0> trash=<0> > > session=<6QC6C3VMF/jc9w8u> > > > > --- > > > > Didn't see these errors before introducing imap-hibernate, but then again > > this _could_ be something introduced between 2.2.13 (previous generation > > of servers) and .27, but I highly doubt it. > > > > My reading of the log is that the original IMAP session > > (<2jBV+HRM1Pbc9w8u>) fell over and terminated, resulting in the client to > > start up a new session. > > If so and with no false data/state transmitted to the client it would be > > not ideal, but not a critical problem either. > > > > Would be delighted if Aki or Timo could comment on this. > > > > > > If you need any further data, let me know. > > > > Christian > > Hi! > > You could try updating to 2.2.28, if possible. I believe this bug is > fixed in 2.2.28, with > https://github.com/dovecot/core/commit/1fd44e0634ac312d0960f39f9518b71e08248b65 > Ah yes, that looks like the culprit indeed. I shall poke the powers that be over at Debian bugs to expedite the 2.2.28 release and backport. Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
[BUG] client state / Message count mismatch with imap-hibernate and mixed POP3/IMAP access
Hello, this is on Debian Jessie box, dovecot 2.2.27 from backports, imap-hibernate obviously enabled. I've been seeing a few of these since starting this cluster (see previous mail), they all follow the same pattern, a user who accesses their mailbox with both POP3 and IMAP deletes mails with POP3 and the IMAP (imap-hibernate really) is getting confused and upset about this: --- Apr 6 09:55:49 mbx11 dovecot: imap-login: Login: user=<redac...@gol.com>, method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.113, mpid=951561, secured, session=<2jBV+HRM1Pbc9w8u> Apr 6 10:01:06 mbx11 dovecot: pop3-login: Login: user=<redac...@gol.com>, method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.41, mpid=35447, secured, session= Apr 6 10:01:07 mbx11 dovecot: pop3(redac...@gol.com): Disconnected: Logged out top=0/0, retr=0/0, del=1/1, size=20674 session= Apr 6 10:01:07 mbx11 dovecot: imap(redac...@gol.com): Error: imap-master: Failed to import client state: Message count mismatch after handling expunges (0 != 1) Apr 6 10:01:07 mbx11 dovecot: imap(redac...@gol.com): Client state initialization failed in=0 out=0 head=<0> del=<0> exp=<0> trash=<0> session=<2jBV+HRM1Pbc9w8u> Apr 6 10:01:15 mbx11 dovecot: imap-login: Login: user=<redac...@gol.com>, method=PLAIN, rip=xxx.xxx.x.46, lip=xxx.xxx.x.113, mpid=993303, secured, session=<6QC6C3VMF/jc9w8u> Apr 6 10:07:42 mbx11 dovecot: imap-hibernate(redac...@gol.com): Connection closed in=85 out=1066 head=<0> del=<0> exp=<0> trash=<0> session=<6QC6C3VMF/jc9w8u> --- Didn't see these errors before introducing imap-hibernate, but then again this _could_ be something introduced between 2.2.13 (previous generation of servers) and .27, but I highly doubt it. My reading of the log is that the original IMAP session (<2jBV+HRM1Pbc9w8u>) fell over and terminated, resulting in the client to start up a new session. If so and with no false data/state transmitted to the client it would be not ideal, but not a critical problem either. Would be delighted if Aki or Timo could comment on this. If you need any further data, let me know. Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
IMAP hibernate and scalability in general
Hello, as some may remember, we're running very dense IMAP cluster here, in excess of 50k IMAP sessions per node (current record holder is 68k, design is for 200k+). The first issue we ran into was that the dovecot master process (which is single thread and thus a bottleneck) was approaching 100% CPU usage (aka using a full core) when trying to spawn off new IMAP processes. This was rectified by giving IMAP a service count of 200 to create a pool of "idling" processes eventually, reducing the strain for the master process dramatically. That of course required generous cranking up ulimits, FDs in particular. The next issues of course is (as mentioned before) the memory usage of all those IMAP processes and the fact that quite a few things outside of dovecote (ps, etc) tend to get quite sedate when dealing with tens of thousands of processes. We just started to deploy a new mailbox cluster pair with 2.2.27 and having IMAP hibernate configured. Getting this work is a PITA though with regards to ownership and access rights to the various sockets, this part could definitely do with some better (I know, difficult) defaults or at least more documentation (none besides the source and this ML). Initial results are very promising, depending on what your clients are doing (are they well behaved, are your users constantly looking at other folders, etc) the vast majority of IDLE processes will be in hibernated at any given time and thus not only using a fraction of the RAM otherwise needed but also freeing up process space. Real life example: 240 users, 86 imap processes (80% of those not IDLE) and: dovecot 119157 0.0 0.0 10452 3236 ?SApr01 0:21 dovecot/imap-hibernate [237 connections] That's 237 hibernated connections and thus less processes than otherwise. I assume that given the silence on the ML what we are going to be the first hibernate users where the term "large scale" does apply. Despite that I have some questions, clarifications/confirmations: Our current default_client_limit is 16k, which can be seen by having 5 config processes on our 65k+ session node. ^_- This would also apply to imap-hibernate, one wonders if that's fine (config certainly has no issues) or if something smaller would be appropriate here? Since we have idling IMAP processes around most of the time, the strain of re-spawning proper processes from imap-hibernate should be just as reduced as for the dovecot master, correct? I'll keep reporting our experiences here, that is if something blows up spectacularly. ^o^ Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: Scaling to 10 Million IMAP sessions on a single server
On Tue, 21 Feb 2017 09:49:39 -0500 KT Walrus wrote: > I just read this blog: > https://mrotaru.wordpress.com/2013/10/10/scaling-to-12-million-concurrent-connections-how-migratorydata-did-it/ > > <https://mrotaru.wordpress.com/2013/10/10/scaling-to-12-million-concurrent-connections-how-migratorydata-did-it/> > about scaling to 12 Million Concurrent Connections on a single server and it > got me thinking. > While that's a nice article, nothing in it was news to me or particular complex when one does large scale stuff, like Ceph for example. > Would it be possible to scale Dovecot IMAP server to 10 Million IMAP sessions > on a single server? > I'm sure Timo's answer will (or would, if he could be bothered) be along the lines of: "Sure, if you give me all your gold and then some for a complete rewrite of, well, everything". What you're missing and what the bad idea here is that as mentioned before scale-up only goes so far. I was feeling that my goal of 500k users/sessions in 2-node active/active cluster was quite ambitious and currently I'm looking at 200k sessions as something achievable with the current Dovecot and other limitations. But even if you were to implement something that can handle 1 million or more sessions per server, would you want to? As in, if that server goes down, the resulting packet, authentication storm will be huge and most like result in a proverbial shit storm later. Having more than 10% or so of your customers on one machine and thus involved in an outage that you KNOW will hit you eventually strikes me as a bad idea. I'm not sure how the design below meshes with Timo's lofty goals and standards when it comes to security as well. And a push with the right people (clients) to support IMAP NOTIFY would of course reduce the number of sessions significantly. Finally, Dovecot in proxy mode already scales quite well. Christian > I think the current implementation of having a separate process manage each > active IMAP session (w/ the possibility of moving idling sessions to a single > hibernate process) will never be able to deploy a single server managing 10 > Million IMAP sessions. > > But, would it be possible to implement a new IMAP server plugin that uses a > fixed configurable pool of “worker” processes, much like NGINX or PHP-FPM > does. These servers can probably scale to 10 Million TCP connections, if the > server is carefully tuned and has enough cores/memory to support that many > active sessions. > > I’m thinking that the new IMAP server could use some external database (e.g., > Redis or Memcached) to save all the sessions state and have the “worker” > processes poll the TCP sockets for new IMAP commands to process (fetching the > session state from the external database when it has a command that is > waiting on a response). The Dovecot IMAP proxies could even queue incoming > commands to proxy many incoming requests to a smaller number of backend > connections (like ProxySQL does for MySQL requests). That might allow each > Dovecot proxy to support 10 Million IMAP sessions and a single backend could > support multiple front end Dovecot proxies (to scale to 100 Million > concurrent IMAP connections using 10 proxies for 100 Million connections and > 1 backend server for 10 Million connections). > > Of course, the backend server may need to be beefy and have very fast NVMe > SSDs for local storage, but changing the IMAP server to manage a pool of > workers instead of requiring a process per active session, would allow bigger > scale up and could save large sites a lot of money. > > Is this a good idea? Or, am I missing something? > > Kevin -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: dovecot config for 1500 simultaneous connection
On Wed, 15 Feb 2017 08:43:23 +0530 Rajesh M wrote:
> christian
>
> the servers i currently own are dell servers. The servers i plan to buy are
> Dell R530, 2U rack servers with 8 x 3.5 inch drives, with 64 gb ram each,
> Hardware raid. I am thinking of 2 X 300 gb ssds raid1 and 6 x 2 tb drives in
> raid10 for data. I do not have any experience in setting up drdb (that would
> be my next step) ... primarily using standalone servers with hardware level
> redundancy. For backup-ups i rsync data to backup servers, also use qmail
> taps for backing up sent/recd emails on a per user basis.
>
> i am using mysql authentication backend
> my mail server is qmailtoaster, ie with vpopmail, spamdyke, simscan.
>
> you had mentioned as follows
> ##
> With 4000 users and 5 logins/s I can't imagine anything that wouldn't be
> able to:
> a) put all of the relevant data into RAM and
> b) thus handle auth requests in very timely manner.
> The example I gave earlier about my servers with 140 logins/s
> authenticates against LDAP and isn't very busy doing so while also
> fitting all 2.7GB of data for about 1 million users into RAM.
> ##
>
> >>>>>>>>>>>>>>>>>>how do i put all relevant data into the RAM ?
>
By configuring MySQL accordingly of course, for example:
--- my.cnf:
# Set buffer pool size to 50-80% of your computer's memory,
# but make sure on Linux x86 total memory usage is < 2GB
innodb_buffer_pool_size=8G
innodb_additional_mem_pool_size=32M
---
--- top:
PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND
39045 mysql 20 0 9170m 784m 10m S 0 2.4 3:38.69 mysqld
--- tuning-primer:
INNODB STATUS
Current InnoDB index space = 49 M
Current InnoDB data space = 109 M
Current InnoDB buffer pool free = 98 %
Current innodb_buffer_pool_size = 8.00 G
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory
---
Christian
> thanks
> rajesh
>
>
> - Original Message -
> From: Christian Balzer [mailto:ch...@gol.com]
> To: dovecot@dovecot.org
> Sent: Tue, 14 Feb 2017 16:40:01 +0900
> Subject:
>
>
> Hello,
>
> On Mon, 13 Feb 2017 22:53:23 +0530 Rajesh M wrote:
>
> > thanks for your help
> >
> > happy to say that the performance dramatically improved after i use the
> > high performance settings from here
> > http://wiki.dovecot.org/LoginProcess
> >
> That's why that page is there.
>
> > grep Login: /var/log/mail.log.1 |wc -l
> > with the mail.log being of a typical, busy day.
> > 412992
> >
> So about 5 login/s, not that busy, but that of course also depends on your
> HW and authentication backend.
>
> > i also picked up the imap and pop3 connections during peak hours
> > [root@ns1 domains]# doveadm who | awk '/imap/{m+=$2}/pop3/{n+=$2}END{print
> > m,n}'
> > username# proto (pids)
> > (ips)
> >
> > max figures i got was 535 for imap and 97 for pop
> >
> >
> > Question 1
> > i wish to improve the performance further by caching the logins. current
> > the same is kept disable because when user's change passwords then they are
> > not able to immediately login with the new password for some time. How to
> > solve this issue.
> >
> There is no solution, caching auth data is by default fraught with that
> sort of peril.
>
> And how did you determine that caching auth data would actually improve
> things?
>
> What is your authentication backend?
>
> With 4000 users and 5 logins/s I can't imagine anything that wouldn't be
> able to:
> a) put all of the relevant data into RAM and
> b) thus handle auth requests in very timely manner.
>
> The example I gave earlier about my servers with 140 logins/s
> authenticates against LDAP and isn't very busy doing so while also
> fitting all 2.7GB of data for about 1 million users into RAM.
>
> >
> >
> > further to the above.
> > i am planning to put up a large mail server with one of the following
> > options
> >
> Don't plan 1, always plan a pair and then use whatever you're comfortable
> with to turn them into a HA cluster, like Pacemaker with DRBD.
>
> > option 1
> > OS and mailqueue : 2 X 600 gb 15k rpm raid 1
> What kind of RAID, HW or MD?
>
> > data : 4 X 2000 gb raid 10 for data
> > backup : 2 X 2000 gb raid 10 for backup
>
> While with Linux MD a 2 disk RAID10 would be possible, it wouldn't be
> particular helpful (slower than a RAID 1 with writes).
>
> >
Re: dovecot config for 1500 simultaneous connection
Hello,
On Mon, 13 Feb 2017 22:53:23 +0530 Rajesh M wrote:
> thanks for your help
>
> happy to say that the performance dramatically improved after i use the high
> performance settings from here
> http://wiki.dovecot.org/LoginProcess
>
That's why that page is there.
> grep Login: /var/log/mail.log.1 |wc -l
> with the mail.log being of a typical, busy day.
> 412992
>
So about 5 login/s, not that busy, but that of course also depends on your
HW and authentication backend.
> i also picked up the imap and pop3 connections during peak hours
> [root@ns1 domains]# doveadm who | awk '/imap/{m+=$2}/pop3/{n+=$2}END{print
> m,n}'
> username# proto (pids)
> (ips)
>
> max figures i got was 535 for imap and 97 for pop
>
>
> Question 1
> i wish to improve the performance further by caching the logins. current the
> same is kept disable because when user's change passwords then they are not
> able to immediately login with the new password for some time. How to solve
> this issue.
>
There is no solution, caching auth data is by default fraught with that
sort of peril.
And how did you determine that caching auth data would actually improve
things?
What is your authentication backend?
With 4000 users and 5 logins/s I can't imagine anything that wouldn't be
able to:
a) put all of the relevant data into RAM and
b) thus handle auth requests in very timely manner.
The example I gave earlier about my servers with 140 logins/s
authenticates against LDAP and isn't very busy doing so while also
fitting all 2.7GB of data for about 1 million users into RAM.
>
>
> further to the above.
> i am planning to put up a large mail server with one of the following options
>
Don't plan 1, always plan a pair and then use whatever you're comfortable
with to turn them into a HA cluster, like Pacemaker with DRBD.
> option 1
> OS and mailqueue : 2 X 600 gb 15k rpm raid 1
What kind of RAID, HW or MD?
> data : 4 X 2000 gb raid 10 for data
> backup : 2 X 2000 gb raid 10 for backup
While with Linux MD a 2 disk RAID10 would be possible, it wouldn't be
particular helpful (slower than a RAID 1 with writes).
> 32 gb ram
>
>
> option 2
> os and mail queue : 4 X 600 gb 15k rpm raid 10
Those 15k HDDs tend to be expensive.
Estimate your expected writes per day (iostat on an existing server can
give you some hints) and get 2 SSDs instead, DC S3520s if 1 DWPD is
sufficient, DC S3610s if you need 3 DWPD (doubtful), S3710 if you need 10
DWPD (very doubtful).
Similar for Samsung DC level SSDs, check their specs.
For example, my busiest mailbox servers only write about 250KB/s averaged,
which mean about 50GB/day and thus would be a fit even for a small S3520.
I'm still using 200GB S3710s because I can afford them, like the speed and
never ever want to worry about wearout.
> data : 4 X 2000 gb raid 10 for data
> 32 gb ram
>
> i am having the OS and mail queue on primary drives
>
> i understand that Raid10 give a faster write and read whereas Raid 1 will
> have slow writes and fast reads
>
While that is true, you're missing the real difference here, the number of
HDDs. With 4 of them in the RAID10 they will be twice as fast as the 2
disk RAID1.
This can be tweaked even further with the various RAID 10 layout options,
"man md".
> the question is will there be huge performance difference between Raid1 and
> Raid 10 when it comes small files like mail queue ?
>
Again as explained above, double the IOPS, so yes.
Christian
>
> thanks
> rajesh
> - Original Message -
> From: Christian Balzer [mailto:ch...@gol.com]
> To: dovecot@dovecot.org
> Cc: ke...@my.walr.us
> Sent: Mon, 13 Feb 2017 10:46:13 +0900
> Subject:
>
>
> Hello,
>
> On Sun, 12 Feb 2017 08:27:21 -0500 KT Walrus wrote:
>
> > Thanks for the info. I do have one further question for you. On your
> > servers that are currently handling 50k IMAP sessions, how many users does
> > that correspond to? Since many users will have multiple IMAP sessions on
> > multiple devices, I’d like to hear about some real-world numbers that could
> > be used for budgeting a new project like mine.
> >
>
> Those servers would actually be the wrong ones to look at, as they are
> primarily accessed by the aforementioned broken client, so the numbers are
> somewhat skewed.
> However looking at other servers with a more "normal" user spread things
> aren't actually too different.
>
> The average number of sessions per user tends to be 2.
> The clear majority (over 50%) only has one session open (people with well
> behaved and configured clients watching the INBOX mostly).
> Another 30% has 2 sessions open, again the typical state of this w
Re: dovecot config for 1500 simultaneous connection
Hello, On Sun, 12 Feb 2017 08:27:21 -0500 KT Walrus wrote: > Thanks for the info. I do have one further question for you. On your servers > that are currently handling 50k IMAP sessions, how many users does that > correspond to? Since many users will have multiple IMAP sessions on multiple > devices, I’d like to hear about some real-world numbers that could be used > for budgeting a new project like mine. > Those servers would actually be the wrong ones to look at, as they are primarily accessed by the aforementioned broken client, so the numbers are somewhat skewed. However looking at other servers with a more "normal" user spread things aren't actually too different. The average number of sessions per user tends to be 2. The clear majority (over 50%) only has one session open (people with well behaved and configured clients watching the INBOX mostly). Another 30% has 2 sessions open, again the typical state of this would be clients watching another mailbox besides INBOX, typically SENT. The rest has 3 and more sessions open. The number of sessions could of course be drastically reduced if any client would support IMAP NOTIFY, alas none that I'm aware of do. Lastly no more than 60% of the session seem to be in IDLE at any given time, so my comments about RAM and IMAP hibernation effectiveness stand. > Also, do you use Dovecot IMAP proxies in front of your backend servers? If > so, how many IMAP sessions can one proxy server handle (assuming the proxy > does authorization using MySQL running on a separate server)? And, could the > proxy server be tuned to help in optimizing mostly IDLE backend sessions? > Yes to Dovecot Proxying, of course. No idea about MySQL, with (Open)LDAP nothing is breaking a sweat at an average of 140 logins per second (IMAP and POP) on the 2 proxy servers. If you can fit your entire dataset into RAM it should be fine, my LDAP servers fall into that category and take about 10% of a slow (1.2GHz, 34%, power-save mode) core only to handle the that load (also 2 servers). And the rate of logins/s is what you need to worry about most and optimize for. The proxies will of course have to do the shuffling of data and SSL en/de-coding, but again they're not particular busy with that. The number of sessions comes into play when looking at the number of login processes on the proxies and their memory footprint. An IMAP login process on the proxies in performance mode with a client limit of 1000 will consume about 55MB at most. So assume at least 55KB RAM per session. Read Timo's mail I linked to about IMAP hibernation, AFAIK nothing has happened to make proxies more supportive for this though. Christian > > On Feb 12, 2017, at 1:58 AM, Christian Balzer <ch...@gol.com> wrote: > > > > > > Hello, > > > > On Fri, 10 Feb 2017 14:50:03 -0500 KT Walrus wrote: > > > >>> 1. 256GB of real RAM, swap is for chums. > >> > >> Are you sure that 100,000 IMAP sessions wouldn’t work well with SWAP, > >> especially with fast SSD storage (which is a lot cheaper than RAM)? > >> > > > > I'm sure about tax and death, not much else. > > > > But as a rule of thumb I'd avoid swapping out stuff on production servers, > > even if it were to SSDs. > > Incidentally the servers I'm talking about here have their OS and swap on > > Intel DC S3710s (200GB) and the actual storage on plenty of 1.6TB DC > > S3610s. > > > > Relying on the kernel to make swap decisions is likely to result in much > > reduced performance even with fast SWAP when you're overcommitting things > > on that scale. > > > > > > But read on. > > > >> Seems that these IMAP processes are long lived processes (idling most of > >> the time) that don’t need that much of the contents of real memory > >> available for much of the life of the process. I use a database proxy in > >> front of MySQL (for my web apps) so that there can be a large number of > >> TCP connections to the proxy where the frontend requests are queued for > >> execution using a small number of backend connections. > >> > >> Could Dovecot IMAP be re-written to be more efficient so it works more > >> like MySQL (or other scalable data servers) that could handle a million or > >> more IMAP sessions on a server with 32GBs or less of RAM? Those IMAP > >> sessions aren’t doing much most of the time and shouldn’t really average > >> 2MB of active data per session that needs to be resident in main memory at > >> all times. > >> > > See IMAP hibernation: > > https://www.mail-archive.com/dovecot@dovecot.org/msg63429.html > > <https://www.mail-archive.com/dove
Re: dovecot config for 1500 simultaneous connection
Hello, On Fri, 10 Feb 2017 14:50:03 -0500 KT Walrus wrote: > > 1. 256GB of real RAM, swap is for chums. > > Are you sure that 100,000 IMAP sessions wouldn’t work well with SWAP, > especially with fast SSD storage (which is a lot cheaper than RAM)? > I'm sure about tax and death, not much else. But as a rule of thumb I'd avoid swapping out stuff on production servers, even if it were to SSDs. Incidentally the servers I'm talking about here have their OS and swap on Intel DC S3710s (200GB) and the actual storage on plenty of 1.6TB DC S3610s. Relying on the kernel to make swap decisions is likely to result in much reduced performance even with fast SWAP when you're overcommitting things on that scale. But read on. > Seems that these IMAP processes are long lived processes (idling most of the > time) that don’t need that much of the contents of real memory available for > much of the life of the process. I use a database proxy in front of MySQL > (for my web apps) so that there can be a large number of TCP connections to > the proxy where the frontend requests are queued for execution using a small > number of backend connections. > > Could Dovecot IMAP be re-written to be more efficient so it works more like > MySQL (or other scalable data servers) that could handle a million or more > IMAP sessions on a server with 32GBs or less of RAM? Those IMAP sessions > aren’t doing much most of the time and shouldn’t really average 2MB of active > data per session that needs to be resident in main memory at all times. > See IMAP hibernation: https://www.mail-archive.com/dovecot@dovecot.org/msg63429.html I'm going to deploy/test this in production in about 2 months from now, but if you look at the link and the consequent changelog entries you'll see that it has certain shortcomings and bug fixes in pretty much each release after it was introduced. But this is the correct way to tackle things, not SWAP. Alas I'm not expecting miracles and if more than 20% of the IMAP sessions here will be hibernated at any given time I'd be pleasantly surprised. Because between: 1. Finding a sensible imap_hibernate_timeout. 2. Having well behaved clients that keep idling instead of restarting the sequence (https://joshdata.wordpress.com/2014/08/09/how-bad-is-imap-idle/) 3. Having lots of mobile clients who either get disconnected (invisible to Dovecot) or have aggressive IDLE timers to overcome carrier NAT timeouts (a large mobile carrier here times out idle TCP sessions after 2 minutes, forcing people to use 1 minute IDLE renewals, making 1. up there a nightmare). 4. Having really broken clients (don't ask, I can't tell) which open IMAP sessions, don't put them into IDLE and thus having them expire after 30 minutes. the pool of eligible IDLE sessions isn't as big as it could be, in my case at least. > My mail server isn’t that large yet as I haven’t fully deployed Dovecot > outside my own small group yet, but it would be nice if scaling Dovecot IMAP > to millions of users wasn’t limited to 50,000 IMAP sessions on a server... > Scaling up is nice and desirable from a cost (rack space, HW) perspective, but the scalability of things OTHER than Dovecot as I pointed out plus that little detail of failure domains (do you really want half of your eggs in one basket?) argue for scaling out after a certain density. I'm feeling my way there at this time, but expect more than 100k sessions per server to be tricky. Lastly, when I asked about 500k sessions per server here not so long ago, ( http://www.dovecot.org/list/dovecot/2016-November/106284.html ) Timo mentioned that he's not aware of anybody doing more than 50k per server, something I got licked already and definitely will go to 100k eventually. Regards, Christian > > On Feb 10, 2017, at 11:07 AM, Christian Balzer <ch...@gol.com> wrote: > > > > On Fri, 10 Feb 2017 07:59:52 -0500 KT Walrus wrote: > > > >>> 1500 IMAP sessions will eat up about 3GB alone. > >> > >> Are you saying that Dovecot needs 2MB of physical memory per IMAP session? > >> > > That depends on the IMAP session, read the mailbox size and index size, > > etc. > > Some are significantly larger: > > --- > >PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > > > > 1033864 mail 20 0 97600 67m 54m S 0 0.1 0:01.15 imap > > > > --- > > > > But yes, as somebody who has mailbox servers with 55k+ session the average > > is around 1.6MB. > > > >> If I want to support a max 100,000 IMAP sessions per server, I should > >> configure the server to have at least 200GBs of SWAP? > >> > > You will want: > &
Re: dovecot config for 1500 simultaneous connection
On Fri, 10 Feb 2017 14:59:51 -0800 (PST) Joseph Tam wrote:
> "Rajesh M" <24x7ser...@24x7server.net> writes:
>
> > during peak times here are the results for connections
> >
> > [root@ns1 domains]# doveadm who |grep imap |wc -l
> > username# proto (pids) (ips)
> > 631
> > [root@ns1 domains]# doveadm who |grep pop3 |wc -l
> > username# proto (pids) (ips)
> > 233
>
> Thare are user counts, not connections. Ae a user can launch multiple
> IMAP connections, and if they have some MaxOSX reader, that can peak to
> several dozens during searches.
>
> This counts connections
>
> doveadm who | awk '/imap/{m+=$2}/pop3/{n+=$2}END{print m,n}'
>
> Or you can parse the output of netstat.
>
> I'm suprised you have so many POP3 connections though -- they tend to be
> connect/process/disconnect. n=0 most of the time on my modest server.
>
That vastly depends on things like client network speed and mailbox size.
People pop'ing multi GB mailboxes (leave mail on server) with crappy
clients (a big factor, some get disconnected for inactivity and thus
hang around for 10 minutes) and over slow links tend to linger for quite a
while.
Typical example:
---
Feb 11 13:54:27 mbx09 dovecot: pop3-login: Login: user=,
method=PLAIN, rip=redacted, lip=redacted, mpid=381958, secured,
session=
Feb 11 14:05:18 mbx09 dovecot: pop3(redacted): Disconnected: Logged out
top=0/0, retr=0/0, del=0/6188, size=103540794 session=
---
This is on a server that's bored stiff when it comes to CPU usage or I/O
utilization (pure SSD). All that twiddling of thumbs up there is purely
client based.
Christian
--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com Global OnLine Japan/Rakuten Communications
http://www.gol.com/
Re: dovecot config for 1500 simultaneous connection
Hello, On Fri, 10 Feb 2017 21:59:49 +0530 Rajesh M wrote: You replied below my signature, making a normal reply/quotation impossible for decent mail clients, which is worse than top-quoting. Please reply in-line or at the top if must be. > thanks christian > > during peak times here are the results for connections > > [root@ns1 domains]# doveadm who |grep imap |wc -l > username# proto (pids) (ips) > 631 > [root@ns1 domains]# doveadm who |grep pop3 |wc -l > username# proto (pids) (ips) > 233 > As Joseph mentioned, these are users, not sessions. And while this gives us some ideas, it doesn't answer my question about login rates. Do something like this: "grep Login: /var/log/mail.log.1 |wc -l" with the mail.log being of a typical, busy day. On my larger servers that average to 35 logins per second, with obviously higher peaks. Without the mail process re-usage (idling), that would have the dovecot master process use about 35% of a decent cpu core, with 100% being a hard limit. > > could you please guide me concerning the dovecot config files settings to > handle the above 631 imap and 233 pop connections. > What do you mean, isn't your current system handling that? See the various tuning, performance hints on the wiki, but w/o more info where your system is stalling, dovecot config changes might not be enough. > number of mailboxes is around 4000 -- some users would consume 25 GB while > others would be just around 10 MB > So that at least puts an upper limit to the users, however w/o quotas your users could easily swamp your storage. And those 25GB mailbox users will have a rather large IMAP mail process memory footprint. > this is a hex core machine with hyperthreading -- so 12 cores > With the exception of that dovecot master forking issue, I've never run out of CPU resources with it. > [root@ns1 domains]# iostat > Linux 2.6.32-431.29.2.el6.x86_64 (ns1.bizmailserver.net)02/10/2017 > _x86_64_(12 CPU) > > avg-cpu: %user %nice %system %iowait %steal %idle >2.67 0.000.65 3.430.00 > 93.25 > > Device:tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn > sdd 44.95 1094.25 765.10 720884842 504041712 > sdc 1.9232.15 0.03 21178186 21248 > sdb 34.71 1377.37 625.54 907398402 412102224 > sda 49.88 124.29 2587.32 81879548 1704506408 > > Rather meaningless w/o knowning which drive is which. Also an "iostat -x" oneshot summary and a few samples of when the machine is busy would be vastly more informative. atop is a good tool (when not running with 20k+ processes) to give you an idea about bottlenecks and what resource is being utilized how much. Christian > > thanks > rajesh > - Original Message - > From: Christian Balzer [mailto:ch...@gol.com] > To: dovecot@dovecot.org > Cc: 24x7ser...@24x7server.net > Sent: Fri, 10 Feb 2017 17:58:58 +0900 > Subject: > > On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M wrote: > > > hello > > > > could somebody with experience let me know the dovecot config file settings > > to handle around 1500 simultaneous connections over pop3 and 1500 > > connection over imap simultaneously. > > > > Be very precise here, you expect to see 1500 as the result of > "doveadm who |grep pop3 |wc -l"? > > Because that implies an ungodly number of POP3 connects per second, given > the typically short duration of these. > > 1500 IMAP connections (note that frequently a client will have more than > the INBOX open and thus have more than one session and thus process on the > server) are a much easier proposition, provided they are of the typical > long lasting type. > > So can you put a number to your expected logins per second (both protocols)? > > > my server > > > > server configuration > > hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive and 2 X > > 2000 > > gb hdd for data (No raid) > > > No RAID and no other replication like DRBD? > Why would you even bother? > > How many users/mailboxes in total with what quota? > > 1500 IMAP sessions will eat up about 3GB alone. > You will want more memory, simply to keep all relevant SLAB bits (inodes, > dentries) in RAM. > > If you really have several hundreds logins/s, you're facing several > bottlenecks: > 1. Login processes themselves (easily fixed by high performance mode) > 2. Auth processes (that will depend on your backends, method mostly) > 3. Doveco
Re: dovecot config for 1500 simultaneous connection
On Fri, 10 Feb 2017 07:59:52 -0500 KT Walrus wrote: > > 1500 IMAP sessions will eat up about 3GB alone. > > Are you saying that Dovecot needs 2MB of physical memory per IMAP session? > That depends on the IMAP session, read the mailbox size and index size, etc. Some are significantly larger: --- PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 1033864 mail 20 0 97600 67m 54m S 0 0.1 0:01.15 imap --- But yes, as somebody who has mailbox servers with 55k+ session the average is around 1.6MB. > If I want to support a max 100,000 IMAP sessions per server, I should > configure the server to have at least 200GBs of SWAP? > You will want: 1. 256GB of real RAM, swap is for chums. 2. Understanding how to tune Dovecot and more importantly the overall system to such a task (see that PID up there?). 3. Be willing to deal with stuff like top and ps taking ages to start/run and others like atop actually killing dovecot (performance wise, not literally) when doing their obviously flawed cleanup on exit. Some things clearly do NOT scale well. My current goal is to have 100k capable servers that work well, 200k in a failover scenario, but that won't be particular enjoyable. Christian > > On Feb 10, 2017, at 3:58 AM, Christian Balzer <ch...@gol.com> wrote: > > > > On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M wrote: > > > >> hello > >> > >> could somebody with experience let me know the dovecot config file > >> settings to handle around 1500 simultaneous connections over pop3 and 1500 > >> connection over imap simultaneously. > >> > > > > Be very precise here, you expect to see 1500 as the result of > > "doveadm who |grep pop3 |wc -l"? > > > > Because that implies an ungodly number of POP3 connects per second, given > > the typically short duration of these. > > > > 1500 IMAP connections (note that frequently a client will have more than > > the INBOX open and thus have more than one session and thus process on the > > server) are a much easier proposition, provided they are of the typical > > long lasting type. > > > > So can you put a number to your expected logins per second (both protocols)? > > > >> my server > >> > >> server configuration > >> hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive and 2 X > >> 2000 > >> gb hdd for data (No raid) > >> > > No RAID and no other replication like DRBD? > > Why would you even bother? > > > > How many users/mailboxes in total with what quota? > > > > 1500 IMAP sessions will eat up about 3GB alone. > > You will want more memory, simply to keep all relevant SLAB bits (inodes, > > dentries) in RAM. > > > > If you really have several hundreds logins/s, you're facing several > > bottlenecks: > > 1. Login processes themselves (easily fixed by high performance mode) > > 2. Auth processes (that will depend on your backends, method mostly) > > 3. Dovecot master process (spawning mail processes) > > > > The later is a single-threaded process, so it will benefit from a faster > > CPU core. > > It can be dramatically improved by enabling process re-usage, see: > > http://wiki.dovecot.org/PerformanceTuning > > > > However that also means more memory usage. > > > > > > > > Christian > > > >> > >> thanks > >> rajesh > >> > > > > [snip] > > -- > > Christian BalzerNetwork/Systems Engineer > > ch...@gol.com Global OnLine Japan/Rakuten Communications > > http://www.gol.com/ > -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: dovecot config for 1500 simultaneous connection
On Fri, 10 Feb 2017 01:13:20 +0530 Rajesh M wrote: > hello > > could somebody with experience let me know the dovecot config file settings > to handle around 1500 simultaneous connections over pop3 and 1500 connection > over imap simultaneously. > Be very precise here, you expect to see 1500 as the result of "doveadm who |grep pop3 |wc -l"? Because that implies an ungodly number of POP3 connects per second, given the typically short duration of these. 1500 IMAP connections (note that frequently a client will have more than the INBOX open and thus have more than one session and thus process on the server) are a much easier proposition, provided they are of the typical long lasting type. So can you put a number to your expected logins per second (both protocols)? > my server > > server configuration > hex core processor, 16 gb ram 1 X 600 gb 15 k rpm for main drive and 2 X 2000 > gb hdd for data (No raid) > No RAID and no other replication like DRBD? Why would you even bother? How many users/mailboxes in total with what quota? 1500 IMAP sessions will eat up about 3GB alone. You will want more memory, simply to keep all relevant SLAB bits (inodes, dentries) in RAM. If you really have several hundreds logins/s, you're facing several bottlenecks: 1. Login processes themselves (easily fixed by high performance mode) 2. Auth processes (that will depend on your backends, method mostly) 3. Dovecot master process (spawning mail processes) The later is a single-threaded process, so it will benefit from a faster CPU core. It can be dramatically improved by enabling process re-usage, see: http://wiki.dovecot.org/PerformanceTuning However that also means more memory usage. Christian > > thanks > rajesh > [snip] -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: auth client limit versus service count of mail processes
On Tue, 29 Nov 2016 14:30:37 +0200 Timo Sirainen wrote:
> On 29 Nov 2016, at 2.57, Christian Balzer <ch...@gol.com> wrote:
> >
> > service imap {
> > # Most of the memory goes to mmap()ing files. You may need to increase this
> > # limit if you have huge mailboxes.
> > #vsz_limit = $default_vsz_limit
> > vsz_limit = 512M
> >
> > # Max. number of IMAP processes (connections)
> > #process_limit = 1024
> > process_limit = 524288
> > }
> ..
> > But adding a "service_count = 100" line (any value larger than 1 really) to
> > the imap section we get the dreaded:
> > ---
> > Nov 28 17:05:40 mbx09 dovecot: config: Warning: service auth {
> > client_limit=16384} is lower than required under max. load (528384)
> > ---
> >
> > 1. Where's the difference in Dovecot's logic between a mail service that
> > has a service count of 1 versus one with >1?
>
> With service_count=1 it disconnects from auth immediately after logging in.
> With service_count>0 the auth connection is kept open for the entire
> existence of the imap process. This is mainly because after dropping
> privileges it wouldn't be able to connect to the auth-master socket again. In
> theory if the socket permissions were changed, it could keep reconnecting to
> auth-master and not keep connections open all the time.
>
Alright then, that's what I was suspecting.
Too bad, but totally understandable.
> > 2. Any way to get the process recycling for IMAP going w/o setting the fd
> > limit to a ridiculous amount?
>
>
> How about shrinking the imap process_limit? I highly doubt you can actually
> run 500k imap processes per server and have it still working. The largest
> I've ever heard people running has been 50k processes per server.
>
Well, that's the limit these servers could theoretically take IOPS wise,
other things like memory might curtail that earlier.
Also this is the fail-over level of this cluster pair, a single node
normally would only have to handle half of this.
Incidentally these 2 servers are currently running about 45k IMAP
processes each and the most busy process is (unsurprisingly) dovecot, the
master.
But that's only using about 20% of one core and the system is currently
operating with the on-demand CPU governor, so that core is only at half
speed typically.
It's a pure SSD system, I/O utilization tends to peak around 3% and
averages less than 1%.
Memory is still half "free" (page cache) and an upgrade of that is planned.
So from where I'm standing 100k per server (200k in fail-over) at the least
should be achievable easily.
Guess cranking up the fd limit it is then, still got 10 million spares
after all.
Thanks for the feedback,
Christian
--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com Global OnLine Japan/Rakuten Communications
http://www.gol.com/
auth client limit versus service count of mail processes
Hello,
We've got a pretty substantial dovecot installation (versions 2.1.7
and 2.2.13 on the backends, but tested with 2.2.24 as well) and this is the
relevant config snippet from 10-master.conf:
---
default_process_limit = 1024
default_client_limit = 16384
[...]
service imap {
# Most of the memory goes to mmap()ing files. You may need to increase this
# limit if you have huge mailboxes.
#vsz_limit = $default_vsz_limit
vsz_limit = 512M
# Max. number of IMAP processes (connections)
#process_limit = 1024
process_limit = 524288
}
service pop3 {
# Max. number of POP3 processes (connections)
process_limit = 2048
# Reduce spawns from hell
service_count = 100
}
---
The above works fine, no warnings.
Since 2 of our mailbox servers get a high number (2million/day) of IMAP
logins and resulting mail process spawns, I pondered doing the service
count bit for IMAP as well.
But adding a "service_count = 100" line (any value larger than 1 really) to
the imap section we get the dreaded:
---
Nov 28 17:05:40 mbx09 dovecot: config: Warning: service auth {
client_limit=16384} is lower than required under max. load (528384)
---
And that's quite true, once reaching that limit auth will pile up and fail
eventually.
Clearly the pop3 part with a max of 2048 processes (we see about 300 at
peak times) neatly fits into the client limit of 16384.
Now setting the client limit in the auth section to the required value of
course will do no real good in this scenario, as the fd limit becomes the
next bottleneck and I'm not going to raise that to 500k, thank you very
much. ^o^
So basically my questions here are:
1. Where's the difference in Dovecot's logic between a mail service that
has a service count of 1 versus one with >1?
2. Any way to get the process recycling for IMAP going w/o setting the fd
limit to a ridiculous amount?
Thanks,
Christian
--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com Global OnLine Japan/Rakuten Communications
http://www.gol.com/
Re: Reporting on CephFS being ready to use with Dovecot
ng RBD (and FS) from it? That should have been significantly more performant. > Our previous DRBD+Heartbeat > setup didn't allow for online maintenance and had a few problems. Now we > can do 100% online maintenance on storage without users noticing, and on > frontends with just a reconnect but without any downtime. > DRBD and Pacemaker can have issues, especially with some buggy resource agents around. Failing over a node in a controlled fashion takes a few seconds at most here, also in the "not noticeable" ballpark. Given that: a) with DRBD reads are local b) considering a) Ceph will always have the disadvantage of having to go via the net for everything and the resulting latency issues. c) to get roughly the same level of performance and reliability, one needs at least 33% more HW (storage) with Ceph and that's not including the additional frontends. So again, for the time being I'm happier to stay with DRBD pairs. Especially since we have a custom, in-house made migration system in place that will move dead-ish/large/low-usage mailboxes to slower clusters and smallish/high-usage mailboxes to faster ones. > Ceph is hard to learn at first but those with bigger setups and stronger > SLAs will want to take a look at that. I really recommend that the Dovecot > community take at look at that setup. > I agree with all parts of this, particular if you're not trying to squeeze the last ounce of speed from the least amount of rack space. There's another aspect of Ceph that may be of interest with Dovecot, using the object storage interface. However that's not supporting native Ceph interfaces and by its very nature also is slowish, but has nice scalability. Regards, Christian > Good luck! > > Best, > Daniel Colchete > > [1] http://docs.ceph.com/docs/hammer/dev/differences-from-posix/ > -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: "Plaintext authentication disallowed on non-secure (SSL/TLS) connections" despite correct configuration to allow this
Hello, talking to oneself seems to be all the rage on this ML, so I shall join that trend. As it turns out this was a case of slightly muddled/unclear error messages, the client sees: --- -ERR Plaintext authentication disallowed on non-secure (SSL/TLS) connections. --- But the actual issue was that the newly added "login_source_ips" (the main reason for this upgrade, as we're running out of ports) was not not in the "trusted_networks" of the target mailbox server. So the failure was between proxy and mailbox server, not client and proxy. After adding that network all is working now as expected. Christian On Tue, 2 Aug 2016 16:02:34 +0900 Christian Balzer wrote: > > Hello, > > this is basically a repeat of this query from last year, which > unfortunately got a deafening silence for replies: > --- > http://dovecot.org/pipermail/dovecot/2015-August/101720.html > --- > > I have mostly 2.1.7 (Debian Wheezy) mailbox servers and the current proxies > are also of that vintage. > > So with "ssl=yes" and "disable_plaintext_auth=no" plaintext logins work, > as per the documentation > (http://wiki2.dovecot.org/SSL/DovecotConfiguration) > and historically expected. > > Trying to use a 2.2.24 (Debian Jessie backports) dovecot proy with the > same parameters fails like this: > --- > Aug 2 15:45:57 smtp12 dovecot: pop3-login: proxy(chibi...@gol.com): Login > failed to mbxx.xxx.gol.com:110: Plaintext authentication disallowed on > non-secure (SSL/TLS) connections.: user=<chibi...@gol.com>, method=PLAIN, > rip=x.x.x.x, lip=x.x.x.x, pid=16066 > --- > > Changing things to "ssl=no" doesn't help and setting trusted networks only > changes the last bit to have "secured" appended but still fails the same > otherwise. > > I really need 2.2.x to behave the same way as before and documented. > > Any ideas and feedback would be most welcome. > > Regards, > > Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
"Plaintext authentication disallowed on non-secure (SSL/TLS) connections" despite correct configuration to allow this
Hello, this is basically a repeat of this query from last year, which unfortunately got a deafening silence for replies: --- http://dovecot.org/pipermail/dovecot/2015-August/101720.html --- I have mostly 2.1.7 (Debian Wheezy) mailbox servers and the current proxies are also of that vintage. So with "ssl=yes" and "disable_plaintext_auth=no" plaintext logins work, as per the documentation (http://wiki2.dovecot.org/SSL/DovecotConfiguration) and historically expected. Trying to use a 2.2.24 (Debian Jessie backports) dovecot proy with the same parameters fails like this: --- Aug 2 15:45:57 smtp12 dovecot: pop3-login: proxy(chibi...@gol.com): Login failed to mbxx.xxx.gol.com:110: Plaintext authentication disallowed on non-secure (SSL/TLS) connections.: user=<chibi...@gol.com>, method=PLAIN, rip=x.x.x.x, lip=x.x.x.x, pid=16066 --- Changing things to "ssl=no" doesn't help and setting trusted networks only changes the last bit to have "secured" appended but still fails the same otherwise. I really need 2.2.x to behave the same way as before and documented. Any ideas and feedback would be most welcome. Regards, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Rakuten Communications http://www.gol.com/
Re: Scalability with high density servers and proxies, TCP port limits
On Fri, 3 Jul 2015 14:29:55 +0900 Christian Balzer wrote: On Fri, 03 Jul 2015 07:05:43 +0200 Urban Loesch wrote: Hi, Am 03.07.2015 um 05:14 schrieb Christian Balzer: 2. Here is where the fun starts. Each IMAP session that gets proxied to the real mailbox server needs a port for the outgoing connection. So to support 2 million sessions we need 40 IP addresses here. Ouch. And from a brief test having multiple IP addresses per server won't help either (Dovecot unsurprisingly picks the main IP when establishing a proxy session to the real mailbox), at least not with just one default GW. To follow up on myself, with multiple IPs and appropriate(*) iproute rules this works as well. (*) for each IP in interfaces add something like this: --- up ip route add 192.168.1.0/24 dev eth0 src 192.168.1.109 table T2 up ip route add default via 192.168.1.1 table T2 up ip rule add from 192.168.1.109 table T2 --- And the tables in /etc/iproute/rt_tables. Christian If I remeber correctly there is a config option in dovecot 2.x where you can set the ip addresses which dovecot should use for outgoing proxy connections. Sorry, but I can't remeber the option. Looking at the documentation on the Wiki I was going to say That won't help, as it says address. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy But since that page is rather terse, I looked up the changelog and found that it indeed was added for use cases like mine: http://www.dovecot.org/list/dovecot-cvs/2014-June/024574.html Unfortunately the latest dovecot version in Debian is 2.2.13... Additionally this still leaves the actual mailbox servers, which in my case will need to be able to handle more than 50k sessions as well. Thanks for the info, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/
Scalability with high density servers and proxies, TCP port limits
Hello, first post in 3 years, kinda shows how painless Dovecot is. ^o^ Also this isn't really a dovecot issue, alas it's involved and since there are some large scale implementations of it I hope somebody here has some insights I might have missed. Currently we're running this setup: 1. LVS (DR mode) in a HA configuration (2 node cluster) 2. Dovecot in proxy mode on a 2 node cluster 3. Dovecot on actual mailbox servers (dual node DRBD clusters) There are about 500k users, but most of them use POP3, so there are usually less than 6k IMAP sesions at any given time. This is about to change, I'm looking at potentially millions of users who will have all semi-permanent IMAP sessions. We already have a pure SSD based mailbox cluster and based on the experiences with that another one is on order that will be able to easily handle about 500k users with regards to IOPS and other needs. However there's the issue of having all these concurrent IMAP sessions. Namely, running out of ephemeral ports. Lets assume 2 million users and 50k ports per IP and revisit the setup above. 1. LVS should have no problem, from experience and tests I expect a well tuned and spec'ed machine to handle millions of connections. This is in DR mode, in NAT mode I assume things would run into a wall a lot quicker. But even if LVS should run out of steam, there's a wide selection of high capacity load balancers available. 2. Here is where the fun starts. Each IMAP session that gets proxied to the real mailbox server needs a port for the outgoing connection. So to support 2 million sessions we need 40 IP addresses here. Ouch. And from a brief test having multiple IP addresses per server won't help either (Dovecot unsurprisingly picks the main IP when establishing a proxy session to the real mailbox), at least not with just one default GW. 3. All of this gets repeated on the actual mailbox servers, by either having a lot of low density servers or (preferably) high density servers with multiple IP addresses. Am I on track so far or missing something obvious? How many concurrent connections do you (hello Timo) think dovecot in proxy mode can handle? High performance mode of course in this case. I'm interested in internal limitations, assume that CPU and RAM are amply supplied. Any and all feedback is appreciated. Regards, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/
Re: Scalability with high density servers and proxies, TCP port limits
On Fri, 03 Jul 2015 07:05:43 +0200 Urban Loesch wrote: Hi, Am 03.07.2015 um 05:14 schrieb Christian Balzer: 2. Here is where the fun starts. Each IMAP session that gets proxied to the real mailbox server needs a port for the outgoing connection. So to support 2 million sessions we need 40 IP addresses here. Ouch. And from a brief test having multiple IP addresses per server won't help either (Dovecot unsurprisingly picks the main IP when establishing a proxy session to the real mailbox), at least not with just one default GW. If I remeber correctly there is a config option in dovecot 2.x where you can set the ip addresses which dovecot should use for outgoing proxy connections. Sorry, but I can't remeber the option. Looking at the documentation on the Wiki I was going to say That won't help, as it says address. http://wiki2.dovecot.org/PasswordDatabase/ExtraFields/Proxy But since that page is rather terse, I looked up the changelog and found that it indeed was added for use cases like mine: http://www.dovecot.org/list/dovecot-cvs/2014-June/024574.html Unfortunately the latest dovecot version in Debian is 2.2.13... Additionally this still leaves the actual mailbox servers, which in my case will need to be able to handle more than 50k sessions as well. Thanks for the info, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/
[Dovecot] Slow DNS warnings (proxy/auth)
Hello, I've just finished transiting our proxies from perdition to dovecot (2.1.7-7 Debian). Yesterday 12 messages (all within the same second) like this caught my attention: --- Apr 25 17:19:09 pp11 dovecot: auth: Warning: proxy(redac...@gol.com,xx.xx.xx.xx,26hUEivbfQBlMrMS): DNS lookup for mb04.dentaku.gol.com took 5.002 s --- Now this machine at that time was handling a load of about 2 logins per second, about 20% of what it previously handled with perdition w/o a hiccup. It also runs a local caching nameserver and the A record for the mailbox server in question was most definitely cached at the time (verified via TTL). The machine in question was very bored and certainly capable of handling hundreds if not thousands of DNS queries per second at that moment. In short, I can't see any reason how the lookup could have taken so long, so my guess is there are some issues with the dns-helper (locking, stepping on each others feet, not being spawned fast enough) causing this. Some general remarks, dovecot as proxy feels heavier than perdition. In the CPU area that's probably a more subjective impression, because all the little helper processes make it clear what's going on where. Though the config process being rather active is something that perdition definitely doesn't do, it reads the config once at start time and that's it. All the IPC and central processes of course also make dovecot rather file handle hungry. Memory wise it's about 35% bigger than perdition and that's not subjective at all. ^o^ About one MB per proxy process/connection for dovecot in my case. Caveat emptor. ^o^ Regards, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/
Re: [Dovecot] Proxying, pertinent values and features, SNI
On Thu, 4 Apr 2013 22:21:43 +0300 Timo Sirainen wrote:
On 3.4.2013, at 10.59, Christian Balzer ch...@gol.com wrote:
I'm looking into deploying dovecot as a proxy, currently using
perdition. Have been using dovecot on the actual servers for years,
nearly a decade. So far just 1.x, but for the proxy it will have to be
2.x (2.1.7 is the current Debian version), as the trigger for this
change is the need to support multiple SSL certificates.
All that happens on the proxy seems to be handled by the login
processes, so that is why we're not seeing anything useful in the
process titles or with doveadm, right?
And from past comments by Timo I guess that adding such functionality
isn't on his to-do list at all.
doveadm proxy list
That will teach me to look at man pages. ^o^
Internal help all the way, man pages are for chums. ^o^
Thanks!
A configurable capabilities string for POP would be quite welcome, but
at least nothing is different between the 1.x backends and the 2.x
proxy in that protocol.
v2.2 backends actually add some new POP3 capabilities. I guess there
could be such a setting, although it's a bit annoying to develop..
I guess so, but that will really make it an universally deployable proxy
and help people transitioning to dovecot from other environments, too.
[snip]
I presume to best support all(?) clients out there is to have
local_name sections for SNI first and then local sections for IP
address based certs. It is my understanding that SNI needs to be
requested by the client, so aside from client bugs (nah, those don't
exist ^o^) every client should get an appropriate response for TLS.
Has anybody done a setup like that already?
If you have separate IPs for each sertificate, you don't need to
support/configure SNI, so local {} blocks are enough.
I know that, the idea was/is to determine how many (connects and clients)
do a proper TLS/SNI negotiation if offered.
However are these even differently logged by dovecot? I suspect not.
Regards,
Christian
--
Christian BalzerNetwork/Systems Engineer
ch...@gol.com Global OnLine Japan/Fusion Communications
http://www.gol.com/
[Dovecot] Proxying, pertinent values and features, SNI
Hello, I'm looking into deploying dovecot as a proxy, currently using perdition. Have been using dovecot on the actual servers for years, nearly a decade. So far just 1.x, but for the proxy it will have to be 2.x (2.1.7 is the current Debian version), as the trigger for this change is the need to support multiple SSL certificates. All that happens on the proxy seems to be handled by the login processes, so that is why we're not seeing anything useful in the process titles or with doveadm, right? And from past comments by Timo I guess that adding such functionality isn't on his to-do list at all. A configurable capabilities string for POP would be quite welcome, but at least nothing is different between the 1.x backends and the 2.x proxy in that protocol. Speaking of 1.x versus 2.x, the feature to pass on the remote IP from the proxy to the backend is a 2.x thing only, correct? So I suppose any parameters really affecting this setup are default_process_limit and default_client_limit as well as the settings in service-imap-login and service pop-login. In particular mail_max_userip_connections never is looked at on the proxy as this check happens in the respective protocol AFTER login, rite? I presume to best support all(?) clients out there is to have local_name sections for SNI first and then local sections for IP address based certs. It is my understanding that SNI needs to be requested by the client, so aside from client bugs (nah, those don't exist ^o^) every client should get an appropriate response for TLS. Has anybody done a setup like that already? Regards, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/
Re: [Dovecot] Proxying, pertinent values and features, SNI
On Wed, 03 Apr 2013 11:13:41 +0100 Ed W wrote: Hi I presume to best support all(?) clients out there is to have local_name sections for SNI first and then local sections for IP address based certs. It is my understanding that SNI needs to be requested by the client, so aside from client bugs (nah, those don't exist ^o^) every client should get an appropriate response for TLS. Has anybody done a setup like that already? Although not what you asked for, just so you are aware, Godaddy (boo hiss, etc) offer reasonably inexpensive multi subject alt name based certs. This means you can have a single cert which is valid for lots of completely different domain names. The mild benefit is that this doesn't require SNI support for SSL (which I'm unsure is supported by many mail clients?) Yeah, I'm aware of multi-domain (SAN) certs, however there are 2 gotchas with those that our support and the OEMs this is for might not approve of: 1. Only the primary host will actually be validated/authenticated, which at least with some browsers will result in this being pointed out to the user when they connect to a SAN. Not sure about mail clients, but webmail is also in that overall deal, so support is probably not going to like the potential concerned you got hacked phone calls from customers. 2. Despite the fact that it will be trivial for anybody to determine that OEM A is now hosted with us, a SAN SSL makes all the SANs visible in one go, something they probably don't want. We're talking a small (10ish) number of OEMs here, so I'm happy and willing to throw some IP addresses at this particular problem and have everybody use (and deal with!) their own certs. As for SNI, supposedly most PC clients will support it, while most mobile ones don't. In my scenario it doesn't matter either way, the idea is to hand the correct cert to a client that requests it via SNI and for all the others based on the IP address they connected to. If everybody can be taught to use only TLS (not IMAPS/POP3S) and all the clients do support SNI, we can do away with the dedicated IP addresses. Might even happen before the heat death of the universe. ^o^ Regards, Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/
Re: [Dovecot] rule of thumb for indexing overhead
On Wed, 29 Aug 2007 12:31:01 -0700 (PDT) WJCarpenter [EMAIL PROTECTED] wrote: Also, are there any drawbacks of using exim to do the local delivery? I'm very interested in the answer to this question, too. So far I have found (through reading, not trying things yet) that Dovecot's quota handling is more flexible than Exim's (exim is pretty much limited to FS quotas, I think, which is no good for virtual users). Dovecot's Sieve implementation has more features than Exim's. Both of these happen to matter to me (and led me to Dovecot in the first place.) We use exim all the way to local delivery. And it handles Maildir++ quotas just fine. Depending on your actual needs and userdb/authdb choices the dovecot LDA might be the better choice. But exim can do pretty much everything one needs (we don't use sieve) and the advantages of indexing during delivery are negligible for us in general and potentially negative in some scenarios (mass mail to many/all users). An upside for Exim doing the delivery is that you theoretically can arrange for some additional rejections to happen at SMTP time. Pragmatically, those additional rejections are typically not done at SMTP time anyhow (e.g., an explicit fail from a user Exim filter). I don't think there are many situations where a MTA can do better than a LDA when it comes to rejections during SMTP time. Since at least with exim local delivery happens in a separate stage AFTER SMTP has been successfully completed.Now having consistent errors, logs and configurations for all mail delivery stages is something that might you want to stick with your MTA from the edge to the mailbox. Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] rule of thumb for indexing overhead
Hello, On Wed, 29 Aug 2007 18:10:00 -0700 Tom Bombadil [EMAIL PROTECTED] wrote: of indexing during delivery are negligible for us in general and potentially negative in some scenarios (mass mail to many/all users). Is this negative hypothetical or have you actually seen load spikes in situations like this? I actually did see that. In the case of exim, for each piece of mail going to the mailbox, one exim process is spawned, and this exim delivery process spawns the dovecot's LDA. The load pretty much doubles. There is that little detail of the additional processes spawned, but the overhead for that alone is not so much of an issue here (linux, plenty of CPU and memory to spare). But when you have 8 emails rushing towards your mailstorage (and yes, of course we have sensible limits on number of process, maximum load before defer, etc) your I/O will be a bottleneck and adding the additional strain of indexing on top of that is not a good idea. It is only at times of mass mails like that I see our mailstorge boxes break into a light sweat and avoiding any additional load then is just the sensible thing to do. The indexing for these mails will be spread out over hours to days (frequency of access by the clients) instead of being lumped on top of the existing peak and I/O saturation. It all boils down to your hardware and usage patterns. Our systems are plenty fast and any delay by having to (re)build the indexes is hardly noticeable. But if you have a system with less reserves, users that access mail very infrequently but get plenty of mail (so indexing at access time will be an involved process) and no mass mail spikes, dovecot LDA starts to look a lot more promising. ^_^ Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] Dovecot strong or not for a big Webmail architecture
Hello, On Thu, 2 Aug 2007 19:06:05 + (GMT) John fistack [EMAIL PROTECTED] wrote: Do you think Dovecot could handle millions of active users in a big architecture ? Sure, but the architecture will play a bigger role than just Dovecot. This cluster could be for example (each server is a bi quad Xeon 2.66 Ghz) : - 40 Dovecot servers - 4 LVS - 20 Apache+PHP - 2 Openldap - 20 Postfix + ClamAV + SpamAssassin - 1 NFS Netapps I'm not sure where you are pulling these numbers from (but then again webmail is a lot more resource intensive than real mail clients). We are handling about 100k users with a total of 9 machines and one SAN backend. And yes, this includes all the components up there with Exim instead of Postfix. About 10-20% are using webmail. 6 machines are cluster pairs, 3 are MXs (no need for clustering there) and yes, any single machine can fail and things will still work, pretty much w/o noticeable performance impact, too. The whole thing is designed to be pretty much indefinitely scalable. For historical reasons we stayed clear of NFS (this system architecture has been in production for 7 years, the last 3 with Dovecot), so I can not comment on that part from own experience. But I have a hard time imagining any single storage backend not melting from the traffic that these amounts of machines would generate (if they are actually needed). And even if the backend could handle it, the network towards it would be saturated a long time before those 40 Dovecot servers would run out of steam CPU wise. Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] Mark E-Mail as read
On Thu, 05 Jul 2007 17:13:04 +0200 Jens M. [EMAIL PROTECTED] wrote: Ok, I'll take a look at it. Thank's a lot! Timo Sirainen schrieb: On Thu, 2007-07-05 at 13:28 +0200, Jens M. wrote: Hi all, is there a way to configure Dovecot, so that it marks my e-mails as read, when I recieve them with pop3, but leave them on the server? It does that by default. # Don't try to set mails non-recent or seen with POP3 sessions. This is # mostly intended to reduce disk I/O. With maildir it doesn't move files # from new/ to cur/, with mbox it doesn't write Status-header. #pop3_no_flag_updates = no The leaving on the server bit is a CLIENT configuration issue and not part of dovecot, the marking as read is the default as pointed out by Timo. Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] Leaky dovecot-auth ?
On Wed, 27 Jun 2007 23:15:32 +0300 Timo Sirainen [EMAIL PROTECTED] wrote: On Thu, 2007-06-21 at 16:49 +0900, Christian Balzer wrote: You could try http://dovecot.org/patches/debug/mempool-accounting.diff and send USR1 signal to dovecot-auth after a while. It logs how much memory is used by all existing memory pools. Each auth request has its own pool, so if it's really leaking them it's probably logging a lot of lines. If not, then the leak is elsewhere. I grabbed the Debian package source on a test machine (not gonna chance anything on the production servers), applied the patch, did add --enable-debug to the debian/rules file (and got the #define DEBUG in config.h), created the binary packages, installed, configured, started them, tested a few logins and... nothing gets logged in mail.* if I send a USR1 to dovecot-auth. Anything I'm missing? Bug, fixed: http://hg.dovecot.org/dovecot-1.0/rev/a098e94cd318 Thanks, that fixed the silence of the auth-sheep. This is the output after start-up: --- Jul 2 13:59:54 engtest03 dovecot: auth(default): pool auth request handler: 104 / 4080 bytes Jul 2 13:59:54 engtest03 last message repeated 19 times Jul 2 13:59:54 engtest03 dovecot: auth(default): pool passwd_file: 56 / 10224 bytes Jul 2 13:59:54 engtest03 dovecot: auth(default): pool Environment: 224 / 2032 bytes Jul 2 13:59:54 engtest03 dovecot: auth(default): pool ldap_connection: 576 / 1008 bytes Jul 2 13:59:54 engtest03 dovecot: auth(default): pool auth: 1520 / 2032 bytes --- Used memory of dovecot-auth after 1 login was 3148KB(RSS). This is after a good trashing with rabid (from the postal package), with just 2 users though, using POP3 logins: --- Jul 2 14:12:30 engtest03 dovecot: auth(default): pool auth request handler: 104 / 4080 bytes Jul 2 14:12:30 engtest03 last message repeated 128 times Jul 2 14:12:30 engtest03 dovecot: auth(default): pool passwd_file: 56 / 10224 bytes Jul 2 14:12:30 engtest03 dovecot: auth(default): pool Environment: 224 / 2032 bytes Jul 2 14:12:30 engtest03 dovecot: auth(default): pool ldap_connection: 576 / 1008 bytes Jul 2 14:12:30 engtest03 dovecot: auth(default): pool auth: 1520 / 2032 bytes --- Note that the amount of auth request handler pools have grown to 128. After another short round of rabid the handler pools grew to 137 and the size of dovecot-auth to 5100KB. The number of handler pools never fell, nor did the memory footprint, obviously. :-p At about 800k logins/day/node here it's obvious now why dovecot-auth explodes after less than a week with max size of 512MB. But no matter, it is clearly leaking just as bad as 0.99 and I venture that his is the largest installation with LDAP as authentication backend. I wonder if this leak would be avoided by having LDAP lookups performed by worker processes as with SQL. Then you'd only have multiple leaking worker processes. Yes, I realize that. But I presume since these are designed to die off and be recreated on the fly the repercussions would be much better. ;) Of course now it looks like this is not LDAP related after all. The same as 0.99. You could also kill -HUP dovecot when dovecot-auth is nearing the limit. That makes it a bit nicer, although not perfectly safe either (should fix this some day..). If that leak can't be found I would very much appreciate a solution that at least avoids failed and/or delayed logins. That would require that login processes don't fail logins if connection to dovecot-auth drops, but instead wait until they can connect back to it and try again. And maybe another alternative would be to just disconnect the client instead of giving login failure. Anything that fixes this one way or the other would be nice. ^_^ Oh and HUP'ing the master is not an option here, I guess the system load triggers a race condition in dovecot because several times when doing so I got this: --- Jun 22 15:08:58 mb11 dovecot: listen(143) failed: Interrupted system call --- Which results in a killed off dovecot, including all active sessions. The self terminating dovecot-auth is not nice, but at least more predictable and does recover by itself: --- Jun 30 19:03:27 mb12 dovecot: auth(default): pool_system_malloc(): Out of memory Jun 30 19:03:27 mb12 dovecot: child 0 (auth) returned error 83 (Out of memory) Jun 30 19:03:28 mb12 dovecot: pop3-login: Can't connect to auth server at default: Resource temporarily unavailable Jun 30 19:03:28 mb12 last message repeated 11 times --- Of course the 12 users that tried to log in at this time are probably not amused or at least confused. Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] Leaky dovecot-auth ?
On Mon, 02 Jul 2007 17:37:05 +0300 Timo Sirainen [EMAIL PROTECTED] wrote: On Mon, 2007-07-02 at 15:20 +0900, Christian Balzer wrote: Jul 2 14:12:30 engtest03 dovecot: auth(default): pool auth request handler: 104 / 4080 bytes Jul 2 14:12:30 engtest03 last message repeated 128 times Auth request handler is created for each imap-login connection. So if you have 128 imap-login processes this isn't a leak. At that point in time only POP3 was tried, since this is by far the most used protocol here and rabid defaults to it anyway. But there were plenty of pop3-login processes indeed. Enough to make up that number combined with the IMAP ones. Which is interesting, as this does NOT happen on the production servers, I guess rabid can dish out even more stress than my users (and cause these login processes to be left hanging around). But that's not the issue anyway, with identical pool outputs the local DB incarnation retains its size (I got an internal IMAP server with 1.0.0 and PAM and a few dozen intense users which also shows no signs of a growing dovecot-auth) while the LDAP DB one keeps growing with nothing to show for in that pool debug output. Hmm. Does this help: http://hg.dovecot.org/dovecot-1.0/rev/50c79521e8f5 Will try that tomorrow if I can. Oh and HUP'ing the master is not an option here, I guess the system load triggers a race condition in dovecot because several times when doing so I got this: --- Jun 22 15:08:58 mb11 dovecot: listen(143) failed: Interrupted system call Did you use killall? I think this happens only with it. Nope, this is a Debian/Linux show and I did HUP just the master process. It only happened some of the times on the (then) busiest node, but it clearly is a race condition of sorts. Set up a test environment with about 30-50 logins/second and I'm sure you can reproduce it. ;) Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] Leaky dovecot-auth ?
On Mon, 02 Jul 2007 17:37:05 +0300 Timo Sirainen [EMAIL PROTECTED] wrote: Hmm. Does this help: http://hg.dovecot.org/dovecot-1.0/rev/50c79521e8f5 We have a winner! Auth process grows to the same size as with a local DB and stays there. Now I just have to get this into a security maintained Debian package... (looks around for the official package masters and backports maintainers) Am I correct in assuming that this code did not change since 0.99, read that the leak I saw for the last 4 years was the same thing? ;) Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] quota calculating
Hello, On Mon, 25 Jun 2007 10:10:00 -0400 (EDT) Scott Zahn [EMAIL PROTECTED] wrote: Currently where I work we use qmail. My condolences. The worlds most patched PoS (software, really, I mean software) by the biggest ego in the business. Every time a message is delivered, qmail seems to walk through the user's entire maildir in order to calculate quota usage. That doesn't seem right, not if it is using maildirsize maildir++ quotas, see: http://www.inter7.com/courierimap/README.maildirquota.html The worst case scenario would be a full mailbox, which would trigger a full recalculation every 15 minutes (Exim does it every 20 minutes). This recalculation is part of the specification, of course a smartly designed mail system will (with a few scripts and a database of sorts) prevent delivery attempts (and thus recalcs) to full mailboxes. So the first thing to check is if these mailboxes are perma-full or if they get so much activity that the maildirsize quickly grows to the 5k size that triggers a recalculation. If neither is the case, then yes, qmail is your enemy (surprise, surprise). Or if you are delivering with something else, that is the culprit then. Delivering messages simultaneously to just a few users who are allowed large maildirs (1-2GB) will bring our servers to their knees. Which brings me to dovecot. I use dovecot whereever I can, but haven't yet needed to set up quotas. I would like to migrate my servers at work to use dovecot, but have a few questions regarding quotas. Could someone explain how dovecot calculates quota usage on mail delivery? Also, is there anyone here on the list that uses quoatas on large maildirs (=2GB)? How well has it worked for you? Again, having to recalculate the quota eventually is part of the specification. If your hardware (I/O) can stand the infrequent recalculations working maildir++ quota requires you are gold. Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] Leaky dovecot-auth ?
On Tue, 19 Jun 2007 14:39:59 +0300 Timo Sirainen [EMAIL PROTECTED] wrote: On Tue, 2007-06-19 at 13:40 +0900, Christian Balzer wrote: 1. How and why would the memory footprint of dovecot-auth grow when there is no change in the amount of users in the DB? The only thing that's changing the size of dovecot-auth is how many requests it's simultaneously handling. For example if you try to login with invalid user/pass 1000 times within a second, dovecot-auth keeps those 1000 requests in memory for a couple of seconds until it returns with failure. But this happens also with normal requests, just not for so long. There is nowhere near anything of this kind of concurrency and memory should be returned after a while, but that is clearly not happening. The dovecot-auth is now at 200/160MB and thus prone to blow up over the weekend I guess. You could try http://dovecot.org/patches/debug/mempool-accounting.diff and send USR1 signal to dovecot-auth after a while. It logs how much memory is used by all existing memory pools. Each auth request has its own pool, so if it's really leaking them it's probably logging a lot of lines. If not, then the leak is elsewhere. I grabbed the Debian package source on a test machine (not gonna chance anything on the production servers), applied the patch, did add --enable-debug to the debian/rules file (and got the #define DEBUG in config.h), created the binary packages, installed, configured, started them, tested a few logins and... nothing gets logged in mail.* if I send a USR1 to dovecot-auth. Anything I'm missing? But no matter, it is clearly leaking just as bad as 0.99 and I venture that his is the largest installation with LDAP as authentication backend. I wonder if this leak would be avoided by having LDAP lookups performed by worker processes as with SQL. 2. What will happen when the single dovecot-auth process reaches 256MB in the end? Internal housekeeping attempts of that process? A whack to the head from the master process like in 0.99 and thus more erroneous authentication failures, potentially aggravated by the fact that there is just single dovecot-auth process? The same as 0.99. You could also kill -HUP dovecot when dovecot-auth is nearing the limit. That makes it a bit nicer, although not perfectly safe either (should fix this some day..). If that leak can't be found I would very much appreciate a solution that at least avoids failed and/or delayed logins. Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] pop3 creates bad index file
On Thu, 21 Jun 2007 11:20:26 +0200 Onno Molenkamp [EMAIL PROTECTED] wrote: Hi, When I use pop3 to log on to a (Maildir) mailbox that doesn't yet exist in the filesystem, the pop3 server autocreates it for me. This works. [...] I think you're seeing another variant of this bug in action: http://www.dovecot.org/list/dovecot/2007-May/022677.html (hurry up Debian package maintainers and bring me 1.0.1 :-p ) Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
Re: [Dovecot] No auto-conversion of .customflags?
Hello Timo, On Tue, 12 Jun 2007 18:22:13 +0300 Timo Sirainen [EMAIL PROTECTED] wrote: On Tue, 2007-06-12 at 10:50 +0900, Christian Balzer wrote: Jun 12 09:48:06 mb11 dovecot: IMAP(username): stat(/longandboringpath/username/.customflags/cur) failed: Not a directory .. But more importantly, obviously .customflags does NOT get converted anymore and thus shows up in IMAP LIST commands. Actually that could happen simply if the LIST command was done before any messages with keywords were tried to be accessed. I hadn't really thought about this before. Anyway, for now you can fix this by setting: maildir_stat_dirs = yes After there are no .customflags files left, you can remove that. Since this could take months if not years with some of our zombie mailboxes I guess I'll not take that route. ;) Most of them are empty aka from pop3 only accounts anyway, happily created by the 0.99 dovecot. Alternatively you can just rename all existing .customflags to dovecot-keywords files. That's how Dovecot internally converts them also. Ah, that sounds like a plan. And seeing that 1.0 does not create an empty default dovecot-keywords file when accessed by pop3, I presume I could just get rid of these (the ones) altogether, right? Looking forward to 1.0.1 anyways, as I'm getting bitten by the UID larger than next_uid bug here. ^_^ Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
[Dovecot] No auto-conversion of .customflags?
Hello, I'm in the process of migrating 0.99 dovecot maildir boxes to a pair of new machine. These servers were running the stable (etch) 1.0rc15 package of dovecot until yesterday and existing .customflags files were correctly converted to dovecot-keywords. Yesterday I upgraded to the (finally, thanks to who did this) 1.0.0 etch backports package and since then freshly migrated boxes get this error: --- Jun 12 09:48:06 mb11 dovecot: IMAP(username): stat(/longandboringpath/username/.customflags/cur) failed: Not a directory --- Firstly, I thought Linux was one of the OS' that had a free dir stat, but I guess I could have just hallucinated that. But more importantly, obviously .customflags does NOT get converted anymore and thus shows up in IMAP LIST commands. Since I doubt that the backport people did change anything in the code I suppose that this is a genuine 1.0.0 problem which probably should be addressed for the upcoming 1.0.1 release. Regards, Christian -- Christian BalzerNetwork/Systems EngineerNOC [EMAIL PROTECTED] Global OnLine Japan/Fusion Network Services http://www.gol.com/
