Re: Dovecot With DSpam AntiSpam Plugin Replacement With Sieve

[Trever L. Adams]

Thank you for this. I have a DSPAM setup that did work with the
suggested scrips, but I used the old dovecot antispam plugin and enjoyed
the signature method. I am going to try this out next week to see if it
works for me. If it does, I will love this.

Thanks again.
Trever

On 02/24/2018 02:02 PM, col...@nxtgn.com wrote:
> Hello
>
> I have recently had to rebuild my email server, it is a small server
> for my various email accounts and I also host the email for a clients
> business.
>
> Switched to Dovecot from Courier many years ago, and it has been a
> very reliable solution With the new rebuild I have updated it to
> Dovecot 2.2.33.2 with PigeonHole 4.21, it runs DSpam 3.10.2 to scan
> and tag all incoming mails as Spam or not then a Sieve rule to put the
> Spam emails into the Spam folder, I was also using the DSpam Antispam
> plugin to enable moving messages from one folder to another to teach
> failures to DSpam, this works well for my and my client.
>
> Doing the rebuild I saw that the AntiSpam plugin had been deprecated
> and the preferred solution now was to use Sieve Filters, so I
> implemented that and had struggled with it for a few hours until I
> finally got it working.
>
> It may be something about my setup but I am not sure it is, but the
> suggested way of doing it here
> https://wiki.dovecot.org/HowTo/AntispamWithSieve did not work for me
> and I don't think  it was ever going to.
>
> Here are the shell scripts and sieve filters that I came up with that
> work for me, your mileage may vary.
>
> I use DSpam as a standalone binary not as a daemon so you may need to
> add the --client option on the dspam command line in the shell scripts
> if you run DSpam in daemon mode.
>
> === report-spam.sieve ===
>
> require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment",
> "variables"];
>
>
> if environment :matches "imap.user" "*" {
>   set "userid" "${1}";
>   if header :matches "X-DSPAM-Signature" "*"
>   {
>    set "signature" "${1}";
>   }
> }
>
> pipe :copy "learn-spam.sh" [ "${userid}", "${signature}" ];
>
> === report-ham.sieve ===
>
> require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment",
> "variables"];
>
> if environment :matches "imap.mailbox" "*" {
>   set "mailbox" "${1}";
> }
>
> if string "${mailbox}" "Trash" {
>   stop;
> }
>
> if environment :matches "imap.user" "*" {
>   set "userid" "${1}";
>   if header :matches "X-DSPAM-Signature" "*"
>   {
>    set "signature" "${1}";
>   }
> }
>
> pipe :copy "learn-ham.sh" [ "${userid}", "${signature}" ];
>
> === learn-spam.sh ===
>
> #!/bin/sh
> exec /usr/local/bin/dspam --user ${1} --signature=${2} --class=spam
> --source=error
>
> === learn-ham.sh ===
>
> #!/bin/sh
> exec /usr/local/bin/dspam --user ${1} --signature=${2}
> --class=innocent --source=error
>
>
> I hope this saves someone some time or gives someone some tips
>




signature.asc
Description: OpenPGP digital signature

Re: Really slow IMAP performance

[Joseph Tam]

On Sat, 24 Feb 2018, Neil Jerram wrote:


My INBOX file has 22990 messages.  Is the slowness that I am seeing
definitely expected for an mbox of that size?  (It may also be relevant
that the HDD it's stored on is pretty old now, and has been known to
report SMART errors...)


Yeah, a copy of that mailbox will be that slow, esp. if the messages
have large attachments.  Even a simple operation like deleting/expunging the
the first message will cause data shuffling of the entire mailbox.

Joseph Tam 

Re: Really slow IMAP performance

[@lbutlr]

On 2018-02-24 (07:14 MST), Aki Tuomi  wrote:
> 
> https://wiki2.dovecot.org/Migration/MailFormat

That didn't show up when searching wiki2 for "Migration" :/


-- 
"...Life is not a journey to the grave with the intention of arriving
safely in one pretty and well-preserved piece, but to slide across the
finish line broadside, thoroughly used up, worn out, leaking oil, and
shouting GERONIMO!!!" -- Bill McKenna

Re: Really slow IMAP performance

[@lbutlr]

On 2018-02-24 (07:04 MST), Neil Jerram  wrote:
> 
> My INBOX file has 22990 messages.  Is the slowness that I am seeing
> definitely expected for an mbox of that size?  (It may also be relevant
> that the HDD it's stored on is pretty old now, and has been known to
> report SMART errors...)


back int he dark ages I would send and alert to users if their inbox for over 
1,000 messages because an mbox that large drove the server to its knees and 
made merely logging in to mail take an excruciating amount of time. If they 
didn't fix it I'd archive their inbox and start over.

I am astonished your machine can process an mbox with over 22 times that many 
messages.

https://wiki1.dovecot.org/Migration/MailFormat

That page doesn't exist on the wiki for dovecot 2, but that script to convert 
mbox to milder should still work. Obviously, keep backups and such.

-- 
Did they get you to trade your heroes for ghosts? Hot ashes for trees?
Hot air for a cool breeze? Cold comfort for change?

Re: Remove "Duplicate" emails (and documentation update)

[@lbutlr]

On 2018-02-23 (18:01 MST), @lbutlr  wrote:
> 
>>> First, even after expunging a message and running doveadm index -u
>>> kremels ?Archive?, subsequent runs still show the same duplicate
>>> messages.
>> 
>> I suspect client side caching.
> 
> No, there is no client side involved. I am executing all of these these 
> commands on the mail server. I expunge the messages, I index (or even 
> force-resync) and the next search shows the same messages even though they 
> are not in the Maildir anymore.

OK, I did another run yesterday and expunged 732 message dupes. I checked for a 
few of them before the expunge and there were files there. I checked after the 
expunge (several minutes after) and the files were NOT there.

They still show up if I do another search.

just now: 
 # doveadm -f table fetch -u kremels 'hdr.message-id guid uid hdr.x-listname 
mailbox' mailbox 'Archive' | sort | awk 'cnt[$1]++{if (cnt[$1]>=2) print 
prev[$1]; print} {prev[$1]=$0}'  | sort -u -k1,1| awk '{print "doveadm expunge 
-u kremels MAILBOX-GUID "$2" UID "$3}' | grep 1514660575.66506_1
hdr.message-id guiduid hdr.x-listname mailbox
doveadm expunge -u kremels MAILBOX-GUID 1514660575.66506_1.mail.covisp.net UID 
60608
 # find ~kremels/Maildir -name "1514660575.66506_1*"
 # doveadm fetch  -u kremels MAILBOX-GUID 1514660575.66506_1.mail.covisp.net 
UID 60608
Fatal: Invalid messageset

So, how do I get expunged messages to not show up in the fetch?

(Yes, I ran doveadm index -u kremels "*") 

# doveadm -Dv index -u kremels "*"
Debug: Loading modules from directory: /usr/local/lib/dovecot
Debug: Module loaded: /usr/local/lib/dovecot/lib20_virtual_plugin.so
Debug: Loading modules from directory: /usr/local/lib/dovecot/doveadm
Debug: Skipping module doveadm_acl_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/lib10_doveadm_acl_plugin.so: Undefined symbol 
"acl_user_module" (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_expire_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/lib10_doveadm_expire_plugin.so: Undefined symbol 
"expire_set_lookup" (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_quota_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/lib10_doveadm_quota_plugin.so: Undefined symbol 
"quota_user_module" (this is usually intentional, so just ignore this message)
Debug: Module loaded: 
/usr/local/lib/dovecot/doveadm/lib10_doveadm_sieve_plugin.so
Debug: Skipping module doveadm_fts_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/lib20_doveadm_fts_plugin.so: Undefined symbol 
"fts_filter_filter" (this is usually intentional, so just ignore this message)
Debug: Skipping module doveadm_mail_crypt_plugin, because dlopen() failed: 
/usr/local/lib/dovecot/doveadm/libdoveadm_mail_crypt_plugin.so: Undefined 
symbol "mail_crypt_user_get_public_key" (this is usually intentional, so just 
ignore this message)
doveadm(kremels): Debug: Effective uid=1004, gid=1004, home=/home/kremels
doveadm(kremels): Debug: Namespace inbox: type=private, prefix=, sep=, 
inbox=yes, hidden=no, list=yes, subscriptions=yes location=maildir:~/Maildir
doveadm(kremels): Debug: maildir++: root=/home/kremels/Maildir, index=, 
indexpvt=, control=, inbox=/home/kremels/Maildir, alt=
doveadm(kremels): Debug: Archive: Mailbox opened because: index
doveadm(kremels): Info: Archive: Cache is already up to date
doveadm(kremels): Debug: Drafts: Mailbox opened because: index
doveadm(kremels): Info: Drafts: Cache is already up to date
doveadm(kremels): Debug: Junk: Mailbox opened because: index
doveadm(kremels): Info: Junk: Cache is already up to date
doveadm(kremels): Debug: Misc.not-to-me: Mailbox opened because: index
doveadm(kremels): Info: Misc.not-to-me: Cache is already up to date
doveadm(kremels): Debug: bind: Mailbox opened because: index
doveadm(kremels): Info: bind: Cache is already up to date
doveadm(kremels): Debug: Sent: Mailbox opened because: index
doveadm(kremels): Info: Sent: Cache is already up to date
doveadm(kremels): Debug: Trash: Mailbox opened because: index
doveadm(kremels): Info: Trash: Cache is already up to date
doveadm(kremels): Debug: dovecot: Mailbox opened because: index
doveadm(kremels): Info: dovecot: Cache is already up to date
doveadm(kremels): Debug: freebsd: Mailbox opened because: index
doveadm(kremels): Info: freebsd: Cache is already up to date
doveadm(kremels): Debug: httpd: Mailbox opened because: index
doveadm(kremels): Info: httpd: Cache is already up to date
doveadm(kremels): Debug: spamassassin: Mailbox opened because: index
doveadm(kremels): Info: spamassassin: Cache is already up to date
doveadm(kremels): Debug: bbedit: Mailbox opened because: index
doveadm(kremels): Info: bbedit: Cache is already up to date
doveadm(kremels): Debug: postfix: Mailbox opened because: index
doveadm(kremels): Info: postfix: Cache is already up to date
doveadm(kremels): 

Dovecot With DSpam AntiSpam Plugin Replacement With Sieve

[colinh]

Hello

I have recently had to rebuild my email server, it is a small server for 
my various email accounts and I also host the email for a clients 
business.


Switched to Dovecot from Courier many years ago, and it has been a very 
reliable solution With the new rebuild I have updated it to Dovecot 
2.2.33.2 with PigeonHole 4.21, it runs DSpam 3.10.2 to scan and tag all 
incoming mails as Spam or not then a Sieve rule to put the Spam emails 
into the Spam folder, I was also using the DSpam Antispam plugin to 
enable moving messages from one folder to another to teach failures to 
DSpam, this works well for my and my client.


Doing the rebuild I saw that the AntiSpam plugin had been deprecated and 
the preferred solution now was to use Sieve Filters, so I implemented 
that and had struggled with it for a few hours until I finally got it 
working.


It may be something about my setup but I am not sure it is, but the 
suggested way of doing it here
https://wiki.dovecot.org/HowTo/AntispamWithSieve did not work for me and 
I don't think  it was ever going to.


Here are the shell scripts and sieve filters that I came up with that 
work for me, your mileage may vary.


I use DSpam as a standalone binary not as a daemon so you may need to 
add the --client option on the dspam command line in the shell scripts 
if you run DSpam in daemon mode.


=== report-spam.sieve ===

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", 
"variables"];



if environment :matches "imap.user" "*" {
  set "userid" "${1}";
  if header :matches "X-DSPAM-Signature" "*"
  {
   set "signature" "${1}";
  }
}

pipe :copy "learn-spam.sh" [ "${userid}", "${signature}" ];

=== report-ham.sieve ===

require ["vnd.dovecot.pipe", "copy", "imapsieve", "environment", 
"variables"];


if environment :matches "imap.mailbox" "*" {
  set "mailbox" "${1}";
}

if string "${mailbox}" "Trash" {
  stop;
}

if environment :matches "imap.user" "*" {
  set "userid" "${1}";
  if header :matches "X-DSPAM-Signature" "*"
  {
   set "signature" "${1}";
  }
}

pipe :copy "learn-ham.sh" [ "${userid}", "${signature}" ];

=== learn-spam.sh ===

#!/bin/sh
exec /usr/local/bin/dspam --user ${1} --signature=${2} --class=spam 
--source=error


=== learn-ham.sh ===

#!/bin/sh
exec /usr/local/bin/dspam --user ${1} --signature=${2} --class=innocent 
--source=error



I hope this saves someone some time or gives someone some tips

--
Regards

Colin

Re: Really slow IMAP performance

[Neil Jerram]

Aki Tuomi  writes:

> Yes. You deffo are looking at several reasons for slowness.
>
> I can only recommend moving into maildir or sdbox format, and probably new 
> HDD too.
>
> https://wiki2.dovecot.org/Tools/Doveadm/Sync here is example of 'converting' 
> between mailbox formats using dsync. You should also read 
> https://wiki2.dovecot.org/Migration/MailFormat
>
> mbox format has been known to act up with dsync occasionally, so I recommend 
> using 
>
> doveadm backup maildir:~/Maildir

Thanks, I've done that now, and things are looking much better.

I rediscovered that I've configured postfix to deliver locally using
dovecot-lmtp - which meant that I then only needed to change dovecot's
mail_location setting, and nothing at all in the postfix config.

Many thanks! - Neil

Re: Assertion during dsync receive

[Aki Tuomi]

> On 23 February 2018 at 23:12 Tanstaafl  wrote:
> 
> 
> On Fri Feb 23 2018 14:53:53 GMT-0500 (Eastern Standard Time), Aki Tuomi
>  wrote:
> > It is problem for any mailbox
> 
> Ok, so, I'm still unclear...
> 
> what cache are we talking about? Are you saying that there is a limit to
> how many emails dovecot can store in a single... 'folder'?

Dovecot has cache for commonly fetched fields. This is called 
dovecot.index.cache. The maximum file size for the cache is about 4G, the 
maximum offset is 0x4000 * 4, because all offsets are divided by 4. If the 
cache file becomes full, there are things you can do..

1. you can rm the cache file and hope it does not get full again too fast.
2. remove some mails

Aki

Re: Sieve

[Bill Shirley]

This might get you pointed in the right direction:
require "include";
require "fileinto";
require "body";
require "imap4flags";
require "regex";

if header :contains "subject" ["delayed mail", "undelivered mail", "returned to 
sender"] {
  addflag "$label1";
  stop;
}

if body :regex "rejected:[[:space:]]+unverified[[:space:]]+address" {
  fileinto "AASystemAdministration.Postmaster.Unverified";
  stop;
}

if body :regex 
"rejected:[[:space:]]+(domain|host)[[:space:]]+not[[:space:]]+found" {
  fileinto "AASystemAdministration.Postmaster.NXDOMAIN";
  stop;
}


Bill


On 2/19/2018 1:14 PM, IT wrote:


Hello,

Im newbie at sieve. I have a nice script running on my boxes,

and I have the following question:

Using Sieve,

is there a RULE to find out if String "MYTESTSTRING"

is ANYWHERE inside the Email, or ANYWHERE inside the HEADER at least --

in order to trigger actions like "discard;" consequently ?



Thank you so much in advance

best Regards

Florian




--

Mit freundlichen Grüßen / best regards

Florian Bojack

i...@u-eitner.de 



logoEitner

U.Eitner

Formenbau, Erodier-& Kunststofftechnik GmbH

Benzstraße 7

D-64807 Dieburg


Geschäftsführer Ulrich und Lars Eitner

Tel: +49 (0) 6071 23262

Fax: +49 (0) 6071 5562

Web: www.i...@u-eitner.de

Der Inhalt dieser Email und etwaiger Anhänge ist vertraulich und ausschließlich 
für den bezeichneten Adressaten bestimmt.

Jede unbefugte Kenntnisnahme, Veröffentlichung,Vervielfältigung oder Weitergabe 
ist unzulässig.

Sollten Sie nicht der vorgesehene Empfänger sein, bitten wir Sie, sich mit uns in Verbindung zu setzen und die Email zu 
löschen sowie alle Kopien zu vernichten.


This e-mail is intended solely for the addressee and may contain confidential 
and/or privileged information.

If you are not the intended recipient, please notify the sender immediately and 
destroy this e-mail.

In this case any form of reproduction, disclosure, distribution or any action taken or refrained from in reliance on it, is 
strictly prohibited.

Re: Really slow IMAP performance

[Håkon Alstadheim]

Den 24. feb. 2018 15:04, skrev Neil Jerram:
> Aki Tuomi  writes:
> 
>>> On 24 February 2018 at 15:47 Neil Jerram  wrote:
> 
> [...]
  Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning:
 Transaction log file
 /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked
 for 98 seconds (Mailbox was synchronized)
> [...]
> 
>> You are using mbox format. This is ... bit slow. =)
>>
>> When you move mails between mbox files, it has to rewrite the entire mbox 
>> file every time. You should probably start using maildir or sdbox instead.
> 
> Ah, right, thanks.
> 
> My INBOX file has 22990 messages.  Is the slowness that I am seeing
> definitely expected for an mbox of that size?  (It may also be relevant
> that the HDD it's stored on is pretty old now, and has been known to
> report SMART errors...)
> 
> If so, I'll start looking at how to migrate, given that my system is
> Postfix + Dovecot.  If you have any particular recommendations or
> migration pointers for a system like that, I'd appreciate them.
> 

I'd go with whatever tools you are familiar with. If you don't know
where to start, formail(1) can read an mbox and do whatever for each
mail contained therein. This, together with procmail, used to be the
go-to tools in the days before IMAP.

Theese days you'd probably want to involve your local delivery agent on
the output from formail. The lda would invoke sieve instead of procmail
if that is your thing.

Whatever you do, try to set up so you can do some tests before you blast
22000 mails to somewhere you do not want them :-)

Re: Stale mailbox lock file

[Aki Tuomi]

It gets called when some imap process has left a lock file, but didn't clean it 
up (i.e. it crashed?). It means it cannot lock uidlist, but will notify if 
count of messages change.

It is not very simple to debug this kind of issue from outside.

Aki

> On 23 February 2018 at 13:53 Raffaele Gambelli 
>  wrote:
> 
> 
> So you don't know when this piece of code is executed:
> 
> 
> 
> case MAILBOX_LOCK_NOTIFY_MAILBOX_OVERRIDE:if 
> (storage->callbacks->notify_ok == NULL)
> break;str = t_strdup_printf("Stale mailbox lock file 
> detected, "
> "will override in %u seconds", secs_left);
> storage->callbacks->notify_ok(>box, str,
> storage->callback_context);
> break;}
> 
> 
> 
> 
> 
> > Also do you have some other process(es) that touch the mailbox?
> 
> 
> Without having access to this dovecot installation, I can't answer your 
> question, but which could be the problem if there were some other processes?
> 
> 
> 
> I try to reformulate question, is it possible that when dovecot sends that 
> notification, it blocks and don't send any other notifications like for 
> example when the count of messages in a folder is changed?
> 
> 
> Raffaele Gambelli
> 
> -"dovecot"  ha scritto: -
> Per: Raffaele Gambelli , 
> dovecot@dovecot.org
> Da: Aki Tuomi 
> Inviato da: "dovecot" 
> Data: 23/02/2018 12.10PM
> Oggetto: Re: Stale mailbox lock file
> 
> Yes. I am one of them.
> 
> 
> 
> ---
> Aki Tuomi
> Dovecot oy
> 
>  Original message 
> From: Raffaele Gambelli 
> Date: 23/02/2018 12:25 (GMT+02:00)
> To: dovecot@dovecot.org
> Subject: Re: Stale mailbox lock file
> 
> Thanks Aki,
> 
> do you know if this list is read by dovecot developers team too? Could you 
> point me to the right list otherwise?
> 
> 
> Raffaele Gambelli
> 
> 
> 
> -Aki Tuomi  ha scritto: -
> Per: Raffaele Gambelli 
> Da: Aki Tuomi 
> Data: 23/02/2018 11.22AM
> Cc: dovecot@dovecot.org
> Oggetto: Re: Stale mailbox lock file
> 
> 
> I have no idea how you'd even get that error then.
> 
> Aki
> 
> 
> On 23.02.2018 12:07, Raffaele Gambelli wrote:
> 
> > Thanks but what answer it mean " doveconf -n"? 
> > I have neither control nor access in the dovecot installation to 
> > which I'm connecting.
> > 
> > I'm waiting to know the dovecot version but it could be that for 
> > legal reasons I will never receive it.
> > 
> > So please, if you know it obviously, why and when a dovecot server 
> > sends that notice?
> > 
> > Thanks, bye
> > 
> > Raffaele Gambelli
> > 
> > -Aki Tuomi  ha scritto: -
> >   Per: Raffaele Gambelli , 
> > dovecot@dovecot.org
> >   Da: Aki Tuomi 
> >   Data: 23/02/2018 10.54AM
> >   Oggetto: Re: Stale mailbox lock file
> > 
> > 
> > 
> > 
> >   On 23.02.2018 11:41, Raffaele Gambelli wrote:
> > 
> >   > >   Hi all,
> > >   I would like to understand the meaning of "Stale mailbox 
> > > lock file detected"
> > >   In index-storage.c ( 
> > > https://mis.dopa.go.th/atmail/server_source/dovecot/src/lib-storage/index/index-storage.c
> > >  ) there is this notice "Stale mailbox lock file detected" ...
> > > 
> > >   I have a java application which connects via javamail to a 
> > > dovecot server and I have some anomalies when that message comes.
> > > 
> > >   Thanks, bye
> > > 
> > >   Raffaele Gambelli
> > > 
> > > 
> > >   >   This is third email you've sent about this...
> > 
> >   doveconf -n
> > 
> >   Also do you have some other process(es) that touch the mailbox?
> > 
> >   Aki
> > 
> >   

Re: Really slow IMAP performance

[Neil Jerram]

Aki Tuomi  writes:

>> On 24 February 2018 at 15:47 Neil Jerram  wrote:

[...]
>> >  Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning:
>> > Transaction log file
>> > /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked
>> > for 98 seconds (Mailbox was synchronized)
[...]

> You are using mbox format. This is ... bit slow. =)
>
> When you move mails between mbox files, it has to rewrite the entire mbox 
> file every time. You should probably start using maildir or sdbox instead.

Ah, right, thanks.

My INBOX file has 22990 messages.  Is the slowness that I am seeing
definitely expected for an mbox of that size?  (It may also be relevant
that the HDD it's stored on is pretty old now, and has been known to
report SMART errors...)

If so, I'll start looking at how to migrate, given that my system is
Postfix + Dovecot.  If you have any particular recommendations or
migration pointers for a system like that, I'd appreciate them.

Best wishes - Neil

Re: dovecot auth and horde webmail

[Aki Tuomi]

This sounds awfully like problem in horde.

Aki

> On 24 February 2018 at 01:21 David Mehler  wrote:
> 
> 
> Hello,
> 
> I'm not sure if this is a Dovecot-specific question, or Postfix or
> Horde webmail. As Dovecot is used for authenticating both Postfix as
> well as horde I thought i'd start here.
> 
> I've got a new horde webmail install going on a FreeBSD 11.1 jail.
> I've got Dovecot set up so that it appends a domain name if one is not
> given, so that user and u...@example.com can both log in.
> 
> When I logged in with horde webmail I used for the first attempt
> username with no @example.com suffix. I logged in ok, but couldn't
> send an email, gave me a weird error no address associated with host.
> I logged out, logged back in using u...@example.com the full address,
> and this time the message sending went through.
> 
> Any ideas or if this is not a Dovecot question let me know, as that
> means I'll have two other places to try.
> 
> Thanks.
> Dave.
> 
> doveconf -n
> # 2.2.33.2 (d6601f4ec): /usr/local/etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.21 (92477967)
> # OS: FreeBSD 11.1-RELEASE-p4 amd64
> auth_cache_size = 16 k
> auth_default_realm = example.com
> auth_mechanisms = plain login
> auth_realms = example.com example2.com
> dict {
>   acl = mysql:/usr/local/etc/dovecot/shared-folders.conf
>   sqlquota = mysql:/usr/local/etc/dovecot/quota.conf
> }
> first_valid_gid = 999
> first_valid_uid = 999
> hostname = mail.example.com
> imap_client_workarounds = delay-newmail tb-extra-mailbox-sep tb-lsub-flags
> last_valid_gid = 999
> last_valid_uid = 999
> lda_mailbox_autocreate = yes
> lda_mailbox_autosubscribe = yes
> listen = 127.0.0.1 xxx.xxx.xxx.xxx
> lmtp_rcpt_check_quota = yes
> mail_access_groups = vmail
> mail_fsync = never
> mail_gid = vmail
> mail_home = /home/vmail/%d/%n
> mail_location = maildir:~/mail/:LAYOUT=fs:INDEX=~/mail/
> mail_plugins = acl mail_log notify quota quota_clone trash virtual welcome 
> zlib
> mail_server_admin = mailto:postmas...@example.com
> mail_uid = vmail
> mailbox_list_index = yes
> managesieve_notify_capability = mailto
> managesieve_sieve_capability = fileinto reject envelope
> encoded-character vacation subaddress comparator-i;ascii-numeric
> relational regex imap4flags copy include variables body enotify
> environment mailbox date index ihave duplicate mime foreverypart
> extracttext imapflags notify imapsieve vnd.dovecot.imapsieve
> namespace {
>   hidden = no
>   list = yes
>   location = 
> maildir:/home/vmail/public/:LAYOUT=fs:CONTROL=~/mail/public:INDEXPVT=~/mail/public:INDEX=~/mail/public
>   mailbox TestFolder {
> auto = subscribe
> comment = Public Folder for message sharing
>   }
>   prefix = public/
>   separator = /
>   subscriptions = yes
>   type = public
> }
> namespace {
>   list = yes
>   location = maildir:~/mail/:INDEX=~/mail/shared/%%Ld/%%Ln
>   prefix = shared/%%u/
>   separator = /
>   subscriptions = yes
>   type = shared
> }
> namespace {
>   location = virtual:/usr/local/etc/dovecot/virtual
>   mailbox All {
> auto = subscribe
> comment = All my messages
> special_use = \All
>   }
>   prefix = virtual/
>   separator = /
> }
> namespace inbox {
>   inbox = yes
>   location =
>   mailbox Archive {
> auto = no
> special_use = \Archive
>   }
>   mailbox Archives {
> auto = subscribe
> special_use = \Archive
>   }
>   mailbox "Deleted Messages" {
> auto = no
> autoexpunge = 30 days
> special_use = \Trash
>   }
>   mailbox Drafts {
> auto = subscribe
> special_use = \Drafts
>   }
>   mailbox Junk {
> auto = no
> autoexpunge = 30 days
> special_use = \Junk
>   }
>   mailbox "Junk E-mail" {
> auto = no
> autoexpunge = 30 days
> special_use = \Junk
>   }
>   mailbox Sent {
> auto = subscribe
> special_use = \Sent
>   }
>   mailbox "Sent Items" {
> auto = no
> special_use = \Sent
>   }
>   mailbox "Sent Messages" {
> auto = no
> special_use = \Sent
>   }
>   mailbox Spam {
> auto = subscribe
> autoexpunge = 30 days
> special_use = \Junk
>   }
>   mailbox Trash {
> auto = subscribe
> autoexpunge = 30 days
> special_use = \Trash
>   }
>   prefix =
>   separator = /
>   type = private
> }
> passdb {
>   args = /usr/local/etc/dovecot/dovecot-sql.conf.ext
>   driver = sql
> }
> plugin {
>   acl = vfile:/usr/local/etc/dovecot/global-acls:cache_secs=300
>   acl_anyone = allow
>   acl_globals_only = yes
>   acl_shared_dict = proxy::acl
>   imapsieve_mailbox1_before =
> file:/usr/local/lib/dovecot/sieve/report-spam.sieve
>   imapsieve_mailbox1_causes = COPY
>   imapsieve_mailbox1_name = Spam
>   imapsieve_mailbox2_before = 
> file:/usr/local/lib/dovecot/sieve/report-ham.sieve
>   imapsieve_mailbox2_causes = COPY
>   imapsieve_mailbox2_from = Spam
>   imapsieve_mailbox2_name = *
>   mail_log_events = delete undelete expunge copy mailbox_delete mailbox_rename
>   mail_log_fields = uid box msgid 

Re: Really slow IMAP performance

[Aki Tuomi]

> On 24 February 2018 at 15:47 Neil Jerram  wrote:
> 
> 
> Aki Tuomi  writes:
> 
> >  On 24 February 2018 at 12:45 Neil Jerram < n...@ossau.homelinux.net> 
> > wrote: 
> >
> >  Please could you help me to understand and fix why my dovecot IMAP 
> >  performance is so bad? I've read through a lot of the 
> >  performance-related material on the website, but I don't think that any 
> >  of it could account for slowness at the level that I am seeing. 
> >
> >  The simplest scenario is moving a message from my Inbox to another IMAP 
> >  folder. Using Gnus as the client, the whole UI freezes for about 2 
> >  minutes (which I assume is until the move is complete), and journalctl 
> >  on the dovecot server says: 
> >
> >  Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log 
> > file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 
> > 98 seconds (Mailbox was synchronized) 
> >
> >  or the same message with (rotating while syncing). 
> >
> >  There must be something badly wrong in my setup, or perhaps in the spec 
> >  of the server that dovecot is running on. What should I look at to 
> >  start understanding this better? 
> >
> >  Many thanks - Neil 
> >
> > Can you tell a bit more about your environment? Sounds like io issue 
> 
> Thanks for your reply.  I'm not sure exactly what you have in mind, but
> here are some starting points:
> 
> arudy:~# uname -a
> Linux arudy 4.13.0-1-686-pae #1 SMP Debian 4.13.4-2 (2017-10-15) i686 
> GNU/Linux
> 
> arudy:~# dovecot -n
> # 2.2.32 (dfbe293d4): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.20 (7cd71ba)
> # OS: Linux 4.13.0-1-686-pae i686 Debian buster/sid 
> auth_mechanisms = plain login
> auth_username_format = %Ln
> auth_verbose = yes
> login_trusted_networks = 192.168.11.8
> mail_access_groups = mail
> mail_fsync = never
> mail_location = mbox:~/dovecot-mail:INBOX=/var/mail/%u

You are using mbox format. This is ... bit slow. =)

When you move mails between mbox files, it has to rewrite the entire mbox file 
every time. You should probably start using maildir or sdbox instead.

Aki

Re: Really slow IMAP performance

[Neil Jerram]

Aki Tuomi  writes:

>  On 24 February 2018 at 12:45 Neil Jerram < n...@ossau.homelinux.net> wrote: 
>
>  Please could you help me to understand and fix why my dovecot IMAP 
>  performance is so bad? I've read through a lot of the 
>  performance-related material on the website, but I don't think that any 
>  of it could account for slowness at the level that I am seeing. 
>
>  The simplest scenario is moving a message from my Inbox to another IMAP 
>  folder. Using Gnus as the client, the whole UI freezes for about 2 
>  minutes (which I assume is until the move is complete), and journalctl 
>  on the dovecot server says: 
>
>  Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log 
> file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 
> seconds (Mailbox was synchronized) 
>
>  or the same message with (rotating while syncing). 
>
>  There must be something badly wrong in my setup, or perhaps in the spec 
>  of the server that dovecot is running on. What should I look at to 
>  start understanding this better? 
>
>  Many thanks - Neil 
>
> Can you tell a bit more about your environment? Sounds like io issue 

Thanks for your reply.  I'm not sure exactly what you have in mind, but
here are some starting points:

arudy:~# uname -a
Linux arudy 4.13.0-1-686-pae #1 SMP Debian 4.13.4-2 (2017-10-15) i686 GNU/Linux

arudy:~# dovecot -n
# 2.2.32 (dfbe293d4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.20 (7cd71ba)
# OS: Linux 4.13.0-1-686-pae i686 Debian buster/sid 
auth_mechanisms = plain login
auth_username_format = %Ln
auth_verbose = yes
login_trusted_networks = 192.168.11.8
mail_access_groups = mail
mail_fsync = never
mail_location = mbox:~/dovecot-mail:INBOX=/var/mail/%u
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix = 
}
passdb {
  driver = pam
}
plugin {
  antispam_backend = dspam
  antispam_dspam_args = --deliver;--user;%u
  antispam_dspam_binary = /usr/bin/dspam
  antispam_signature = X-DSPAM-Signature
  antispam_signature_missing = error
  antispam_spam = Spam
  antispam_trash = trash;Trash;Deleted Items; Deleted Messages
  fts = solr
  fts_solr = url=http://localhost:8080/solr/
  sieve = file:~/sieve;active=~/.dovecot.sieve
}
postmaster_address = postmas...@ossau.homelinux.net
protocols = " imap lmtp"
service auth {
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0666
user = postfix
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
  }
}
ssl_cert = 

Re: Really slow IMAP performance

[Aki Tuomi]

 
 
  
   
  
  
   
On 24 February 2018 at 12:45 Neil Jerram <
n...@ossau.homelinux.net> wrote:
   
   

   
   

   
   
Please could you help me to understand and fix why my dovecot IMAP
   
   
performance is so bad? I've read through a lot of the
   
   
performance-related material on the website, but I don't think that any
   
   
of it could account for slowness at the level that I am seeing.
   
   

   
   
The simplest scenario is moving a message from my Inbox to another IMAP
   
   
folder. Using Gnus as the client, the whole UI freezes for about 2
   
   
minutes (which I assume is until the move is complete), and journalctl
   
   
on the dovecot server says:
   
   

   
   
Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds (Mailbox was synchronized)
   
   

   
   
or the same message with (rotating while syncing).
   
   

   
   
There must be something badly wrong in my setup, or perhaps in the spec
   
   
of the server that dovecot is running on. What should I look at to
   
   
start understanding this better?
   
   

   
   
Many thanks - Neil
   
  
  
   
  
  
   Can you tell a bit more about your environment? Sounds like io issue
  
  
   ---
   Aki Tuomi
   
 

Really slow IMAP performance

[Neil Jerram]

Please could you help me to understand and fix why my dovecot IMAP
performance is so bad?  I've read through a lot of the
performance-related material on the website, but I don't think that any
of it could account for slowness at the level that I am seeing.

The simplest scenario is moving a message from my Inbox to another IMAP
folder.  Using Gnus as the client, the whole UI freezes for about 2
minutes (which I assume is until the move is complete), and journalctl
on the dovecot server says:

Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file 
/home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds 
(Mailbox was synchronized)

or the same message with (rotating while syncing).

There must be something badly wrong in my setup, or perhaps in the spec
of the server that dovecot is running on.  What should I look at to
start understanding this better?

Many thanks - Neil