Re: Really slow IMAP performance

[Tanstaafl]

On Sat Feb 24 2018 17:01:01 GMT-0500 (Eastern Standard Time), @lbutlr
 wrote:
> On 2018-02-24 (07:14 MST), Aki Tuomi  wrote:
>>
>> https://wiki2.dovecot.org/Migration/MailFormat
> 
> That didn't show up when searching wiki2 for "Migration" :/

Don't search 'wiki2', search just wiki.dovecot.org

I never liked the way Timo rolled out the wiki for the new version 2
when he did, I knew it would do nothing but create confusion...

Really, he should just redirect all references to wiki2 to wiki and kill
the old content...

Re: Really slow IMAP performance

[Joseph Tam]

On Sat, 24 Feb 2018, Neil Jerram wrote:


My INBOX file has 22990 messages.  Is the slowness that I am seeing
definitely expected for an mbox of that size?  (It may also be relevant
that the HDD it's stored on is pretty old now, and has been known to
report SMART errors...)


Yeah, a copy of that mailbox will be that slow, esp. if the messages
have large attachments.  Even a simple operation like deleting/expunging the
the first message will cause data shuffling of the entire mailbox.

Joseph Tam 

Re: Really slow IMAP performance

[@lbutlr]

On 2018-02-24 (07:14 MST), Aki Tuomi  wrote:
> 
> https://wiki2.dovecot.org/Migration/MailFormat

That didn't show up when searching wiki2 for "Migration" :/


-- 
"...Life is not a journey to the grave with the intention of arriving
safely in one pretty and well-preserved piece, but to slide across the
finish line broadside, thoroughly used up, worn out, leaking oil, and
shouting GERONIMO!!!" -- Bill McKenna

Re: Really slow IMAP performance

[@lbutlr]

On 2018-02-24 (07:04 MST), Neil Jerram  wrote:
> 
> My INBOX file has 22990 messages.  Is the slowness that I am seeing
> definitely expected for an mbox of that size?  (It may also be relevant
> that the HDD it's stored on is pretty old now, and has been known to
> report SMART errors...)


back int he dark ages I would send and alert to users if their inbox for over 
1,000 messages because an mbox that large drove the server to its knees and 
made merely logging in to mail take an excruciating amount of time. If they 
didn't fix it I'd archive their inbox and start over.

I am astonished your machine can process an mbox with over 22 times that many 
messages.

https://wiki1.dovecot.org/Migration/MailFormat

That page doesn't exist on the wiki for dovecot 2, but that script to convert 
mbox to milder should still work. Obviously, keep backups and such.

-- 
Did they get you to trade your heroes for ghosts? Hot ashes for trees?
Hot air for a cool breeze? Cold comfort for change?

Re: Really slow IMAP performance

[Neil Jerram]

Aki Tuomi  writes:

> Yes. You deffo are looking at several reasons for slowness.
>
> I can only recommend moving into maildir or sdbox format, and probably new 
> HDD too.
>
> https://wiki2.dovecot.org/Tools/Doveadm/Sync here is example of 'converting' 
> between mailbox formats using dsync. You should also read 
> https://wiki2.dovecot.org/Migration/MailFormat
>
> mbox format has been known to act up with dsync occasionally, so I recommend 
> using 
>
> doveadm backup maildir:~/Maildir

Thanks, I've done that now, and things are looking much better.

I rediscovered that I've configured postfix to deliver locally using
dovecot-lmtp - which meant that I then only needed to change dovecot's
mail_location setting, and nothing at all in the postfix config.

Many thanks! - Neil

Re: Really slow IMAP performance

[Håkon Alstadheim]

Den 24. feb. 2018 15:04, skrev Neil Jerram:
> Aki Tuomi  writes:
> 
>>> On 24 February 2018 at 15:47 Neil Jerram  wrote:
> 
> [...]
  Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning:
 Transaction log file
 /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked
 for 98 seconds (Mailbox was synchronized)
> [...]
> 
>> You are using mbox format. This is ... bit slow. =)
>>
>> When you move mails between mbox files, it has to rewrite the entire mbox 
>> file every time. You should probably start using maildir or sdbox instead.
> 
> Ah, right, thanks.
> 
> My INBOX file has 22990 messages.  Is the slowness that I am seeing
> definitely expected for an mbox of that size?  (It may also be relevant
> that the HDD it's stored on is pretty old now, and has been known to
> report SMART errors...)
> 
> If so, I'll start looking at how to migrate, given that my system is
> Postfix + Dovecot.  If you have any particular recommendations or
> migration pointers for a system like that, I'd appreciate them.
> 

I'd go with whatever tools you are familiar with. If you don't know
where to start, formail(1) can read an mbox and do whatever for each
mail contained therein. This, together with procmail, used to be the
go-to tools in the days before IMAP.

Theese days you'd probably want to involve your local delivery agent on
the output from formail. The lda would invoke sieve instead of procmail
if that is your thing.

Whatever you do, try to set up so you can do some tests before you blast
22000 mails to somewhere you do not want them :-)

Re: Really slow IMAP performance

[Neil Jerram]

Aki Tuomi  writes:

>> On 24 February 2018 at 15:47 Neil Jerram  wrote:

[...]
>> >  Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning:
>> > Transaction log file
>> > /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked
>> > for 98 seconds (Mailbox was synchronized)
[...]

> You are using mbox format. This is ... bit slow. =)
>
> When you move mails between mbox files, it has to rewrite the entire mbox 
> file every time. You should probably start using maildir or sdbox instead.

Ah, right, thanks.

My INBOX file has 22990 messages.  Is the slowness that I am seeing
definitely expected for an mbox of that size?  (It may also be relevant
that the HDD it's stored on is pretty old now, and has been known to
report SMART errors...)

If so, I'll start looking at how to migrate, given that my system is
Postfix + Dovecot.  If you have any particular recommendations or
migration pointers for a system like that, I'd appreciate them.

Best wishes - Neil

Re: Really slow IMAP performance

[Aki Tuomi]

> On 24 February 2018 at 15:47 Neil Jerram  wrote:
> 
> 
> Aki Tuomi  writes:
> 
> >  On 24 February 2018 at 12:45 Neil Jerram < n...@ossau.homelinux.net> 
> > wrote: 
> >
> >  Please could you help me to understand and fix why my dovecot IMAP 
> >  performance is so bad? I've read through a lot of the 
> >  performance-related material on the website, but I don't think that any 
> >  of it could account for slowness at the level that I am seeing. 
> >
> >  The simplest scenario is moving a message from my Inbox to another IMAP 
> >  folder. Using Gnus as the client, the whole UI freezes for about 2 
> >  minutes (which I assume is until the move is complete), and journalctl 
> >  on the dovecot server says: 
> >
> >  Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log 
> > file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 
> > 98 seconds (Mailbox was synchronized) 
> >
> >  or the same message with (rotating while syncing). 
> >
> >  There must be something badly wrong in my setup, or perhaps in the spec 
> >  of the server that dovecot is running on. What should I look at to 
> >  start understanding this better? 
> >
> >  Many thanks - Neil 
> >
> > Can you tell a bit more about your environment? Sounds like io issue 
> 
> Thanks for your reply.  I'm not sure exactly what you have in mind, but
> here are some starting points:
> 
> arudy:~# uname -a
> Linux arudy 4.13.0-1-686-pae #1 SMP Debian 4.13.4-2 (2017-10-15) i686 
> GNU/Linux
> 
> arudy:~# dovecot -n
> # 2.2.32 (dfbe293d4): /etc/dovecot/dovecot.conf
> # Pigeonhole version 0.4.20 (7cd71ba)
> # OS: Linux 4.13.0-1-686-pae i686 Debian buster/sid 
> auth_mechanisms = plain login
> auth_username_format = %Ln
> auth_verbose = yes
> login_trusted_networks = 192.168.11.8
> mail_access_groups = mail
> mail_fsync = never
> mail_location = mbox:~/dovecot-mail:INBOX=/var/mail/%u

You are using mbox format. This is ... bit slow. =)

When you move mails between mbox files, it has to rewrite the entire mbox file 
every time. You should probably start using maildir or sdbox instead.

Aki

Re: Really slow IMAP performance

[Neil Jerram]

Aki Tuomi  writes:

>  On 24 February 2018 at 12:45 Neil Jerram < n...@ossau.homelinux.net> wrote: 
>
>  Please could you help me to understand and fix why my dovecot IMAP 
>  performance is so bad? I've read through a lot of the 
>  performance-related material on the website, but I don't think that any 
>  of it could account for slowness at the level that I am seeing. 
>
>  The simplest scenario is moving a message from my Inbox to another IMAP 
>  folder. Using Gnus as the client, the whole UI freezes for about 2 
>  minutes (which I assume is until the move is complete), and journalctl 
>  on the dovecot server says: 
>
>  Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log 
> file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 
> seconds (Mailbox was synchronized) 
>
>  or the same message with (rotating while syncing). 
>
>  There must be something badly wrong in my setup, or perhaps in the spec 
>  of the server that dovecot is running on. What should I look at to 
>  start understanding this better? 
>
>  Many thanks - Neil 
>
> Can you tell a bit more about your environment? Sounds like io issue 

Thanks for your reply.  I'm not sure exactly what you have in mind, but
here are some starting points:

arudy:~# uname -a
Linux arudy 4.13.0-1-686-pae #1 SMP Debian 4.13.4-2 (2017-10-15) i686 GNU/Linux

arudy:~# dovecot -n
# 2.2.32 (dfbe293d4): /etc/dovecot/dovecot.conf
# Pigeonhole version 0.4.20 (7cd71ba)
# OS: Linux 4.13.0-1-686-pae i686 Debian buster/sid 
auth_mechanisms = plain login
auth_username_format = %Ln
auth_verbose = yes
login_trusted_networks = 192.168.11.8
mail_access_groups = mail
mail_fsync = never
mail_location = mbox:~/dovecot-mail:INBOX=/var/mail/%u
namespace inbox {
  inbox = yes
  location = 
  mailbox Drafts {
special_use = \Drafts
  }
  mailbox Junk {
special_use = \Junk
  }
  mailbox Sent {
special_use = \Sent
  }
  mailbox "Sent Messages" {
special_use = \Sent
  }
  mailbox Trash {
special_use = \Trash
  }
  prefix = 
}
passdb {
  driver = pam
}
plugin {
  antispam_backend = dspam
  antispam_dspam_args = --deliver;--user;%u
  antispam_dspam_binary = /usr/bin/dspam
  antispam_signature = X-DSPAM-Signature
  antispam_signature_missing = error
  antispam_spam = Spam
  antispam_trash = trash;Trash;Deleted Items; Deleted Messages
  fts = solr
  fts_solr = url=http://localhost:8080/solr/
  sieve = file:~/sieve;active=~/.dovecot.sieve
}
postmaster_address = postmas...@ossau.homelinux.net
protocols = " imap lmtp"
service auth {
  unix_listener /var/spool/postfix/private/auth {
group = postfix
mode = 0666
user = postfix
  }
}
service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
group = postfix
mode = 0600
user = postfix
  }
}
ssl_cert = 

Re: Really slow IMAP performance

[Aki Tuomi]

 
 
  
   
  
  
   
On 24 February 2018 at 12:45 Neil Jerram <
n...@ossau.homelinux.net> wrote:
   
   

   
   

   
   
Please could you help me to understand and fix why my dovecot IMAP
   
   
performance is so bad? I've read through a lot of the
   
   
performance-related material on the website, but I don't think that any
   
   
of it could account for slowness at the level that I am seeing.
   
   

   
   
The simplest scenario is moving a message from my Inbox to another IMAP
   
   
folder. Using Gnus as the client, the whole UI freezes for about 2
   
   
minutes (which I assume is until the move is complete), and journalctl
   
   
on the dovecot server says:
   
   

   
   
Feb 24 10:24:24 arudy dovecot[1712]: imap(neil): Warning: Transaction log file /home/neil/dovecot-mail/.imap/INBOX/dovecot.index.log was locked for 98 seconds (Mailbox was synchronized)
   
   

   
   
or the same message with (rotating while syncing).
   
   

   
   
There must be something badly wrong in my setup, or perhaps in the spec
   
   
of the server that dovecot is running on. What should I look at to
   
   
start understanding this better?
   
   

   
   
Many thanks - Neil
   
  
  
   
  
  
   Can you tell a bit more about your environment? Sounds like io issue
  
  
   ---
   Aki Tuomi