Re: [ext] 2.3.7 slower than 2.3.6?

[Ralf Hildebrandt via dovecot]

> > All of the data is an /dev/sda1
> 
> What filesystem is this?

ext4
 
> I did a bunch of testing, and after initially thinking I saw an
> increase it was just due to bad testing because the new TCP_NODELAY
> changes allowed imaptest to do more work. So I can't see that there is
> any disk IO difference.
> 
> There are a few intentional changes to lib-index and also mdbox
> code. One of those is that dovecot.index.cache files are being
> recreated more often now. In some cases they were wasting a lot of
> disk space because expunged mails hadn't been removed from them. I
> guess it's a possibility that all the disk IO was because in your
> system Dovecot started recreating all those files at the same time.
> Maybe you could try upgrading again during slower hours and see if the
> disk IO normalizes after a while?

Still running 2.3.7. 

Looking at the disk IO graphs today's IO usage looks OK!

I restarted dovecot yesterday after fixing the FTS settings:
Jul 16 10:14:45 mail-cbf dovecot: master: Dovecot v2.3.7 (494d20bdc) starting 
up for imap, lmtp (core dumps disabled)

after that I saw quite a bit of IO on sda1 (a bit of data read, lot of write),
which subsided at11:30

After that diskio was low, up to the present day.

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | https://www.charite.de

Re: [ext] 2.3.7 slower than 2.3.6?

[Timo Sirainen via dovecot]

On 16 Jul 2019, at 11.15, Ralf Hildebrandt via dovecot  
wrote:
> 
> * Timo Sirainen via dovecot :
> 
>>> And alas, I had a look at the monitoring and found that disk IO has
>>> increase A LOT after the update: 
>>> https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
>>> 
>>> I zoomed in and found that the sudden increace in IO coincides with
>>> the update to 2.3.7 (14:35): 
>>> https://www.arschkrebs.de/images/dovecot-io-2.png 
>>> 
>> 
> 
>> Do the different disks have different kinds of data? 
> 
> All of the data is an /dev/sda1

What filesystem is this?

I did a bunch of testing, and after initially thinking I saw an increase it was 
just due to bad testing because the new TCP_NODELAY changes allowed imaptest to 
do more work. So I can't see that there is any disk IO difference.

There are a few intentional changes to lib-index and also mdbox code. One of 
those is that dovecot.index.cache files are being recreated more often now. In 
some cases they were wasting a lot of disk space because expunged mails hadn't 
been removed from them. I guess it's a possibility that all the disk IO was 
because in your system Dovecot started recreating all those files at the same 
time. Maybe you could try upgrading again during slower hours and see if the 
disk IO normalizes after a while?

Re: [ext] 2.3.7 slower than 2.3.6?

[Ralf Hildebrandt via dovecot]

* Timo Sirainen via dovecot :

> > And alas, I had a look at the monitoring and found that disk IO has
> > increase A LOT after the update: 
> > https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
> > 
> > I zoomed in and found that the sudden increace in IO coincides with
> > the update to 2.3.7 (14:35): 
> > https://www.arschkrebs.de/images/dovecot-io-2.png 
> > 
> 

> Do the different disks have different kinds of data? 

All of the data is an /dev/sda1

> Like you seem to be using external attachments and fts? 

Jup, using that, but on one disk

> Also your doveconf -n
> doesn't show what fts backend you're using. Or is it just
> half-configured and not really used?

I used to use lucene, but then I had issues building it
and commented it out. 

Looking now, it turns out that your packages come with lucene support...
Reenabling:

mail_plugins = zlib fts fts_lucene

...

plugin {
  fts = lucene
  fts_autoindex = yes
  fts_languages = de,en
  fts_lucene = whitespace_chars=@.
  zlib_save = gz
  zlib_save_level = 5
}



-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | https://www.charite.de

Re: [ext] 2.3.7 slower than 2.3.6?

[Timo Sirainen via dovecot]

On 15 Jul 2019, at 23.17, Ralf Hildebrandt via dovecot  
wrote:
> 
> * Ralf Hildebrandt via dovecot :
>> We're using dovecot (via LMTP) as a backup for all incoming mail.
>> 
>> I upgraded from 2.3.6 to 2.3.7 on the 12th:
>> 2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 
>> 2:2.3.7-8~bionic
>> 
>> and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup
>> IMAP box is suddenly taking quite some time to get delivered and is piling up
>> in the queue.
> 
> And alas, I had a look at the monitoring and found that disk IO has
> increase A LOT after the update: 
> https://www.arschkrebs.de/images/dovecot-disk-io-increase.png
> 
> I zoomed in and found that the sudden increace in IO coincides with
> the update to 2.3.7 (14:35): 
> https://www.arschkrebs.de/images/dovecot-io-2.png 
> 

Do the different disks have different kinds of data? Like you seem to be using 
external attachments and fts? Also your doveconf -n doesn't show what fts 
backend you're using. Or is it just half-configured and not really used?

Re: [ext] 2.3.7 slower than 2.3.6?

[Ralf Hildebrandt via dovecot]

* Ralf Hildebrandt via dovecot :
> We're using dovecot (via LMTP) as a backup for all incoming mail.
> 
> I upgraded from 2.3.6 to 2.3.7 on the 12th:
> 2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 
> 2:2.3.7-8~bionic
> 
> and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup
> IMAP box is suddenly taking quite some time to get delivered and is piling up
> in the queue.

And alas, I had a look at the monitoring and found that disk IO has
increase A LOT after the update: 
https://www.arschkrebs.de/images/dovecot-disk-io-increase.png

I zoomed in and found that the sudden increace in IO coincides with
the update to 2.3.7 (14:35): https://www.arschkrebs.de/images/dovecot-io-2.png

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | https://www.charite.de

2.3.7 slower than 2.3.6?

[Ralf Hildebrandt via dovecot]

We're using dovecot (via LMTP) as a backup for all incoming mail.

I upgraded from 2.3.6 to 2.3.7 on the 12th:
2019-07-12 14:35:44 upgrade dovecot-imapd:amd64 2:2.3.6-2~bionic 
2:2.3.7-8~bionic

and it seems that 2.3.7 is slower than 2.3.6 -- mail to the backup
IMAP box is suddenly taking quite some time to get delivered and is piling up
in the queue.

doveconf -n:


default_vsz_limit = 2 G
lmtp_user_concurrency_limit = 1
mail_attachment_dir = /home/copymail/attachments
mail_fsync = never
mail_location = mdbox:~/mdbox
mail_plugins = zlib fts
mdbox_rotate_size = 128 M
... namespaces ...
plugin {
  fts_autoindex = yes
  fts_languages = de,en
  zlib_save = gz
  zlib_save_level = 5
}
protocols = " imap lmtp"
userdb {
  args = username_format=%u /etc/dovecot/passwd
  driver = passwd-file
}
protocol lmtp {
  mail_plugins = zlib fts
}
  

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | https://www.charite.de