Re: Assertion during dsync receive

[Tanstaafl]

On Tue Mar 13 2018 08:12:29 GMT-0400 (Eastern Standard Time), Aki Tuomi
 wrote:
> On 13.03.2018 14:10, Tanstaafl wrote:
>> Yes... per folder? Or per user?
> Sorry, ment per folder. =)

Heh, no worries, ok, thanks, nothing to worry about so much then.

Re: Assertion during dsync receive

[Aki Tuomi]

On 13.03.2018 14:10, Tanstaafl wrote:
> On Tue Mar 13 2018 07:42:31 GMT-0400 (Eastern Standard Time), Aki Tuomi
>  wrote:
>>> Is this index file kept on a per *folder* basis? Or per user/account?
>> Yes
> Yes... per folder? Or per user?
Sorry, ment per folder. =)

>>> If it is per folder, then it is much less of a problem, and could be
>>> worked around simply by moving messages around (say, archiving).
>> It can help. But if you are not accessing all the mails e.g. migrating
>> them, you can usually just rm the cache file and it'll probably won't
>> grow fast back.
> So, how would doing this impact performance? If it doesn't much, then
> maybe a simple script to test the size and delete when it gets larger
> than (whatever you like)?
>
> Thanks Aki!

Re: Assertion during dsync receive

[Tanstaafl]

On Tue Mar 13 2018 07:42:31 GMT-0400 (Eastern Standard Time), Aki Tuomi
 wrote:
>> Is this index file kept on a per *folder* basis? Or per user/account?

> Yes

Yes... per folder? Or per user?

>> If it is per folder, then it is much less of a problem, and could be
>> worked around simply by moving messages around (say, archiving).

> It can help. But if you are not accessing all the mails e.g. migrating
> them, you can usually just rm the cache file and it'll probably won't
> grow fast back.

So, how would doing this impact performance? If it doesn't much, then
maybe a simple script to test the size and delete when it gets larger
than (whatever you like)?

Thanks Aki!

Re: Assertion during dsync receive

[Aki Tuomi]

On 13.03.2018 13:40, Tanstaafl wrote:
> On Sat Feb 24 2018 11:32:05 GMT-0500 (Eastern Daylight Time), Aki Tuomi
>  wrote:
>>> On 23 February 2018 at 23:12 Tanstaafl  wrote:
>>> Ok, so, I'm still unclear...
>>>
>>> what cache are we talking about? Are you saying that there is a limit to
>>> how many emails dovecot can store in a single... 'folder'?
>> Dovecot has cache for commonly fetched fields. This is called
>> dovecot.index.cache. The maximum file size for the cache is about 4G,
>> the maximum offset is 0x4000 * 4, because all offsets are divided by
>> 4. If the cache file becomes full, there are things you can do..
>>
>> 1. you can rm the cache file and hope it does not get full again too
>> fast. 2. remove some mails
> Ok, so... I'm *still* unclear, and please understand I'm not trying to
> be difficult, but I want/need to understand this. One thing not helping
> is I don't have a working dovecot setup I can access to go peek at the
> filesystem, which would probably let me answer this for myself.
>
> When you say 'commonly fetched fields', can you confirm this means it is
> simply the number of emails that can cause the problem?
Yes

> Is this index file kept on a per *folder* basis? Or per user/account?

Yes

> If it is per folder, then it is much less of a problem, and could be
> worked around simply by moving messages around (say, archiving).
It can help. But if you are not accessing all the mails e.g. migrating
them, you can usually just rm the cache file and it'll probably won't
grow fast back.

> Thanks again


Aki

Re: Assertion during dsync receive

[Tanstaafl]

On Sat Feb 24 2018 11:32:05 GMT-0500 (Eastern Daylight Time), Aki Tuomi
 wrote:
>> On 23 February 2018 at 23:12 Tanstaafl  wrote:
>> Ok, so, I'm still unclear...
>>
>> what cache are we talking about? Are you saying that there is a limit to
>> how many emails dovecot can store in a single... 'folder'?

> Dovecot has cache for commonly fetched fields. This is called
> dovecot.index.cache. The maximum file size for the cache is about 4G,
> the maximum offset is 0x4000 * 4, because all offsets are divided by
> 4. If the cache file becomes full, there are things you can do..
> 
> 1. you can rm the cache file and hope it does not get full again too
> fast. 2. remove some mails

Ok, so... I'm *still* unclear, and please understand I'm not trying to
be difficult, but I want/need to understand this. One thing not helping
is I don't have a working dovecot setup I can access to go peek at the
filesystem, which would probably let me answer this for myself.

When you say 'commonly fetched fields', can you confirm this means it is
simply the number of emails that can cause the problem?

Is this index file kept on a per *folder* basis? Or per user/account?

If it is per folder, then it is much less of a problem, and could be
worked around simply by moving messages around (say, archiving).

Thanks again

Re: Assertion during dsync receive

[Aki Tuomi]

> On 23 February 2018 at 23:12 Tanstaafl  wrote:
> 
> 
> On Fri Feb 23 2018 14:53:53 GMT-0500 (Eastern Standard Time), Aki Tuomi
>  wrote:
> > It is problem for any mailbox
> 
> Ok, so, I'm still unclear...
> 
> what cache are we talking about? Are you saying that there is a limit to
> how many emails dovecot can store in a single... 'folder'?

Dovecot has cache for commonly fetched fields. This is called 
dovecot.index.cache. The maximum file size for the cache is about 4G, the 
maximum offset is 0x4000 * 4, because all offsets are divided by 4. If the 
cache file becomes full, there are things you can do..

1. you can rm the cache file and hope it does not get full again too fast.
2. remove some mails

Aki

Re: Assertion during dsync receive

[Tanstaafl]

On Fri Feb 23 2018 14:53:53 GMT-0500 (Eastern Standard Time), Aki Tuomi
 wrote:
> It is problem for any mailbox

Ok, so, I'm still unclear...

what cache are we talking about? Are you saying that there is a limit to
how many emails dovecot can store in a single... 'folder'?

Re: Assertion during dsync receive

[Aki Tuomi]

It is problem for any mailbox


---Aki TuomiDovecot oy
 Original message From: Tanstaafl <tansta...@libertytrek.org> 
Date: 23/02/2018  21:36  (GMT+02:00) To: dovecot@dovecot.org Subject: Re: 
Assertion during dsync receive 
On Fri Feb 23 2018 13:53:27 GMT-0500 (Eastern Standard Time), Aki Tuomi
<aki.tu...@dovecot.fi> wrote:
> Once you cache grows bigger than 0x400 you have problems

This is for a single mailbox? IS this only a problem for mbox and maybe
sdbox?

Re: Assertion during dsync receive

[Tanstaafl]

On Fri Feb 23 2018 13:53:27 GMT-0500 (Eastern Standard Time), Aki Tuomi
 wrote:
> Once you cache grows bigger than 0x400 you have problems

This is for a single mailbox? IS this only a problem for mbox and maybe
sdbox?

Re: Assertion during dsync receive

[Aki Tuomi]

Once you cache grows bigger than 0x400 you have problems


---Aki TuomiDovecot oy
 Original message From: Ian Bobbitt <ibobb...@globalnoc.iu.edu> 
Date: 23/02/2018  20:33  (GMT+02:00) To: dovecot@dovecot.org Subject: Re: 
Assertion during dsync receive 

Thanks. I've had the user clear out that mailbox, and replication is
working fine for them again.



Is there a better way to catch this than watch for crashes and read
the backtrace to find what mailbox needs to be shrunk?



Where is the threshold for "too big"?



-- Ian



On 2/23/18 11:33 AM, Aki Tuomi wrote:


  
  The mailbox is too big.
  

  
  

  
  

  
  

---
Aki Tuomi
Dovecot oy
  
  

  
  
 Original message 
From: Ian Bobbitt <ibobb...@globalnoc.iu.edu> 
Date: 23/02/2018 17:52 (GMT+02:00) 
To: dovecot@dovecot.org 
Subject: Assertion during dsync receive 



  
  Hi,

  

  I'm getting an assertion failed on the receiving side, causing
  syncs to fail for one user. The servers are setup so that

  only one is receiving any traffic other than replication at any
  time. The one that's only receiving replications is the

  one that's failing.

  

  I've tried deleting the user's home on the receiving server, but
  it still crashes during the sync. Oddly, the user's

  home is 7.4G on the sending server, but ends up at 42G on the
  receiving side, even after deleting and trying a fresh sync.

  

  The mailbox implicated in the backtrace ("Spam") does have a very
  large number of messages in it.

  On sender:

  Spam messages=1217764 recent=0 uidnext=1218103
  uidvalidity=1379509105 unseen=16 highestmodseq=744588
  vsize=34468460093

  guid=090ed93a7a09abf10200fdf6807a firstsaved=1498744186

  On receiver:

  Spam messages=1217766 recent=352 uidnext=1218105
  uidvalidity=1379509105 unseen=16 highestmodseq=744589
  vsize=34468496809

  guid=090ed93a7a09abf10200fdf6807a firstsaved=1519396172

  

  Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: fscking
  index file

  /gnoc/mail/home/bgeels/mail/storage/dovecot.map.index

  Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: mdbox
  /gnoc/mail/home/bgeels/mail/storage: rebuilding indexes

  Feb 23 14:57:33 dovecot: dsync-local(bgeels): Panic: file
  mail-index-util.c: line 10 (mail_index_uint32_to_offset):

  assertion failed: (offset < 0x4000)

  Feb 23 14:57:33 dovecot: dsync-local(bgeels): Error: Raw
  backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x9f3de)

  [0x7feb584143de] ->
  /usr/lib64/dovecot/libdovecot.so.0(+0x9f4be) [0x7feb584144be]
  ->

  /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb583a577c]
  ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_uint32_to_offset+0xa0)
  [0x7feb587906d0] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mail_cache_compress+0x854)
  [0x7feb58774f34] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x25f)
  [0x7feb587884ff] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10de)
  [0x7feb5870b3ae] ->

  /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x858)
  [0x7feb5870ccd8] ->

  /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync+0x4c)
  [0x7feb5870ce7c] ->

/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x4b)
  [0x7feb5870cf3b] ->

  /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44)
  [0x7feb586f2834] ->

  /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37)
  [0x7feb586f28d7] ->

  dovecot/doveadm-server(dsync_mailbox_import_deinit+0x475)
  [0x445495] -> dovecot/doveadm-server() [0x43edc0] ->

  dovecot/doveadm-server(dsync_brain_sync_mails+0x743) [0x43f653]
  -> dovecot/doveadm-server(dsync_brain_run+0x541)

  [0x43acf1] -> dovecot/doveadm-server() [0x43b070] ->
  dovecot/doveadm-server() [0x44fe5f] ->

  /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52)
  [0x7feb58429cd2] ->

/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f)
  [0x7feb5842b3bf] ->

  /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c)
  [0x7feb58429d6c] ->

  /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38)
  [0x7feb58429f28] -> dovecot/doveadm-server() [0x4209c5] ->

  dovecot/doveadm-server() [0x422df6] -> dovecot/doveadm-server()
  [0x4377f4] ->

  /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52)
  [0x7feb58429cd2] ->

/usr/lib64/dovecot/libdovecot.so.0(io_loop_ha

Re: Assertion during dsync receive

[Ian Bobbitt]

Thanks. I've had the user clear out that mailbox, and replication is working 
fine for them again.

Is there a better way to catch this than watch for crashes and read the 
backtrace to find what mailbox needs to be shrunk?

Where is the threshold for "too big"?

-- Ian

On 2/23/18 11:33 AM, Aki Tuomi wrote:
> The mailbox is too big.
>
>
>
> ---
> Aki Tuomi
> Dovecot oy
>
>  Original message 
> From: Ian Bobbitt 
> Date: 23/02/2018 17:52 (GMT+02:00)
> To: dovecot@dovecot.org
> Subject: Assertion during dsync receive
>
> Hi,
>
> I'm getting an assertion failed on the receiving side, causing syncs to fail 
> for one user. The servers are setup so that
> only one is receiving any traffic other than replication at any time. The one 
> that's only receiving replications is the
> one that's failing.
>
> I've tried deleting the user's home on the receiving server, but it still 
> crashes during the sync. Oddly, the user's
> home is 7.4G on the sending server, but ends up at 42G on the receiving side, 
> even after deleting and trying a fresh sync.
>
> The mailbox implicated in the backtrace ("Spam") does have a very large 
> number of messages in it.
> On sender:
> Spam messages=1217764 recent=0 uidnext=1218103 uidvalidity=1379509105 
> unseen=16 highestmodseq=744588 vsize=34468460093
> guid=090ed93a7a09abf10200fdf6807a firstsaved=1498744186
> On receiver:
> Spam messages=1217766 recent=352 uidnext=1218105 uidvalidity=1379509105 
> unseen=16 highestmodseq=744589 vsize=34468496809
> guid=090ed93a7a09abf10200fdf6807a firstsaved=1519396172
>
> Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: fscking index file
> /gnoc/mail/home/bgeels/mail/storage/dovecot.map.index
> Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: mdbox 
> /gnoc/mail/home/bgeels/mail/storage: rebuilding indexes
> Feb 23 14:57:33 dovecot: dsync-local(bgeels): Panic: file mail-index-util.c: 
> line 10 (mail_index_uint32_to_offset):
> assertion failed: (offset < 0x4000)
> Feb 23 14:57:33 dovecot: dsync-local(bgeels): Error: Raw backtrace: 
> /usr/lib64/dovecot/libdovecot.so.0(+0x9f3de)
> [0x7feb584143de] -> /usr/lib64/dovecot/libdovecot.so.0(+0x9f4be) 
> [0x7feb584144be] ->
> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb583a577c] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_uint32_to_offset+0xa0) 
> [0x7feb587906d0] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_cache_compress+0x854) 
> [0x7feb58774f34] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x25f) 
> [0x7feb587884ff] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10de)
>  [0x7feb5870b3ae] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x858) 
> [0x7feb5870ccd8] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync+0x4c) [0x7feb5870ce7c] 
> ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x4b) 
> [0x7feb5870cf3b] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44) 
> [0x7feb586f2834] ->
> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37) 
> [0x7feb586f28d7] ->
> dovecot/doveadm-server(dsync_mailbox_import_deinit+0x475) [0x445495] -> 
> dovecot/doveadm-server() [0x43edc0] ->
> dovecot/doveadm-server(dsync_brain_sync_mails+0x743) [0x43f653] -> 
> dovecot/doveadm-server(dsync_brain_run+0x541)
> [0x43acf1] -> dovecot/doveadm-server() [0x43b070] -> dovecot/doveadm-server() 
> [0x44fe5f] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f) 
> [0x7feb5842b3bf] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c] 
> ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7feb58429f28] -> 
> dovecot/doveadm-server() [0x4209c5] ->
> dovecot/doveadm-server() [0x422df6] -> dovecot/doveadm-server() [0x4377f4] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f) 
> [0x7feb5842b3bf] ->
> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c]
> Feb 23 14:57:33 dsync-local(bgeels): Fatal: master: service(doveadm): child 
> 82098 killed with signal 6 (core dumped)
>
> I've attached the output of `doveconf -n` and the full backtrace from a core 
> dump.
>
> Dovecot 2.2.33.2 (GhettoForge package)
> CentOS 7 x86_64
> XFS, no NFS.
>
>
> -- Ian



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Assertion during dsync receive

[Aki Tuomi]

The mailbox is too big.


---Aki TuomiDovecot oy
 Original message From: Ian Bobbitt  
Date: 23/02/2018  17:52  (GMT+02:00) To: dovecot@dovecot.org Subject: Assertion 
during dsync receive 
Hi,

I'm getting an assertion failed on the receiving side, causing syncs to fail 
for one user. The servers are setup so that
only one is receiving any traffic other than replication at any time. The one 
that's only receiving replications is the
one that's failing.

I've tried deleting the user's home on the receiving server, but it still 
crashes during the sync. Oddly, the user's
home is 7.4G on the sending server, but ends up at 42G on the receiving side, 
even after deleting and trying a fresh sync.

The mailbox implicated in the backtrace ("Spam") does have a very large number 
of messages in it.
On sender:
Spam messages=1217764 recent=0 uidnext=1218103 uidvalidity=1379509105 unseen=16 
highestmodseq=744588 vsize=34468460093
guid=090ed93a7a09abf10200fdf6807a firstsaved=1498744186
On receiver:
Spam messages=1217766 recent=352 uidnext=1218105 uidvalidity=1379509105 
unseen=16 highestmodseq=744589 vsize=34468496809
guid=090ed93a7a09abf10200fdf6807a firstsaved=1519396172

Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: fscking index file
/gnoc/mail/home/bgeels/mail/storage/dovecot.map.index
Feb 23 14:57:20 dovecot: dsync-local(bgeels): Warning: mdbox 
/gnoc/mail/home/bgeels/mail/storage: rebuilding indexes
Feb 23 14:57:33 dovecot: dsync-local(bgeels): Panic: file mail-index-util.c: 
line 10 (mail_index_uint32_to_offset):
assertion failed: (offset < 0x4000)
Feb 23 14:57:33 dovecot: dsync-local(bgeels): Error: Raw backtrace: 
/usr/lib64/dovecot/libdovecot.so.0(+0x9f3de)
[0x7feb584143de] -> /usr/lib64/dovecot/libdovecot.so.0(+0x9f4be) 
[0x7feb584144be] ->
/usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7feb583a577c] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_uint32_to_offset+0xa0) 
[0x7feb587906d0] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_cache_compress+0x854) 
[0x7feb58774f34] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mail_index_sync_commit+0x25f) 
[0x7feb587884ff] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_rebuild_in_context+0x10de)
 [0x7feb5870b3ae] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync_begin+0x858) 
[0x7feb5870ccd8] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_sync+0x4c) [0x7feb5870ce7c] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mdbox_storage_sync_init+0x4b) 
[0x7feb5870cf3b] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync_init+0x44) 
[0x7feb586f2834] ->
/usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_sync+0x37) [0x7feb586f28d7] 
->
dovecot/doveadm-server(dsync_mailbox_import_deinit+0x475) [0x445495] -> 
dovecot/doveadm-server() [0x43edc0] ->
dovecot/doveadm-server(dsync_brain_sync_mails+0x743) [0x43f653] -> 
dovecot/doveadm-server(dsync_brain_run+0x541)
[0x43acf1] -> dovecot/doveadm-server() [0x43b070] -> dovecot/doveadm-server() 
[0x44fe5f] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f) 
[0x7feb5842b3bf] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7feb58429f28] -> 
dovecot/doveadm-server() [0x4209c5] ->
dovecot/doveadm-server() [0x422df6] -> dovecot/doveadm-server() [0x4377f4] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x52) [0x7feb58429cd2] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0x10f) 
[0x7feb5842b3bf] ->
/usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x3c) [0x7feb58429d6c]
Feb 23 14:57:33 dsync-local(bgeels): Fatal: master: service(doveadm): child 
82098 killed with signal 6 (core dumped)

I've attached the output of `doveconf -n` and the full backtrace from a core 
dump.

Dovecot 2.2.33.2 (GhettoForge package)
CentOS 7 x86_64
XFS, no NFS.


-- Ian