Re: Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

[Stefan Neben]

Hi,

> Unfortunately dovecot has an in-built limit of
> 2GB for the index cache file

okay. I assumed there is a limit but not that low, thought about 4GB
because of this:

https://www.dovecot.org/list/dovecot/2015-September/101863.html

And the strange thing is, i had this problem yesterday twice. The
first time the index cache was 1.2GB and the second time it was 1.1GB.
Is there a second factor, which could limit this?

> Your only solution is to move some of the mails to some other folder.

Yes, this was my fallback solution, but this means a lot of work to
adapt the 3rd party tooling around it :-(

2018-01-11 12:33 GMT+01:00 Aki Tuomi :
>
>
> On 10.01.2018 22:00, Stefan Neben wrote:
>> Hello,
>>
>> i have a problem with a very huge mail account.
>>
>> The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
>> from time to time the index cache make problems and i got the
>> following in the logs:
>>
>> --- Snip ---
>> Jan 10 20:43:04 X dovecot: imap(X): Panic: file
>> mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
>> failed: (offset < 0x4000)
>> --- Snap ---
>>
>> I skip the raw backtrace to keep the post "small", but can hand it in
>> later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...
>>
>> The size of the file dovecot.index.cache of INBOX was in the last seen
>> problem situations between 1.1GB to 2.3GB.
>>
>> To solve the problem the last times was to reduce the amount of mails
>> to under 1.7 million (seems to be the limit to trigger the problem),
>> delete the index cache and let him rebuild with a controled request.
>> But this is a odd situation (because it confused 3rd party tooling,
>> which access this mailbox) and i cannot find any helpful informations
>> to solve this problem.
>>
>> Sharding it into several folders to keep the amount of mails in INBOX
>> small is not an option currently.
>>
>> If further informations are needed, just let me know. Thanks for any help!
>>
>>
>> Kind regards
>> Stefan
> Hi!
>
> Unfortunately dovecot has an in-built limit of 2GB for the index cache
> file. Your only solution is to move some of the mails to some other folder.
>
> Aki

Re: Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

[Aki Tuomi]

On 10.01.2018 22:00, Stefan Neben wrote:
> Hello,
>
> i have a problem with a very huge mail account.
>
> The INBOX of this account has ~1.1 - ~1.7 million mails inside. And
> from time to time the index cache make problems and i got the
> following in the logs:
>
> --- Snip ---
> Jan 10 20:43:04 X dovecot: imap(X): Panic: file
> mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion
> failed: (offset < 0x4000)
> --- Snap ---
>
> I skip the raw backtrace to keep the post "small", but can hand it in
> later if needed. Got this for UID FETCH, UID SEARCH, UID SORT, ...
>
> The size of the file dovecot.index.cache of INBOX was in the last seen
> problem situations between 1.1GB to 2.3GB.
>
> To solve the problem the last times was to reduce the amount of mails
> to under 1.7 million (seems to be the limit to trigger the problem),
> delete the index cache and let him rebuild with a controled request.
> But this is a odd situation (because it confused 3rd party tooling,
> which access this mailbox) and i cannot find any helpful informations
> to solve this problem.
>
> Sharding it into several folders to keep the amount of mails in INBOX
> small is not an option currently.
>
> If further informations are needed, just let me know. Thanks for any help!
>
>
> Kind regards
> Stefan
Hi!

Unfortunately dovecot has an in-built limit of 2GB for the index cache
file. Your only solution is to move some of the mails to some other folder.

Aki

Re: Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

[Stefan Neben]

Hi,

> You are storing 1.7 million e-mails in a single mbox file.

no, of course not:

mdbox_rotate_size = 64 M

> I would rather store the archive using one file per e-mail.

I think one file per mail would be not good for this setup. One
keypoint is that the INBOX is searchable (in a acceptable amount of
time). Further this probably could slow down the backup process (i
think).

2018-01-11 0:46 GMT+01:00 Rupert Gallagher :
> You are storing 1.7 million e-mails in a single mbox file. I would rather
> store the archive using one file per e-mail.

Re: Panic: file mail-index-util.c: line 37 (mail_index_uint32_to_offset): assertion failed: (offset < 0x40000000)

[Rupert Gallagher]

You are storing 1.7 million e-mails in a single mbox file. I would rather store 
the archive using one file per e-mail.