Re: Inconsistency in map index with dovecot v2.3.21

[Oscar del Rio]

Apologies for the formatting. Retrying with HTML formatting disabled:

We are getting the same warnings on 2 separate test servers; one with 
local ZFS storage, the other one with NFS storage (single server, data 
not shared with any other servers).
We are imapsync'ing a few hundred user accounts from the production 
server so there is a lot of emails transfer a day. The warnings occur at 
random times and with random users.


I reinstalled 2.3.21 from source (originally was using Dovecot repo 
packages) and tested by reverting that commit. The warnings continued.


Next I tried with Dovecot 2.3.20 and still got the same warnings after a 
while.


# 2.3.20 (80a5ac675d): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.20 (149edcf2)
# OS: Linux 5.4.0-171-generic x86_64 Ubuntu 20.04.6 LTS nfs4

Feb 13 15:38:25 dovecot: imap-login: Login: user=, 
method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=272469, TLS, 
session=
Feb 13 15:38:27 dovecot: imap(username)<272469>: 
Warning: mdbox /mail/spool/username/storage: Inconsistency in map index 
(3,40 != 3,56)
Feb 13 15:38:27 dovecot: imap(username)<272469>: 
Warning: fscking index file /mail/spool/username/storage/dovecot.map.index
Feb 13 15:38:27 dovecot: imap(username)<272469>: 
Warning: mdbox /mail/spool/username/storage: rebuilding indexes
Feb 13 15:38:29 dovecot: imap(username)<272469>: 
Error: Mailbox INBOX: 
/mail/spool/username/mailboxes/INBOX/dbox-Mails/dovecot.index reset, 
view is now inconsistent
Feb 13 15:38:29 dovecot: imap(username)<272469>: 
Warning: fscking index file /mail/spool/username/storage/dovecot.map.index


I will try with an even older version to continue testing...

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Inconsistency in map index with dovecot v2.3.21

[Oscar del Rio]

On 2024-02-10 2:24 a.m., Timo Sirainen wrote:
 On 12. Jan 2024, at 9.37, John Alex. via dovecot
  wrote:
  since I upgraded dovecot to version 2.3.21 on a FreeBSD
  13.2 system, I've been seeing a few occurences of the
  following warnings daily:

  Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp
  (user1)<8209>: Warning: mdbox /
  var/vmail/users/user1/mdbox/storage: Inconsistency in map
  index (5,40 != 5,56)
  Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp
  (user1)<8209>: Warning: fscking
  index file /var/vmail/users/user1/mdbox/storage/
  dovecot.map.index
  Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp
  (user1)<8209>: Warning: mdbox /
  var/vmail/users/user1/mdbox/storage: rebuilding indexes
  Jan 11 13:13:26 dimokritos dovecot[33170]: lmtp
  (user1)<8209>: Warning: fscking
  index file /var/vmail/users/user1/mdbox/storage/
  dovecot.map.index
 Hmm. I can't seem to be able to reproduce it easily at least. Looking
 at the code changes, I do see now that there is one related change
 that may be wrong, or at least the commit message is wrong: https://
 github.com/dovecot/core/commit/
 aae93d3ea73004ba1883bbf91c9e40f613ffc857 - although it shouldn't
 affect anything unless there are already some other errors. But maybe
 there's something I don't see right now. Can you try if the errors go
 away by reverting this change?
We are getting the same warnings on 2 separate test servers; one with local ZFS
storage, the other one with NFS storage (single server, data not shared with
any other servers).
We are imapsync'ing a few hundred user accounts from the production server so
there is a lot of emails transfer a day. The warnings occur at random times and
with random users.
I reinstalled 2.3.21 from source (originally was using Dovecot repo packages)
and tested by reverting that commit. The warnings continued.
Next I tried with Dovecot 2.3.20 and still got the same warnings after a while.
# 2.3.20 (80a5ac675d): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.20 (149edcf2)
# OS: Linux 5.4.0-171-generic x86_64 Ubuntu 20.04.6 LTS nfs4
Feb 13 15:38:25 dovecot: imap-login: Login: user=, method=PLAIN,
rip=127.0.0.1, lip=127.0.0.1, mpid=272469, TLS, session=
Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning:
mdbox /mail/spool/username/storage: Inconsistency in map index (3,40 != 3,56)
Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning:
fscking index file /mail/spool/username/storage/dovecot.map.index
Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning:
mdbox /mail/spool/username/storage: rebuilding indexes
Feb 13 15:38:29 dovecot: imap(username)<272469>: Error:
Mailbox INBOX: /mail/spool/username/mailboxes/INBOX/dbox-Mails/dovecot.index
reset, view is now inconsistent
Feb 13 15:38:29 dovecot: imap(username)<272469>: Warning:
fscking index file /mail/spool/username/storage/dovecot.map.index
I will try with an even older version to continue testing...
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Inconsistency in map index with dovecot v2.3.21

[Timo Sirainen]

On 12. Jan 2024, at 9.37, John Alex. via dovecot  wrote:
> 
> Hi,
> 
> since I upgraded dovecot to version 2.3.21 on a FreeBSD 13.2 system, I've 
> been seeing a few occurences of the following warnings daily:
> 
> Jan 11 13:13:25 dimokritos dovecot[33170]: 
> lmtp(user1)<8209>: Warning: mdbox 
> /var/vmail/users/user1/mdbox/storage: Inconsistency in map index (5,40 != 
> 5,56)
> Jan 11 13:13:25 dimokritos dovecot[33170]: 
> lmtp(user1)<8209>: Warning: fscking index file 
> /var/vmail/users/user1/mdbox/storage/dovecot.map.index
> Jan 11 13:13:25 dimokritos dovecot[33170]: 
> lmtp(user1)<8209>: Warning: mdbox 
> /var/vmail/users/user1/mdbox/storage: rebuilding indexes
> Jan 11 13:13:26 dimokritos dovecot[33170]: 
> lmtp(user1)<8209>: Warning: fscking index file 
> /var/vmail/users/user1/mdbox/storage/dovecot.map.index

Hmm. I can't seem to be able to reproduce it easily at least. Looking at the 
code changes, I do see now that there is one related change that may be wrong, 
or at least the commit message is wrong: 
https://github.com/dovecot/core/commit/aae93d3ea73004ba1883bbf91c9e40f613ffc857 
- although it shouldn't affect anything unless there are already some other 
errors. But maybe there's something I don't see right now. Can you try if the 
errors go away by reverting this change?

> Should I be worried about data loss due to these index inconsistencies?

No data loss, but it can lead to inefficient rebuilds.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Inconsistency in map index with dovecot v2.3.21

[caio]

I have the same problem after updating dovecot 2.3.21, I am receiving the 
errors below for the accounts when I perform backups.

doveadm(u...@user.com.br): Error: Purging namespace '' failed: BUG: Unknown 
internal error
Error: failed sync dir: doveadm(us...@user2.com.br): Warning: mdbox 
/data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2.com.br/mdbox/storage:
 Inconsistency in map index (42,40 != 42,56)
doveadm(us...@user2.com.br): Warning: fscking index file 
/data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2/mdbox/storage/dovecot.map.index
doveadm(us...@user2.com.br): Warning: mdbox 
/data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2.com.br/mdbox/storage:
 rebuilding indexes
doveadm(us...@user2.com.br): Warning: Transaction log file 
/data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2/mdbox/storage/dovecot.map.index.log
 was locked for 284 seconds (mdbox storage rebuild)
doveadm(us...@user2.com.br): Error: Purging namespace '' failed: BUG: Unknown 
internal error

As in the case of our friend the other day, the error no longer happens on the 
same account, it happens on another random one.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Inconsistency in map index with dovecot v2.3.21

[John Alex. via dovecot]

Hi,

since I upgraded dovecot to version 2.3.21 on a FreeBSD 13.2 system, I've been seeing a 
few occurences of the following warnings daily:


Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp(user1)<8209>: 
Warning: mdbox /var/vmail/users/user1/mdbox/storage: Inconsistency in map index (5,40 != 5,56)
Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp(user1)<8209>: 
Warning: fscking index file /var/vmail/users/user1/mdbox/storage/dovecot.map.index
Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp(user1)<8209>: 
Warning: mdbox /var/vmail/users/user1/mdbox/storage: rebuilding indexes
Jan 11 13:13:26 dimokritos dovecot[33170]: lmtp(user1)<8209>: 
Warning: fscking index file /var/vmail/users/user1/mdbox/storage/dovecot.map.index


The numbers in parentheses seem to always be of the format (x,40 != x,56) or 
(x,40 != x,72).

I also get such errors on some users when purging (doveadm-purge fails with 
error 75):

doveadm(user2): Warning: mdbox /var/vmail/users/user2/mdbox/storage: Inconsistency in map 
index (5,40 != 5,72)
doveadm(user2): Warning: fscking index file 
/var/vmail/users/user2/mdbox/storage/dovecot.map.index

doveadm(user2): Warning: mdbox /var/vmail/users/user2/mdbox/storage: rebuilding 
indexes
doveadm(user2): Error: Purging namespace '' failed: BUG: Unknown internal error

If I rerun doveadm purge for this user it completes successfully.

Can anyone shed some light on what might be causing these messages? I didn't have these 
with 2.3.20. Should I be worried about data loss due to these index inconsistencies?

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org

Re: Inconsistency in map index

[Webert de Souza Lima]

On Fri, Aug 18, 2017 at 1:30 PM, Timo Sirainen  wrote:

>
> That would work. Also as a different workaround you could just rm
> storage/dovecot.map.index* and doveadm force-resync -u user@domain '*'.



Thank you Timo, that did the trick without the need of a sudden upgrade.
Mailbox was fixed :)


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*

>
>

Re: Inconsistency in map index

[Timo Sirainen]

On 18 Aug 2017, at 16.43, Webert de Souza Lima  wrote:
> 
> On Fri, Aug 18, 2017 at 9:03 AM, Aki Tuomi  wrote:
> 
>> This is fixed in next release (2.2.32) with
>> https://github.com/dovecot/core/commit/c8be394
>> 
>> Aki Tuomi
>> 
> 
> As this is still a release candidate, I'm thinking of running an isolated
> instance of this version, and do doveadm force-resync just to fix just this
> user's mailbox.
> What do you think?

That would work. Also as a different workaround you could just rm 
storage/dovecot.map.index* and doveadm force-resync -u user@domain '*'. Should 
be also safe to do (won't lose any information), although I guess it has some 
potential of race conditions causing temporary problems if the user accesses 
mails at the same time.

Re: Inconsistency in map index

[Webert de Souza Lima]

On Fri, Aug 18, 2017 at 9:03 AM, Aki Tuomi  wrote:

> This is fixed in next release (2.2.32) with
> https://github.com/dovecot/core/commit/c8be394
>
> Aki Tuomi
>

As this is still a release candidate, I'm thinking of running an isolated
instance of this version, and do doveadm force-resync just to fix just this
user's mailbox.
What do you think?

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*

Re: Inconsistency in map index

[Webert de Souza Lima]

Oh, so that's likely a bug.
I was thinking it would require manual intervention to fix.

Great, I'll do an upgrade ASAP. Praised be Docker.

Thank you very much.


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*

On Fri, Aug 18, 2017 at 9:03 AM, Aki Tuomi <aki.tu...@dovecot.fi> wrote:

>
>
> On 18.08.2017 14:55, Webert de Souza Lima wrote:
> > Hello,
> >
> > The following errors are constantly popping up for 2 accounts. I can't
> get
> > it fixed,
> > I did doveadm backup to another account, the same happens in the new
> > account.
> > I did doveadm force-resync, the problem persists.
> >
> > I'm using dovecot 2.2.
> >
> > 2017-08-18T11:46:12.472821881Z Aug 18 11:46:12 lmtp(
> ramon.lace...@alliar.com):
> > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> > Inconsistency in map index (647,6288 != 647,28333584)
> >
> > 2017-08-18T11:46:12.651002372Z Aug 18 11:46:12 lmtp(
> ramon.lace...@alliar.com):
> > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> > Inconsistency in map index (647,6288 != 647,28333708)
> >
> > 2017-08-18T11:46:12.651059432Z Aug 18 11:46:12 lmtp(
> ramon.lace...@alliar.com):
> > Warning: fscking index file /srv/dovecot2/index/
> > alliar.com/ramon.lacerda/storage/dovecot.map.index
> >
> > 2017-08-18T11:46:12.764926940Z Aug 18 11:46:12 lmtp(
> ramon.lace...@alliar.com):
> > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> > rebuilding indexes
> >
> >
> > Regards,
> >
> > Webert Lima
> > DevOps Engineer at MAV Tecnologia
> > *Belo Horizonte - Brasil*
>
> Hi!
>
> This is fixed in next release (2.2.32) with
> https://github.com/dovecot/core/commit/c8be394
>
> Aki Tuomi
>

Re: Inconsistency in map index

[Aki Tuomi]

On 18.08.2017 14:55, Webert de Souza Lima wrote:
> Hello,
>
> The following errors are constantly popping up for 2 accounts. I can't get
> it fixed,
> I did doveadm backup to another account, the same happens in the new
> account.
> I did doveadm force-resync, the problem persists.
>
> I'm using dovecot 2.2.
>
> 2017-08-18T11:46:12.472821881Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
> Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> Inconsistency in map index (647,6288 != 647,28333584)
>
> 2017-08-18T11:46:12.651002372Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
> Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> Inconsistency in map index (647,6288 != 647,28333708)
>
> 2017-08-18T11:46:12.651059432Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
> Warning: fscking index file /srv/dovecot2/index/
> alliar.com/ramon.lacerda/storage/dovecot.map.index
>
> 2017-08-18T11:46:12.764926940Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
> Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> rebuilding indexes
>
>
> Regards,
>
> Webert Lima
> DevOps Engineer at MAV Tecnologia
> *Belo Horizonte - Brasil*

Hi!

This is fixed in next release (2.2.32) with
https://github.com/dovecot/core/commit/c8be394

Aki Tuomi

Inconsistency in map index

[Webert de Souza Lima]

Hello,

The following errors are constantly popping up for 2 accounts. I can't get
it fixed,
I did doveadm backup to another account, the same happens in the new
account.
I did doveadm force-resync, the problem persists.

I'm using dovecot 2.2.

2017-08-18T11:46:12.472821881Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
Inconsistency in map index (647,6288 != 647,28333584)

2017-08-18T11:46:12.651002372Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
Inconsistency in map index (647,6288 != 647,28333708)

2017-08-18T11:46:12.651059432Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
Warning: fscking index file /srv/dovecot2/index/
alliar.com/ramon.lacerda/storage/dovecot.map.index

2017-08-18T11:46:12.764926940Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
rebuilding indexes


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*

mdbox Inconsistency in map index

[bOnK]

After a (power)crash two accounts have been corrupted.
I tried to rescue things by running force-resync multiple times, but 
that didn't work out.
Searching the archives, I found a recent suggestion (by Timo) to delete 
the index files, but I'm not sure which files to delete and what the 
consequences will be.
When I deleted ``storage/dovecot.map.index'' in an unimportant account 
of my own, things only grew worse and I decided to delete the account 
and start fresh...


Something I obviously don't want to do with the other account as it 
belongs to a client with about 2.5G mails (856 m.* files), which we 
don't want to lose.


I have observed the following behavior in the logs:
When a new mail arrives:
(The second line below is always the same, however there's no mail with 
uid 12819 according to doveadm search uid 12819)
Mar 04 11:43:28 lda(user): Warning: mdbox /var/mail/user/storage: 
Inconsistency in map index (117,1060 != 117,1883660)
Mar 04 11:43:28 lda(user): Error: Log synchronization error at 
seq=117,offset=1546660 for /var/mail/user/storage/dovecot.map.index: 
Extension record inc drops number below zero (uid=12819, diff=-1, orig=0)
Mar 04 11:43:28 lda(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:43:28 lda(user): Info: 
msgid=<d760229d563e99dc787acd0746b18...@feedback.1dayfly.net>: saved 
mail to INBOX
Mar 04 11:43:28 lda(user): Warning: mdbox /var/mail/user/storage: 
Inconsistency in map index (117,1060 != 117,1883784)
Mar 04 11:43:28 lda(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:43:28 lda(user): Warning: mdbox /var/mail/user/storage: 
rebuilding indexes
Mar 04 11:43:28 lda(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:43:28 imap(user): Error: 
/var/mail/user/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is 
now inconsistent
Mar 04 11:43:28 imap(user): Info: IMAP session state is inconsistent, 
please relogin. in=2011 out=108358 deleted=0 expunged=0 trashed=0


When user logs in next (might be from another box/MUA)
Mar 04 11:44:36 imap-login: Info: Login: user=, method=PLAIN, 
rip=82.95.XX.XX, lip=37.97.XX.XX, mpid=14884, TLS, 
session=<VfvML+ZJuO9SX18R>
Mar 04 11:44:37 imap(user): Warning: mdbox /var/mail/user/storage: 
Inconsistency in map index (117,1060 != 117,1884336)
Mar 04 11:44:37 imap(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:44:37 imap(user): Warning: mdbox /var/mail/user/storage: 
rebuilding indexes
Mar 04 11:44:37 imap(user): Error: 
/var/mail/user/mailboxes/INBOX/dbox-Mails/dovecot.index reset, view is 
now inconsistent
Mar 04 11:44:37 imap(user): Info: IMAP session state is inconsistent, 
please relogin. in=331 out=178391 deleted=0 expunged=0 trashed=0
Mar 04 11:44:37 imap(user): Warning: mdbox /var/mail/user/storage: 
Inconsistency in map index (117,1060 != 117,1884396)
Mar 04 11:44:37 imap(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:44:37 imap(user): Warning: fscking index file 
/var/mail/user/storage/dovecot.map.index
Mar 04 11:44:37 imap-login: Info: Login: user=, method=PLAIN, 
rip=82.95.XX.XX, lip=37.97.XX.XX, mpid=14886, TLS, 
session=<AUTbL+ZJuu9SX18R>


Besides the nuisance of getting kicked of every time a new mail arrives, 
user cannot delete mails (as in: move to Trash).


TL;DR: which files to delete and what will be the consequences?

--
b.