Re: Inconsistency in map index with dovecot v2.3.21

2024-02-13 Thread Oscar del Rio

Apologies for the formatting. Retrying with HTML formatting disabled:

We are getting the same warnings on 2 separate test servers; one with 
local ZFS storage, the other one with NFS storage (single server, data 
not shared with any other servers).
We are imapsync'ing a few hundred user accounts from the production 
server so there is a lot of emails transfer a day. The warnings occur at 
random times and with random users.


I reinstalled 2.3.21 from source (originally was using Dovecot repo 
packages) and tested by reverting that commit. The warnings continued.


Next I tried with Dovecot 2.3.20 and still got the same warnings after a 
while.


# 2.3.20 (80a5ac675d): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.20 (149edcf2)
# OS: Linux 5.4.0-171-generic x86_64 Ubuntu 20.04.6 LTS nfs4

Feb 13 15:38:25 dovecot: imap-login: Login: user=, 
method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mpid=272469, TLS, 
session=
Feb 13 15:38:27 dovecot: imap(username)<272469>: 
Warning: mdbox /mail/spool/username/storage: Inconsistency in map index 
(3,40 != 3,56)
Feb 13 15:38:27 dovecot: imap(username)<272469>: 
Warning: fscking index file /mail/spool/username/storage/dovecot.map.index
Feb 13 15:38:27 dovecot: imap(username)<272469>: 
Warning: mdbox /mail/spool/username/storage: rebuilding indexes
Feb 13 15:38:29 dovecot: imap(username)<272469>: 
Error: Mailbox INBOX: 
/mail/spool/username/mailboxes/INBOX/dbox-Mails/dovecot.index reset, 
view is now inconsistent
Feb 13 15:38:29 dovecot: imap(username)<272469>: 
Warning: fscking index file /mail/spool/username/storage/dovecot.map.index


I will try with an even older version to continue testing...

___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: Inconsistency in map index with dovecot v2.3.21

2024-02-13 Thread Oscar del Rio
On 2024-02-10 2:24 a.m., Timo Sirainen wrote:
 On 12. Jan 2024, at 9.37, John Alex. via dovecot
  wrote:
  since I upgraded dovecot to version 2.3.21 on a FreeBSD
  13.2 system, I've been seeing a few occurences of the
  following warnings daily:

  Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp
  (user1)<8209>: Warning: mdbox /
  var/vmail/users/user1/mdbox/storage: Inconsistency in map
  index (5,40 != 5,56)
  Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp
  (user1)<8209>: Warning: fscking
  index file /var/vmail/users/user1/mdbox/storage/
  dovecot.map.index
  Jan 11 13:13:25 dimokritos dovecot[33170]: lmtp
  (user1)<8209>: Warning: mdbox /
  var/vmail/users/user1/mdbox/storage: rebuilding indexes
  Jan 11 13:13:26 dimokritos dovecot[33170]: lmtp
  (user1)<8209>: Warning: fscking
  index file /var/vmail/users/user1/mdbox/storage/
  dovecot.map.index
 Hmm. I can't seem to be able to reproduce it easily at least. Looking
 at the code changes, I do see now that there is one related change
 that may be wrong, or at least the commit message is wrong: https://
 github.com/dovecot/core/commit/
 aae93d3ea73004ba1883bbf91c9e40f613ffc857 - although it shouldn't
 affect anything unless there are already some other errors. But maybe
 there's something I don't see right now. Can you try if the errors go
 away by reverting this change?
We are getting the same warnings on 2 separate test servers; one with local ZFS
storage, the other one with NFS storage (single server, data not shared with
any other servers).
We are imapsync'ing a few hundred user accounts from the production server so
there is a lot of emails transfer a day. The warnings occur at random times and
with random users.
I reinstalled 2.3.21 from source (originally was using Dovecot repo packages)
and tested by reverting that commit. The warnings continued.
Next I tried with Dovecot 2.3.20 and still got the same warnings after a while.
# 2.3.20 (80a5ac675d): /usr/local/etc/dovecot/dovecot.conf
# Pigeonhole version 0.5.20 (149edcf2)
# OS: Linux 5.4.0-171-generic x86_64 Ubuntu 20.04.6 LTS nfs4
Feb 13 15:38:25 dovecot: imap-login: Login: user=, method=PLAIN,
rip=127.0.0.1, lip=127.0.0.1, mpid=272469, TLS, session=
Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning:
mdbox /mail/spool/username/storage: Inconsistency in map index (3,40 != 3,56)
Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning:
fscking index file /mail/spool/username/storage/dovecot.map.index
Feb 13 15:38:27 dovecot: imap(username)<272469>: Warning:
mdbox /mail/spool/username/storage: rebuilding indexes
Feb 13 15:38:29 dovecot: imap(username)<272469>: Error:
Mailbox INBOX: /mail/spool/username/mailboxes/INBOX/dbox-Mails/dovecot.index
reset, view is now inconsistent
Feb 13 15:38:29 dovecot: imap(username)<272469>: Warning:
fscking index file /mail/spool/username/storage/dovecot.map.index
I will try with an even older version to continue testing...
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: Inconsistency in map index with dovecot v2.3.21

2024-02-09 Thread Timo Sirainen
On 12. Jan 2024, at 9.37, John Alex. via dovecot  wrote:
> 
> Hi,
> 
> since I upgraded dovecot to version 2.3.21 on a FreeBSD 13.2 system, I've 
> been seeing a few occurences of the following warnings daily:
> 
> Jan 11 13:13:25 dimokritos dovecot[33170]: 
> lmtp(user1)<8209>: Warning: mdbox 
> /var/vmail/users/user1/mdbox/storage: Inconsistency in map index (5,40 != 
> 5,56)
> Jan 11 13:13:25 dimokritos dovecot[33170]: 
> lmtp(user1)<8209>: Warning: fscking index file 
> /var/vmail/users/user1/mdbox/storage/dovecot.map.index
> Jan 11 13:13:25 dimokritos dovecot[33170]: 
> lmtp(user1)<8209>: Warning: mdbox 
> /var/vmail/users/user1/mdbox/storage: rebuilding indexes
> Jan 11 13:13:26 dimokritos dovecot[33170]: 
> lmtp(user1)<8209>: Warning: fscking index file 
> /var/vmail/users/user1/mdbox/storage/dovecot.map.index

Hmm. I can't seem to be able to reproduce it easily at least. Looking at the 
code changes, I do see now that there is one related change that may be wrong, 
or at least the commit message is wrong: 
https://github.com/dovecot/core/commit/aae93d3ea73004ba1883bbf91c9e40f613ffc857 
- although it shouldn't affect anything unless there are already some other 
errors. But maybe there's something I don't see right now. Can you try if the 
errors go away by reverting this change?

> Should I be worried about data loss due to these index inconsistencies?

No data loss, but it can lead to inefficient rebuilds.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: Inconsistency in map index with dovecot v2.3.21

2024-02-09 Thread caio
I have the same problem after updating dovecot 2.3.21, I am receiving the 
errors below for the accounts when I perform backups.

doveadm(u...@user.com.br): Error: Purging namespace '' failed: BUG: Unknown 
internal error
Error: failed sync dir: doveadm(us...@user2.com.br): Warning: mdbox 
/data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2.com.br/mdbox/storage:
 Inconsistency in map index (42,40 != 42,56)
doveadm(us...@user2.com.br): Warning: fscking index file 
/data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2/mdbox/storage/dovecot.map.index
doveadm(us...@user2.com.br): Warning: mdbox 
/data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2.com.br/mdbox/storage:
 rebuilding indexes
doveadm(us...@user2.com.br): Warning: Transaction log file 
/data/data/mailboxbackup/20240207/user2.com.br/u/ser/user2/mdbox/storage/dovecot.map.index.log
 was locked for 284 seconds (mdbox storage rebuild)
doveadm(us...@user2.com.br): Error: Purging namespace '' failed: BUG: Unknown 
internal error

As in the case of our friend the other day, the error no longer happens on the 
same account, it happens on another random one.
___
dovecot mailing list -- dovecot@dovecot.org
To unsubscribe send an email to dovecot-le...@dovecot.org


Re: Inconsistency in map index

2017-08-18 Thread Webert de Souza Lima
On Fri, Aug 18, 2017 at 1:30 PM, Timo Sirainen  wrote:

>
> That would work. Also as a different workaround you could just rm
> storage/dovecot.map.index* and doveadm force-resync -u user@domain '*'.



Thank you Timo, that did the trick without the need of a sudden upgrade.
Mailbox was fixed :)


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*

>
>


Re: Inconsistency in map index

2017-08-18 Thread Timo Sirainen
On 18 Aug 2017, at 16.43, Webert de Souza Lima  wrote:
> 
> On Fri, Aug 18, 2017 at 9:03 AM, Aki Tuomi  wrote:
> 
>> This is fixed in next release (2.2.32) with
>> https://github.com/dovecot/core/commit/c8be394
>> 
>> Aki Tuomi
>> 
> 
> As this is still a release candidate, I'm thinking of running an isolated
> instance of this version, and do doveadm force-resync just to fix just this
> user's mailbox.
> What do you think?

That would work. Also as a different workaround you could just rm 
storage/dovecot.map.index* and doveadm force-resync -u user@domain '*'. Should 
be also safe to do (won't lose any information), although I guess it has some 
potential of race conditions causing temporary problems if the user accesses 
mails at the same time.


Re: Inconsistency in map index

2017-08-18 Thread Webert de Souza Lima
On Fri, Aug 18, 2017 at 9:03 AM, Aki Tuomi  wrote:

> This is fixed in next release (2.2.32) with
> https://github.com/dovecot/core/commit/c8be394
>
> Aki Tuomi
>

As this is still a release candidate, I'm thinking of running an isolated
instance of this version, and do doveadm force-resync just to fix just this
user's mailbox.
What do you think?

Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*


Re: Inconsistency in map index

2017-08-18 Thread Webert de Souza Lima
Oh, so that's likely a bug.
I was thinking it would require manual intervention to fix.

Great, I'll do an upgrade ASAP. Praised be Docker.

Thank you very much.


Regards,

Webert Lima
DevOps Engineer at MAV Tecnologia
*Belo Horizonte - Brasil*

On Fri, Aug 18, 2017 at 9:03 AM, Aki Tuomi  wrote:

>
>
> On 18.08.2017 14:55, Webert de Souza Lima wrote:
> > Hello,
> >
> > The following errors are constantly popping up for 2 accounts. I can't
> get
> > it fixed,
> > I did doveadm backup to another account, the same happens in the new
> > account.
> > I did doveadm force-resync, the problem persists.
> >
> > I'm using dovecot 2.2.
> >
> > 2017-08-18T11:46:12.472821881Z Aug 18 11:46:12 lmtp(
> ramon.lace...@alliar.com):
> > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> > Inconsistency in map index (647,6288 != 647,28333584)
> >
> > 2017-08-18T11:46:12.651002372Z Aug 18 11:46:12 lmtp(
> ramon.lace...@alliar.com):
> > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> > Inconsistency in map index (647,6288 != 647,28333708)
> >
> > 2017-08-18T11:46:12.651059432Z Aug 18 11:46:12 lmtp(
> ramon.lace...@alliar.com):
> > Warning: fscking index file /srv/dovecot2/index/
> > alliar.com/ramon.lacerda/storage/dovecot.map.index
> >
> > 2017-08-18T11:46:12.764926940Z Aug 18 11:46:12 lmtp(
> ramon.lace...@alliar.com):
> > Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> > rebuilding indexes
> >
> >
> > Regards,
> >
> > Webert Lima
> > DevOps Engineer at MAV Tecnologia
> > *Belo Horizonte - Brasil*
>
> Hi!
>
> This is fixed in next release (2.2.32) with
> https://github.com/dovecot/core/commit/c8be394
>
> Aki Tuomi
>


Re: Inconsistency in map index

2017-08-18 Thread Aki Tuomi


On 18.08.2017 14:55, Webert de Souza Lima wrote:
> Hello,
>
> The following errors are constantly popping up for 2 accounts. I can't get
> it fixed,
> I did doveadm backup to another account, the same happens in the new
> account.
> I did doveadm force-resync, the problem persists.
>
> I'm using dovecot 2.2.
>
> 2017-08-18T11:46:12.472821881Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
> Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> Inconsistency in map index (647,6288 != 647,28333584)
>
> 2017-08-18T11:46:12.651002372Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
> Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> Inconsistency in map index (647,6288 != 647,28333708)
>
> 2017-08-18T11:46:12.651059432Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
> Warning: fscking index file /srv/dovecot2/index/
> alliar.com/ramon.lacerda/storage/dovecot.map.index
>
> 2017-08-18T11:46:12.764926940Z Aug 18 11:46:12 lmtp(ramon.lace...@alliar.com):
> Warning: mdbox /srv/dovecot2/mail/alliar.com/ramon.lacerda/storage:
> rebuilding indexes
>
>
> Regards,
>
> Webert Lima
> DevOps Engineer at MAV Tecnologia
> *Belo Horizonte - Brasil*

Hi!

This is fixed in next release (2.2.32) with
https://github.com/dovecot/core/commit/c8be394

Aki Tuomi