Quoting Guilhem Moulin (2019-11-05 19:48:57) > Hi Jonas, > > On Tue, 05 Nov 2019 at 13:55:59 +0100, Jonas Smedegaard wrote: > > $ interimap --config hb INBOX > > remote: ERROR: UIDVALIDITY changed! (1571588814 != 1154884797) Need to > > invalidate the UID cache. > > > > ..and it seems the --repair option doesn't do its job: > > I guess naming that command ‘--repair’ wasn't the best choice; it's > not meant to reconcile UIDVALIDITY mismatch anyway, but to perform a > so-called “full synchronization” à la offlineimap to compare the > UID+flag list between the local and remote mailboxes. (Unless there > is a bug in interimap or the IMAPd's QRESYNC implementation there > shouldn't be anything to repair, but it's still useful for integrity > checking etc.)
Ah, sorry for misunderstanding its purpose. Then I suspect this is no bug but a support session of my fooling around with delicate imap details... > IMAP4rev1 allows servers without mechanism for storing persistent UID > and UIDVALIDITY values. However synchronizing such server is out of > scope for interimap(1), and I guess other IMAP-based synchronization > software, cf. RFC 3501 sec. 2.3.1.1: “Persistent unique identifiers > are required for a client to resynchronize its state from a previous > session with the server […] The combination of mailbox name, > UIDVALIDITY, and UID must refer to a single immutable message on that > server forever.” > > There is nothing interimap can do so resolve UIDVALIDITY conflicts. > It needs to invalidate its cache, because per RFC 3501 the lUID ↔ rUID > mapping can't be relied upon anymore. (Rebuilding the mapping by > comparing message RFC 5355 data is not reliable either, because IMAP > does allow duplicates.) That was my understanding as well - so I was puzzled about the magic of --repair option when I first saw it documented, but now that I needed it I prayed for the magic to be real :-) > > local: S: * STATUS INBOX (UIDNEXT 290008 UIDVALIDITY 1571588814 > > HIGHESTMODSEQ 71671) > > […] > > remote: S: * STATUS INBOX (UIDNEXT 280739 UIDVALIDITY 1571588814 > > HIGHESTMODSEQ 74538) > > […] > > remote: ERROR: UIDVALIDITY changed! (1571588814 != 1154884797) Need to > > invalidate the UID cache. > > I find it suspicious that both mailboxes have the same UIDVALIDITY > value. By RFC 3501 that value is an arbitrary unsigned >0 32-bits > integer. That value (like UID values) are set by the server and can't > be chosen or altered by the client (otherwise interimap wouldn't need > a database to keep track of the lUID ↔ rUID mapping, a mere state file > would be sufficient). Dovecot doesn't pick it at random, but AFAICT > uses the timestamp at which the mailbox was created. > > It looks kind of weird to me that both your local and remote INBOX > appears to have been created at the very same time (Oct 20 18:26 CEST) > while the timestamp found in the database is many years older (Aug 06 > 2006). The UIDVALIDITY conflict can't be caused by interimap or any > other IMAP client, because that's beyond the IMAP protocol. Did you > maybe run a lower-level synchronization tool like dsync(1) between > these mailboxes? IIRC dsync preserve states, so it could explain why > the UIDVALIDITY value was reset to the very same timestamp. > (UIDVALIDITY reset could also happen when the dovecot index files are > removed, but it seems unlikely that they would be set to the very same > value.) It should never happen with a “normal” layered IMAP stack. > Dovecot does have persistent UID values, at least when the index files > are not stored on a volatile medium. I began using interimap for this account circa October 20th - before that I indeed used dsync. I switched from offlineimap to dsync while visiting a rural town in Peru for a month few years ago - extremely flaky and bufferbloated internet access made offlineimap never finish a sync whereas a carefully/desperately tuned dsync would occationally succeed. It seems that interimap would do far better in such an extreme scenario, as it seems to syncronize small chunks at a time which would make it possible to continue-where-left-off in far smaller chunks than possible (or rather comprehensible to me) to do with either offineimap or dsync. When I switched, I removed the local folder and had interimap syncronize it from scratch. Or at least that is my belief: Does your forensics analysis above imply that I failed at that, and the local copy cannot have been initialized by interimap? > Now, how to fix this? The easiest is to remove the database entry for > that mailbox: `interimap --target=database delete INBOX` (this won't > touch your mails, just the database) and then try to sync again. > However every message in the local INBOX will be copied remotely and > vice-versa, so if you have a lot of messages in both mailboxes this > will create a lot of duplicates :-( This won't cause any data loss > though. That sounds like the least scary approach. And then afterwards try do a "doveadm deduplicate". Before that, however, I will try understand the last bits below... > Another thing which seems to suggest that there is a lower-level > synchronization running and causing cache corruption, is the huge > spike between the local INBOX's current state and the one from the > database (ie the last successful sync): UIDNEXT=9598 / > HIGHESTMODSEQ=165 vs. UIDNEXT=290008 / HIGHESTMODSEQ=71671. While > sudden heavy-traffic mailboxes are certainly possible, it's still a > bit suspicious. Sorry, I don't understand what you are saying here. Could that perhaps be explained by my _previous_ use of dsync and then resyncing from scratch with interimap? I mean, I guess that would show as high counts remote and low counts locally. > The first step is to identify what's causing the cache corruption as > it's likely to happen again otherwise, and that's beyond the scope of > interimap and other plain IMAP clients :-P If it was indeed dsync(1) > causing the cache corruption, then it's possible to avoid duplicates: > you could run it once more so both sides are in sync (with matching/ > overwritten UIDs, UIDVALIDITY, etc. values), then nuke *one* end along > with the database (using `interimap --target=database --target=local > delete INBOX`) and finally sync again. I reckon it's a bit scary > though, so feel free to poke me over IRC if you'd like. I would prefer to _not_ go back to dsync but explore the use of interimap. I suspect the corruption is caused by my use of notmuch with the synchronize_flags=true setting. I remember at your talk in Curitiba there was a question afterwards about notmuch and imap tags - I would very _very_ much like to shift from local-only notmuch tags to distributed imap tags - but that's probably another big discussion... - Jonas -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private
signature.asc
Description: signature
