On Thu, Jun 23, 2016 at 05:06:15PM +0200, Yuri D'Elia wrote:
> On Sun, Jun 12 2016, Oswald Buddenhagen wrote:
> > On Sun, Jun 12, 2016 at 03:25:35PM +0200, Yuri D'Elia wrote:
> >> Does isync currently [attempt to] detect messages which have been moved
> >> across folders, and propagate the change with MOVE?
> >>
> > nope. patches welcome (i can give some direction if you're interested).
>
> Thinking again... can I receive those directions?
>
so, somewhat belated, here they are:
- 1. UIDVALIDITY change recovery
- background: some servers (ms exchange in particular) like to change
mailboxes' UIDVALIDITY for no particular reason while the actual
UIDs stay the same. as a consequence, mbsync refuses to sync the
folder, as continuing could potentially cause data loss (if the UIDs
*did* actually change). the goal is to detect this situation and
simply accept the new UIDVALIDITY (a rather tedious operation when
executed manually).
- how: when fetching new messages (*), also fetch their Message-Id,
and store it in the sync state file (after the flags). when a
changed UIDVALIDITY is reported when a mailbox is opened again,
re-fetch the headers of all messages and verify that all known UIDs
still have the same Message-ID. if so, accept the new UIDVALIDITY
(and emit a notice).
- (*) this also needs to cover the case of messages which were
initially synced before the feature was added.
- it happens that messages have no Message-Id (in particular, this
is permissible for drafts). the way around that is synthetizing
one: fetch all available headers (subject, from, to, cc, bcc,
date), concatenate them, and calculate a hash. the paranoid
version would hash the whole message.
- why: you'll need the Message-Id storage, and this is a nice stepping
stone.
- 2. parallel syncing of multiple mailboxes
- background: mbsync currently syncs only one mailbox at a time. this
increases latency in particular when there are almost no changes to
sync.
- how: mostly just refactoring in main.c.
- add option MaxConnections to limit parallelity (20 simultaneous
connections aren't really nice to the server).
- why: you'll need the ability to have multiple mailboxes open at the
same time, at least logically.
- 3. optional: add support for CONDSTORE & QRESYNC
- background: mbsync currently fetches the headers of all messages it
wants to sync and calculates the delta itself. that costs a lot of
bandwidth. this problem has been recognized by the IMAP authorities
and they defined extensions which put much of these calculations on
the server side.
- how:
- in a first step, factor a big part of the main syncing code into
utility functions. make the drivers use these functions to
calculate event streams instead of providing raw data about the
mailboxes. adjust the main sync loops to this new input.
- in the second step, add optional code paths using said extensions
to obtain the event streams. some hints are in the TODO file.
- why: you'll need some kind of editable event stream anyway.
- 4. implement move detection
- background: re-syncing messages from scratch when they actually just
moved is a waste of time, in particular when doing local moves and
using an asymmetric uplink (the typical case).
- how: when a mailbox' sync has progressed so far that new and deleted
messages have been identified, call back into main.c, new function
done_pre_sync(). this function would "muliplex" the lists of new and
deleted messages by Message-Ids and synthetize moves from that. the
original new/delete entries would be pruned from the lists, and the
move command would be queued to the source mailbox (that's a
practicality dictated by IMAP).
- i have no clue how to integrate this with the transaction system.
possibly, the log replay would also need some multiplexing
callback.
- as the number of simultaneously open mailboxes may be limited, it
may be necessary to re-open the mailbox after returning from the
"multiplexer".
- paranoid mode: in addition to comparing Message-IDs, the actual
messages would be re-fetched and compared. that's because in
principle it would be legal to modify messages while moving them
(they get new UIDs anyway). that's still cheaper then uploading
them over an asymmetric line.
as you can see, this isn't a project for a few hours, and the likelihood
that i'll embark on it myself anytime soon is about zilch.
bonus feature: with the Message-Id handling in place, it would be also
possible to implement smooth migration from other synchronizers.
------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
isync-devel mailing list
isync-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/isync-devel
