Hello all!
I've used `mbsync` for quite some while to mirror my GMail account
to my laptop, but lately this task has become quite impossible...
The reason is simple: having 700k (as in 700.000, don't ask why!)
emails, with about 2k plus each day, just takes for ever... (My guess
is that `mbsync` tries to fetch the meta data of all these, and until
it starts to fetch the first new email, it spends about 10-15 minutes,
and this after about 3 retries (because GMail "barfs" some times)...)
Now, just to be clear, I'm not criticizing `mbsync` in any way, as
my use case is actually a peculiar one... (And from what I see from
the IMAPv4 protocol it's quite impossible to make efficient
synchronization for large mailboxes...)
What I'm trying to find out is how to solve my situation, and I'm
asking you guys to help me "brainstorm" ideas...
== Option A -- Make `mbsync` fetch "missing" new mail ==
After studying briefly the IMAPv4 protocol, and going through a
session of `mbsync --verbose`, I guess one solution would be in the
following lines:
* add a new parameter to the `Sync` option, something like
`PullOnlyMissing`;
* just this option alone implies that any other synchronization
should be ignored, except fetching those emails that haven't been
seen;
* before fetching, we determine which is the largest UID that we know of;
* instead of `5 UID FETCH 1:{$UIDNEXT} (UID FLAGS)`,
* it should send `5 UID FETCH {$UIDMAX+1}:{$UIDNEXT} (UID FLAGS)`;
Pros:
* it would just fetch the "really" new emails without delays;
Cons:
* it would ignore other changes on the server side;
* at some point in time the the file system would just crawl to a
hold; (currently I have JFS, and with 700k files in one folder it is
still usable, but who knows...)
== Option B -- Find a workflow that works ==
I'm testing right now a solution that doesn't imply changes to `mbsync`:
* make a filter in GMail that applies a "hidden" label (like
`drain`) to every incoming email; (it seems that you can get it to
also label sent emails...;)
* set up a channel that fetches this label as a IMAP folder, with
`Sync Pull`;
* after `mbsync` finishes, move all files from either `new` or
`cur` to a custom staging folder (maybe using a fanout folder
structure;)
* now that the files are missing on next `mbsync` run they are
going to be deleted from the `drain` folder, which GMail translates
into removing the label;
* just run the same process again with another channel that has
just `Sync PushDelete` and `Expunge both`;
Pros:
* it works just as the first solution;
* we don't trash the file system;
Cons:
* it involves "convincing" GMail to cooperate;
* it involves a non trivial workflow;
== Option C -- Implement a draining mail store into `mbsync` ==
We apply the same tactic as above related with GMail.
We implement a custom maildir store, that only keeps track of:
* UIDVALIDITY;
* last fetched UID;
* it only answers to list or store commands; (listing always
returns an empty list;)
* it always fails on any other command;
We also implement the fetch "missing" new emails (Option A), or we
can keep with the `drain` hack (Option B).
Pros:
* it just works;
* its the most efficient solution for "backups";
Cons:
* we have to update the C code;
== Option X??? ==
Any other ideas?
So I hope I didn't bored anyone... :)
Ciprian.
P.S.: Bonus point if I can have the "draining" work continuously
without periodically starting `mbsync`... (Although I can hack this
with `Tunnel` not to login multiple times...)
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://ad.doubleclick.net/clk;258768047;13503038;j?
http://info.appdynamics.com/FreeJavaPerformanceDownload.html
_______________________________________________
isync-devel mailing list
isync-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/isync-devel
