Re: [Evolution-hackers] RFClue: Atomic folder updates

2011-12-04 Thread Sankar P
My memory is very rusty and I have not seen Evolution sources in more than 3 
years now. However, we had a similar situation when we working on a GroupWise 
backend and we used a trick to get the number of messages in a folder on 
startup of Evolution (or when the user explicitly presses the getmail button) 
and try to compare the mail counts between summary and server and update if 
they differ (and obviously, after fetching new items in this delta)

I am not sure if Exchange server has such a count API. If so, it can be used 
without breaking the summary code much.

Sankar
http://psankar.blogspot.com 

 On 12/3/2011 at 05:35 AM, in message
1322870713.5191.20.ca...@shinybook.infradead.org, David Woodhouse
dw...@infradead.org wrote: 
 We have at least three mail protocols now which are delta-based. That
 is, you have a bookmark, 'sync key' or timestamp which represents what
 the server *last* told you about a given folder. You say to the server
 what changed since XXX, and get back a list of added/removed/changed
 messages along with a *new* timestamp YYY.
 
 It's a very efficient way to handle mailbox access, and it's used by at
 least ActiveSync, EWS and IMAP+QRESYNC. In the Exchange protocols it's
 called 'SyncState' or 'SyncKey', and in IMAP it's the HIGHESTMODSEQ.
 (It's never *actually* a timestamp, since wall-clock time is a PITA. But
 it's easy to *think* of it as as timestamp; the modification time on the
 folder).
 
 The problem is, we need to handle these updates *atomically*. If we
 store the new timestamp before the changed messages, and we crash in the
 middle of doing so, then we might miss out forever on the messages in
 question. We'd restart, go to the server and say what happened since
 YYY and we never get told *again* about the messages which came in
 between time XXX and time YYY, that we didn't manage to fetch.
 
 And if you do it the other way round and store the changed messages
 first, and crash before you store the new timestamp, you get similar
 issues (cf. bug 664637).
 
 I'm not sure how to fix it. Looking at EWS first, since that's where we
 noticed it, I pondered removing the camel_folder_summary_save_to_db()
 call from the camel_ews_utils_sync_{created,updated,deleted}_items()
 helper functions, so it happens just *once* at the end of the loop in
 ews_refresh_info_sync() and commits all the changes at once. But just
 deferring the camel_folder_summary bits isn't enough, is it? The
 individual message_infos will have a lifetime of their own, and even
 *internally*, camel_folder_summary_save_to_db() doesn't actually write
 things out atomically using transactions in sqlite... does it?
 
 Any suggestions or insight would be gratefully received...
 
 -- 
 dwmw2
 


___
evolution-hackers mailing list
evolution-hackers@gnome.org
To change your list options or unsubscribe, visit ...
http://mail.gnome.org/mailman/listinfo/evolution-hackers


Re: [Evolution-hackers] RFClue: Atomic folder updates

2011-12-04 Thread David Woodhouse
On Sun, 2011-12-04 at 10:04 -0700, Sankar P wrote:
 My memory is very rusty and I have not seen Evolution sources in more
 than 3 years now. However, we had a similar situation when we working
 on a GroupWise backend and we used a trick to get the number of
 messages in a folder on startup of Evolution (or when the user
 explicitly presses the getmail button) and try to compare the mail
 counts between summary and server and update if they differ (and
 obviously, after fetching new items in this delta)
 
 I am not sure if Exchange server has such a count API. If so, it can
 be used without breaking the summary code much. 

We do something similar to that for IMAP — using the counts as a sanity
check and falling back to the old-style non-QRESYNC method of refetching
the flags for *every* message in the folder.

I don't believe we have fallback like that for EWS and ActiveSync. Those
didn't have sane delta-based operation 'tacked on' to an old protocol
which we can fall back to; they were designed to be delta-based from the
*start*, and the client is expected *not* to get itself out of sync.

Even if we could detect a problem in ActiveSync, we'd basically have to
refetch the entire folder to recover — and that would mean that message
IDs all change, so our cache has to be entirely blown away. I *think* we
could keep the cache in EWS and just refetch the message details, but I
still don't like it — it would be a horrid workaround for a bug, if we
ever have to use it.

When the server responds, it tells us here are a bunch of changes, and
your new 'bookmark' for the folder state is XXX. We should remember
that, atomically. If we crash, our cache on restart should reflect
either the state *before* the response, or the state *after* it. Never
something in between.

-- 
dwmw2


smime.p7s
Description: S/MIME cryptographic signature
___
evolution-hackers mailing list
evolution-hackers@gnome.org
To change your list options or unsubscribe, visit ...
http://mail.gnome.org/mailman/listinfo/evolution-hackers