[Evolution-hackers] concurrent modifications of items in GUI and EDS database
Hello! I'm currently thinking about synchronizing data with SyncEvolution in the background while the user is active with the Evolution UI. Some users already do that via cron jobs, but it is known to be problematic for several reasons, among them change tracking and concurrent editing of items. I already discussed this with Ross Burton and Rob Bradford. Now I would like to check that we haven't missed anything. The plan for change tracking is to get rid of the dependency on e_book_get_changes(). I already stopped using e_cal_get_changes() because it was too inflexible. Instead I'll rely on the REV resp. LAST-MODIFIED properties: the backend must update these each time an item is modified. This seems to be supported by most backends. Are there backends which are known to not support this? REV and LAST-MODIFIED are treated as opaque revision strings. SyncEvolution doesn't interpret the content, only compares it against the last synchronized value. The problem with concurrent modifications is two-fold. Stale data in UI: * user opens an item in Evolution * synchronization starts in the background * updates the item in EDS * = When the user saves his changes, will he overwrite the more recent data in EDS? Will he be warned? With Evolution the user is not warned and his changes overwrite the ones in EDS (tested with contacts). Evolution should listen for change signals and warn the user as soon as he has stale data in the edit dialog. The user then can cancel and reopen the item to redo his changes. This is unlikely to happen often, so more elaborate solutions (merging changes, additional buttons to copy from EDS) should not be necessary. Should I file a bug for this? Anyone able and willing to work on it? Stale data in sync: * when the sync starts, it builds a list of new/updated/deleted items * user modifies data in EDS * this leads to conflicts, f.i. sync modifies item that was modified by user Both cases need to be handled by the program which wants to make changes to EDS data (Evolution, sync engine). To avoid race conditions, support by EDS would be needed which currently doesn't exist. As a workaround the following method would reduce the time window in which conflicts can occur: * get revstring before starting to make modifications (when opening item in UI; when starting sync) * before modifying the item, check the revstring again * if the same as before, do the modification * if different, handle conflict A secure solution would have to put the revision check into EDS itself to make the check and update atomic. The proposal is to add this check to e_book_commit_contact() and e_cal_modify_object(): * The caller is expected to include REV resp. LAST-MODIFIED as read from EDS earlier. * The EDS backend compares against the current value before updating the item. If there is a mismatch, the update is rejected with a suitable error code. * If the values are unset, the update is always executed. Problem 1: a client cannot tell whether the backend that it is using implements this check. If the backend doesn't (not yet updated, not possible), then the client should implement the workaround described above. Problem 2: the backend cannot tell whether the client wants to have the check enabled or not. An older client might send values for REV/LAST-MODIFIED and might expect to always succeed. Such a client could be considered broken and refusing the update might be the right solution. Problem 3: the Exchange Connector does not update LAST-MODIFIED if the VEVENT contains it. I consider this a bug and work around it in SyncEvolution by filtering out the property before importing into EDS. Adding LAST-MODIFIED unconditionally would break change tracking. Exact handling of REV and LAST-MODIFIED in the two calls is basically undefined right now as far as I can tell. I suggest to clarify that: * backends should check them as described above * backends must update them independently of what the client sends To allow clients to query whether the backend supports the check I suggest to add a new capability for e_cal_get_static_capability() and e_book_check_static_capabilities(). Any comments? -- Bye, Patrick Ohly -- patrick.o...@gmx.de http://www.estamos.de/ ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] A Camel API to get the filename of the cache, also a proposal to have one format to rule them all
The Exchange patch looks fine to me. -Srini. On Mon, 2009-01-05 at 12:28 +0100, Philip Van Hoof wrote: On Mon, 2009-01-05 at 00:42 +0530, Srinivasa Ragavan wrote: On Fri, 2009-01-02 at 13:25 +0100, Philip Van Hoof wrote: Hi there evos, For an EPlugin that I'm working on I will need a Camel API to get the filename of the cache. Sure and the patch seems fine to me, but the Exchange portion of the patch is missing. It should be similar/simple. Attached. Let me know when it's all reviewed and/if I can commit it. ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] concurrent modifications of items in GUI and EDS database
Il giorno mer, 07/01/2009 alle 12.45 +0100, Patrick Ohly ha scritto: Hello! I'm currently thinking about synchronizing data with SyncEvolution in the background while the user is active with the Evolution UI. Some users already do that via cron jobs, but it is known to be problematic for several reasons, among them change tracking and concurrent editing of items. I started studing a plugin to do the sync on user request. I'm going to present my idea in another email just to have clearer threads. Bye sc -- Stefano Canepa aka sc: s...@linux.it - http://www.stefanocanepa.it Three great virtues of a programmer: laziness, impatience and hubris. Le tre grandi virtù di un programmatore: pigrizia, impazienza e arroganza. (Larry Wall) signature.asc Description: Questa è una parte del messaggio firmata digitalmente ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
[Evolution-hackers] Funambol sync from evolution
Hi all, I'm trying to develop a plugin to run configure and run syncevolution on user request. My idea is to have a plugin like the one that sync with the ipod adding an option on the context menu in calendar and addressbook. On user request the plugin will run syncevolution, displaing an error message if there is no syncevolution on the system and the log and all messages of syncevolution if there are logs or messages. I would like to try to develop it in python as I see that evolution is now capable of loading plugins written in this language, but it should be no problem in developing in C. Wich language do you suggest to use? TIA sc -- Stefano Canepa aka sc: s...@linux.it - http://www.stefanocanepa.it Three great virtues of a programmer: laziness, impatience and hubris. Le tre grandi virtù di un programmatore: pigrizia, impazienza e arroganza. (Larry Wall) signature.asc Description: Questa è una parte del messaggio firmata digitalmente ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Cannot run evolution built using jhbuild
On Mi, 2009-01-07 at 22:20 +0100, Stefano Canepa wrote: I would like to write an e-plugin to run syncevolution Excellent! Let me know when I can help with the SyncEvolution side of things. As far as e-plugins go I don't know anything myself, so we'll have to depend on the assistance of the Evolution hackers. but as I'm new to evolution development I decide, may be I'm wrong, to start build from source using jhbuild thinking that this is the quickest way to satisfy the requirements. I just followed http://projects.gnome.org/evolution/build.shtml. The build process completed with big problems. When I'm running a Debian unstable with gnome 2.22. When I try to run the new evolution I get the errors pasted here: http://pastebin.com/m6d825bf5 I don't know jhbuild. I suspect that it builds more libraries and later when you try to run evolution-cvs, the environment is not set up correctly so those libraries are not found. So my question are: 1) is using jhbuild the correct way to get latest evolution and develop an e-plugin? It is going to be useful to build Evolution from source without optimization and with debug information, because then you can more easily study how the plugin mechanism works (set breakpoints in a debugger, go up the stack, analyze variables, etc.). I suggest that you give the following Makefile a try before spending more time with jhbuild: http://mad-scientist.us/evolution.html It builds just Evolution and generates an evolution-env script that sets up the right environment for you. The makefile depends on recent enough version of the libraries that Evolution depends on. You shouldn't have a problem in Debian unstable. I also used garnome on machines where those libraries had to be compiled from source. It's more complicated than the makefile. -- Bye, Patrick Ohly -- patrick.o...@gmx.de http://www.estamos.de/ ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Funambol sync from evolution
On Mi, 2009-01-07 at 22:27 +0100, Stefano Canepa wrote: On user request the plugin will run syncevolution, displaing an error message if there is no syncevolution on the system and the log and all messages of syncevolution if there are logs or messages. It wouldn't be hard to create and install a libsyncevolution.so which contains all the necessary logic to execute a sync. That might be easier than wrapping a command line tool. On the other hand, Genesis already does that quite successfully. I would like to try to develop it in python as I see that evolution is now capable of loading plugins written in this language, but it should be no problem in developing in C. Wich language do you suggest to use? Genesis is written in Python, so you could take that as example for writing a syncevolution command line wrapper. I also started with a Python binding for the relevant SyncEvolution classes that a GUI would have to interact with. This was an experiment to learn about Boost.Python. It showed that this is so easy that finishing it should be a matter of a day perhaps. Want to learn how it was done and perhaps finish it? ;-) Browse the source: https://zeitsenke.de/websvn/listing.php?repname=SyncEvolutionpath=%2Fbranches%2Fpython%2Frev=0sc=0 Check out: http://zeitsenke.de/svn/SyncEvolution/branches/python/ If you prefer to work with git (easier for external contributors) then I could also provide a git mirror of the SVN repository. My suggestion regarding the language would be to check out how easy it is to write Evolution plugins in C and Python. Pick the one that you find easier. Python is a nice language. A downside might be that users of older Evolution releases without that support won't be able to use your plugin. When was Python support added? -- Bye, Patrick Ohly -- patrick.o...@gmx.de http://www.estamos.de/ ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Cannot run evolution built using jhbuild
Hi, On 07.01.2009 22:20, Stefano Canepa wrote: 2) what's going wrong the evolution built by jhbuild? The build itself works properly, right? Running your newly build Evolution fails, if I read your log correctly. To run evolution, you should do a jhbuild run evolution. It is not clear to me, what evolution-cvs is or does. Also, you should do an export LANG=C to reset the language before you post any logs, because reading messages in your language might be difficult for somebody ;-) What I can guess from the error messages is, that you run an old gconfd on your host system and don't run the new, dbus-ified GConfD from jhbuild. Try to run the gconf deamon from your jhbuild prefix. i.e. jhbuild run /opt/gnome/libexec/gconfd-2 before you run your evolution-trunk. Happy Hacking, Muelli ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Funambol sync from evolution
On Wed, 2009-01-07 at 23:20 +0100, Patrick Ohly wrote: My suggestion regarding the language would be to check out how easy it is to write Evolution plugins in C and Python. Pick the one that you find easier. Python is a nice language. A downside might be that users of older Evolution releases without that support won't be able to use your plugin. When was Python support added? Maybe have a look at Vala http://live.gnome.org/Vala Kind of best of both worlds, Python/ C Cheers ___ Evolution-hackers mailing list Evolution-hackers@gnome.org http://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] concurrent modifications of items in GUI and EDS database
On Wed, Jan 7, 2009 at 17:15, Patrick Ohly patrick.o...@gmx.de wrote: The plan for change tracking is to get rid of the dependency on e_book_get_changes(). I already stopped using e_cal_get_changes() because it was too inflexible. Instead I'll rely on the REV resp. LAST-MODIFIED properties: the backend must update these each time an item is modified. This seems to be supported by most backends. Are there backends which are known to not support this? a) Looking at [1], I can't find what a REV property is. Did you mean to use [2] ? The problem with concurrent modifications is two-fold. Stale data in UI: * user opens an item in Evolution * synchronization starts in the background * updates the item in EDS * = When the user saves his changes, will he overwrite the more recent data in EDS? Will he be warned? With Evolution the user is not warned and his changes overwrite the ones in EDS (tested with contacts). Evolution should listen for change signals and warn the user as soon as he has stale data in the edit dialog. The user then can cancel and reopen the item to redo his changes. This is unlikely to happen often, so more elaborate solutions (merging changes, additional buttons to copy from EDS) should not be necessary. Should I file a bug for this? Anyone able and willing to work on it? Not so for calendars. When an event is open in the Evolution UI and the backend modifies it during a refresh/update from server, the user _is_ warned of an update to the item. If not, the backend is probably not doing a e_cal_backend_notify_object_modified() which it ought to. FWIW, see the open bug at [3]. Stale data in sync: * when the sync starts, it builds a list of new/updated/deleted items * user modifies data in EDS * this leads to conflicts, f.i. sync modifies item that was modified by user Both cases need to be handled by the program which wants to make changes to EDS data (Evolution, sync engine). To avoid race conditions, support by EDS would be needed which currently doesn't exist. As a workaround the following method would reduce the time window in which conflicts can occur: * get revstring before starting to make modifications (when opening item in UI; when starting sync) * before modifying the item, check the revstring again * if the same as before, do the modification * if different, handle conflict The GroupWise server updates the 'modified' property of the item when it actually gets modified on the server. For newly created items, it also adds the 'created' property at the same time. This behavior invalidates all the 'handle-at-backend' approaches to fix the apparent bug, like: * Check if the item is newly-created by looking for a 'modified' property in the cached-object. This approach does not handle modifying an older (and already cached from server) item. * Pushing the 'modified' property from the client to the server upon creation/modification. This does not help, since the server would modify the property anyway. * A series of 'if-else' trying to handle various scenarios. It was observed that the conditions would fail for at-least one use-case, mainly since we would not have a persistent 'last-modified' time unless it is obtained from the server. These drawbacks probably apply to the Exchange backend too. A secure solution would have to put the revision check into EDS itself to make the check and update atomic. The proposal is to add this check to e_book_commit_contact() and e_cal_modify_object(): * The caller is expected to include REV resp. LAST-MODIFIED as read from EDS earlier. * The EDS backend compares against the current value before updating the item. If there is a mismatch, the update is rejected with a suitable error code. * If the values are unset, the update is always executed. A backend may not have a LAST-MODIFIED property for a particular event in this use case: a) create a new appointment in the GW calendar (while online) b) open the same appointment (before the refresh timeout) Pushing an update in this case is not correct as the event can easily be modified by another client during that refresh interval. We should ensure that the cached object in EDS will always have the same last-modified property as that on the server before opening the event in the UI [This probably implies an (expensive?) 'live' re-cache of the object being opened]. We already listen for changes to the event and so the user automatically knows if anything changed since opening the event editor. We don't continuously listen for updates from the server and hence the problem I mentioned above would still exist, but that is something I can live with :) - Suman [1] http://tools.ietf.org/html/rfc2445 [2] http://tools.ietf.org/html/rfc2445#section-4.8.7.4 [3]
