Re: [Evolution-hackers] Contacts database modifications history
First one is to create new address book backend (which will be inhering from file backend used by Syncevolution) and add there logic for calculating modifications when creating/modifying/removing contacts. These modifications will be stored in additional database, so some additional functions for handling this will need to be added to EBookSqlite class. Here are some pros and cons that I can see: Pros: - If someone don’t want to keep track of modifications history he can still use file backend - We can be sure that contacts database and modifications database will be in sync, by using attach function of sqlite (databases will be separated but transaction for modifying both of them will be treated as one transaction) Does it have to be a separate sqlite database? You could also use a new table directly in the sqlite datbase also used for the contact data. No, it don't need to be separate sqlite database, but I was thinking about case where database will be locked because some operation like locale update is ongoing, with separate database modifications data could be still concurrently accessed. Cons: - Quite complex implementation, especially if some API needs to be exposed by EBookClient Perhaps the API for reading the changes could be limited to direct read access, potentially outside of the core libebook? Direct access will be fine for me Second solution is to extend current implementation of EBookView and add new signals like: contacts_added, contacts_modified, contacts_removed which will be providing ids and list of modified fields. Also new function for setting fields of interest will need to be added. Again in the address book backend implementation of logic for calculating modifications and emitting this signals will have to be added (it can be added to current file backend implementation or like in previous idea, new backend can be created based on file backend). Note that the EBookView could also be used as-is by doing the detailed check of what was modified in the receiving process. For new contacts it is very likely that the fields of interest were added, and updating contacts happens rarely enough that the overhead of sending updated data that then completely gets filtered out shouldn't matter. I'm not sure about using EBookView as-is and checking what was modified in receiving process, because notification will be sent when data in contacts database will be already changed, so receiving process should first get all contacts from database before starting synchronization to be able to do detailed check of what was modified, which in case of large number of contacts may unnecessarily increase memory consumption, especially as you've said that updates happens rarely. Pros: - Way of storing modifications information is up to the client application, - In our case we would like to know what modifications were made during synchronization, EDS is not aware of which modification of contacts are part of which synchronization, but client application has this knowledge. Cons: - We may end up with out of sync data in case if some signals will be not emitted. I see that as the most important reason why I would prefer the more integrated approach. What is the probability of failure of emitting signals ? Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen, Deutschland Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Christian Lamprechter, Hannes Schwaderer, Douglas Lusk Registergericht: Muenchen HRB 47456 Ust.-IdNr./VAT Registration No.: DE129385895 Citibank Frankfurt a.M. (BLZ 502 109 00) 600119052 ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Contacts database modifications history
On Tue, 2014-04-15 at 13:16 +, Potrola, MateuszX wrote: I would like to have ability to receive some kind of notifications (or store information in some additional database) about modifications made to contacts database during synchronization. Hi, the compose of a diff for done changes is probably the hardest thing on this whole thing you'd like to achieve. There is always (at least) one place where this diff should be computed. I'm not much willing to add it into evolution-data-server itself, because it has no good use for it (at least not as of now). As Patrick pointed out, the previous implementation had sever issues which made the feature basically unusable. If we cannot make it done for all backends, then it probably might not be part of the evolution-data-server. I think the best parallel is the Exchange Web Services protocol with its ChangeKey-s. The ChangeKey is used to identify a state on the client side, with which the client asks for a list of changes being done since this state. I have no idea how the history is managed in the background, but it works pretty well. I would be still afraid of the actual disk usage of the history, but the ChangeKey marker is the part which was missing in the previous implementation an the eds side. Anyway, I would prefer if you could create a plugin to SyncEvolution (does it support plugins?), also because it's the software you are targeting, not evolution-data-server code, and use EBookClientView on the book used by SyncEvolution, with optional usage of fields of interest. The flow, from my point of view, would look like this: a) user initiates a synchronization in SyncEvolution b) before SyncEvolution starts the addressbook synchronization, the new plugin opens the EBookClientView on the same book and sets it up according to its needs (either fields of interest, or a basic not field-is-empty X filter (filters might be better here, also because it's possible that fields of interest will be dropped in a favor of Tristan's cursors)). Note that once the EBookClientView is started, the client receives notifications for all the satisfied contacts, which does the initial population of the view (as you mentioned, many contacts will mean performance and memory usage impacts). c) once the EBookClientView receives a complete signal, which indicates that the initial fill of the view is done, the SyncEvolution can starts its synchronization d) each of the modifications will be received in the EBookClientView signals, newly the objects-added signal will be emitted also for old contacts which satisfy the filter now, same as the objects-removed signal not necessarily mean that the contact had been completely removed from the book. f) once the SyncEvolution is done, the EBookClientView can be also stopped and released. Still, all this looks too complicated, adding unnecessary burden, consider that you specifically target SyncEvolution changes, thus why not to extend SyncEvolution to write its own backlog of done changes in some place, where your application may just read them from? If the changes are marked with a date+time+sequence ChangeKey-s, then you can get a very nice overview what SyncEvolution did and when. I think of the changes in the SyncEvolution side also because all the EBookClientView engagement feels like a reinvent of the wheel, considering that SyncEvolution does the changes and it knows what it does the best. On Wed, 2014-04-16 at 09:02 +, Potrola, MateuszX wrote: What is the probability of failure of emitting signals ? The probability is close to 0. If it happens, then it's a bug in the backend (or the related code) and should be fixed as soon as discovered. Bye, Milan ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Contacts database modifications history
On Wed, 2014-04-16 at 12:23 +0200, Milan Crha wrote: On Tue, 2014-04-15 at 13:16 +, Potrola, MateuszX wrote: I would like to have ability to receive some kind of notifications (or store information in some additional database) about modifications made to contacts database during synchronization. Hi, the compose of a diff for done changes is probably the hardest thing on this whole thing you'd like to achieve. There is always (at least) one place where this diff should be computed. I'm not much willing to add it into evolution-data-server itself, because it has no good use for it (at least not as of now). I tend to agree. It would be very hard to come up with a general purpose diff that works across a wider range of use cases. If it is configurable, the config language and implementation would have to be unnecessarily flexible (and thus complicated) for what Mateusz probably has in mind. I still think a plugin doing the diff would be okay. As Patrick pointed out, the previous implementation had sever issues which made the feature basically unusable. If we cannot make it done for all backends, then it probably might not be part of the evolution-data-server. We have a slightly different understanding of EDS here. You see EDS as an abstraction layer that offers just the functionality that can be implemented with all backends. We focus on EDS + local storage and try to make the best use of its capabilities, even if that means adding things to it which won't work with other backends. That's find, I think we can do both. It just means that whatever API gets added that only works with local storage needs to be clearly marked as local only in the documentation and/or there has to be a discovery mechanism for determining whether a certain EBookClient instance supports it. Anyway, I would prefer if you could create a plugin to SyncEvolution (does it support plugins?), also because it's the software you are targeting, not evolution-data-server code, Not necessarily. At the moment the focus is on cached PBAP address books that are only written by SyncEvolution, but that may change in the future. Perhaps someone wants to rip out SyncEvolution and do the caching differently, or wants to have address books that can be edited by the user. The change tracking still needs to work in those cases. Another argument for doing it inside EDS is that it can be combined with the actual sqlite database update. To me, that is the only bullet-proof way of ensuring that the change tracking information will be consistent with the actual content. This must work in a car, where the head unit can loose power at any moment. Ensuring that the sqlite database doesn't get corrupted is difficult enough. Ensuring that different processes get a chance to write different files will be even harder. I'd like to mention further use case for allowing hooks during the execution of EDS writes: this can be used to pre-compute data that is later going to be shown by a UI other than Evolution. I think EDS already does some of this for Evolution (like computing a full name) in a way that, strictly speaking, depends on the UI that is going to show the data. One even crazier idea was to maintain an on-disk data structure that can be memory-mapped to access contact data, including strings and their parsed photos ready to be displayed, without going through sqlite and all the string handling. I'm not a terribly big fan of that idea myself, but if there were hooks in EDS which could be used to track DB changes reliably, that would allow implementing this idea without further changes to EDS itself. Still, all this looks too complicated, adding unnecessary burden, consider that you specifically target SyncEvolution changes, thus why not to extend SyncEvolution to write its own backlog of done changes in some place, where your application may just read them from? SyncEvolution indeed does a read/update/write cycle when updating contacts. This would be a natural place to add a hook for such a plugin. Implementing it will be a bit tricky because it happens under the hood in the Synthesis sync engine, but I guess it could be done, if the alternative (a similar hook in EDS) is not acceptable. On Wed, 2014-04-16 at 09:02 +, Potrola, MateuszX wrote: What is the probability of failure of emitting signals ? The probability is close to 0. If it happens, then it's a bug in the backend (or the related code) and should be fixed as soon as discovered. The more likely failure is that EDS emits the change, but then the computer is shut down before the receiving side gets a chance to handle it = EDS DB was modified, change log in the receiver was not. Bye, Patrick ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Contacts database modifications history
Hi, On Wed, 2014-04-16 at 14:00 +0200, Patrick Ohly wrote: It just means that whatever API gets added that only works with local storage needs to be clearly marked as local only in the documentation and/or there has to be a discovery mechanism for determining whether a certain EBookClient instance supports it. Right, there are the backend capabilities, which can be used for this. If the backend supports some functionality, it also advertises that in its capabilities. The more likely failure is that EDS emits the change, but then the computer is shut down before the receiving side gets a chance to handle it = EDS DB was modified, change log in the receiver was not. I see, you want a fool-proof solution. That makes perfect sense. You already mentioned that it would be better to use the same sqlite database as the contacts are stored in, rather than using a separate sqlite database file, one for data and one for backlog. The local address book uses EBookSqlite, thus what about making it an EExtensible? That way, with some EBookSqlite (internal and external) API changes, you would be able to create a new module, which will be loaded for each EBookSqlite instance and which will be able to step in during e_book_sqlite_add_contact/s, e_book_sqlite_remove_contact/s, possibly updating an internal table with a list of changes as needed, within the same transaction as actual contact's update? With a good API changes you might be able to even get the contact before and after the change, so you can create your diff. You definitely do not want to do the backlog for each EBookSqlite, because it's supposed to be used as the current local contacts cache (to be replaced with an OfflineCache in the future for the remote backends), thus some checking whether the backlog should be created, based on the actual book being this instance used with [1], is necessary. I believe the use of the same transaction is the most important part for you, while that should be satisfied with this EExtensible solution, with absolutely no changes on the EBookClient* public APIs. Bye, Milan [1] Either check based on the actual folder being used for the EBookSqlite, or pass also an ESource to the e_book_sqlite_new functions and create a new ESourceExtension, where you would be able to turn on backlogging selectively, without any guessing. The ESourceExtension can be done in the same module as the backlog creator. ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
