Re: [Evolution-hackers] Regarding API breakage and lost test cases
On 10/06/2012 09:01 AM, Matthew Barnes wrote: On Fri, 2012-10-05 at 22:19 +0900, Tristan Van Berkom wrote: a.) e_source_registry_commit_source_sync() seems not exactly very sync. I haven't looked into that in detail but surely the registry server needs to block on something else before sending the reply. The issue is on the client side in ESourceRegistry. I had some pretty nasty code at one time to deal with this but must have yanked it while reworking the APIs. Even after the remote method call completes, ESourceRegistry will still have to stop and wait for an object-added signal from its internal GDBusObjectManagerClient, which is running in its own isolated thread. The object-added signal has the new GDBusObject needed to build the new ESource instance. And it can't complete on just any object-added signal -- it has to be the object-added signal corresponding to the scratch ESource that was just submitted. I'll see if I can restore that nastiness again in 3.7.x. Hi again. I see there has been an improvement on this (mentioned in https://bugzilla.gnome.org/show_bug.cgi?id=685986, which refers to a recent commit trying to address the problem). I'm not sure what the solution should be exactly, but we should discuss this a bit because the problem space is a little bigger than just the source existing in the client-side before e_source_registry_commit_source_sync() returns. While I did modify the test cases to listen to the source-added signal instead of relying on commit_source_sync() (or the added timeout which I had in place before that), there is still an interesting race condition to solve that is observable, but not always reproducing, with the test cases. Take the following pseudo code for instance: ~ ESourceRegistry *registry = e_source_registry_new_sync (...); ESourceBackend *backend; ESource *scratch = e_source_new_with_uid (test-address-book, ...); ESource *source; EBookClient *book; backend = e_source_get_extension (scratch, ADDRESS_BOOK); e_source_backend_set_backend_name (backend, local); e_source_registry_commit_source_sync (registry, scratch...); g_object_unref (scratch); /* YAY, now we successfully get the source on the client side :) */ source = e_source_registry_ref_source (registry, test-address-book); /* ... but the addressbook factory service might not have it yet * ... Uh oh... race condition here ! */ book = e_book_client_new (source, ...); ~ Basically, the problem is that we hit an assertion from the data factories when creating a book or calendar. I.e. e-data-book-factory.c:data_book_factory_open() (or impl_BookFactory_get_book() in older versions) goes ahead and also calls e_source_registry_ref_source(), but we dont know that _that_ process actually has the ESource yet. So I guess there are 2 potential ways of addressing this problem: a.) The call to e_source_registry_commit_source() needs to postpone completion until _all_ connected ESourceRegistry clients also receive the new ESource, this way we know that the addressbook factory service actually has the ESource before trying to create a book for it. (or that the addressbook factory has not yet been activated, which is also safe since initially creating the ESourceRegistry will load all the existing sources before continuing). b.) I could envision a sort of wait/refresh explicit call on the ESourceRegistry. Perhaps even using e_source_registry_ref_source() would be good, i.e when calling e_source_registry_ref_source(), if the source is not yet in the local cache of sources, it could fallback on a D-Bus call to the registry server explicitly asking for the given source by UID. If a direct D-Bus call to the actual registry service indicates that the source doesn't exist, then it surely really doesn't exist and an error can be reported from e_source_registry_ref_source(). Successful completion of the e_source_registry_commit_source() call would ensure, that any subsequent call to e_source_registry_ref_source() would also succeed on the same source UID. Actually, going with the latter alternative might be a good way of getting rid of the rather complex operation of waiting for the source-added signal before completing the e_source_registry_commit_source() call. Cheers, -Tristan ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Regarding API breakage and lost test cases
On Thu, 2013-01-31 at 18:25 +0900, Tristan Van Berkom wrote: I'm not sure what the solution should be exactly, but we should discuss this a bit because the problem space is a little bigger than just the source existing in the client-side before e_source_registry_commit_source_sync() returns. Note that e_source_registry_commit_source_sync() is just a convenience wrapper. If your test cases are just passing scratch sources then what you're actually calling is e_source_registry_create_sources_sync(). That's the function I fixed in bug 685986. So I guess there are 2 potential ways of addressing this problem: There's a 3rd approach. If you want an immediate client connection to a newly created address book, we could have evolution-addressbook-factory create the ESource on your behalf -- since it too has an ESourceRegistry -- and immediately instantiate a backend for it. That would eliminate the race without resorting to client-side hacks. It would require a new method on the addressbook and calendar factory interfaces, but that is easily done now that all our D-Bus interfaces are generated from XML specifications. That was the purpose of my recent rash of commits to master. Just this week I added e_book_client_connect_sync(), which combines the now-deprecated e_book_client_new() and e_client_open_sync() functions into one step. I could enhance this function to detect whether the given ESource is a scratch source, and invoke a different factory method which behaves as described above. No additional client-side APIs required. To summarize: The current addressbook factory D-Bus API looks like this: method name=OpenAddressBook arg name=source_uid direction=in type=s/ arg name=object_path direction=out type=s/ /method It only works for existing ESources already exported by the registry. I'm proposing to add a new method that takes the raw key file content from a scratch ESource in addition to a UID: method name=CreateAddressBook arg name=source_uid direction=in type=s/ arg name=source_data direction=in type=s/ arg name=object_path direction=out type=s/ /method Invoked by passing a scratch source to e_book_client_connect_sync(). The factory method would call e_source_registry_commit_source_sync() on your behalf, and assuming that goes well, return the object path to its newly-created EBookBackend. With equivalent changes to the calendar factory, of course. Matthew Barnes ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Regarding API breakage and lost test cases
On Thu, 2013-01-31 at 08:11 -0500, Matthew Barnes wrote: On Thu, 2013-01-31 at 18:25 +0900, Tristan Van Berkom wrote: I'm not sure what the solution should be exactly, but we should discuss this a bit because the problem space is a little bigger than just the source existing in the client-side before e_source_registry_commit_source_sync() returns. Note that e_source_registry_commit_source_sync() is just a convenience wrapper. If your test cases are just passing scratch sources then what you're actually calling is e_source_registry_create_sources_sync(). That's the function I fixed in bug 685986. So I guess there are 2 potential ways of addressing this problem: There's a 3rd approach. Hi, I vote for Tristan's 2), basically because it's the right approach in client-server architecture, also used in evo-mapi (and maybe in evo-ews, I do not know). Basically, if client doesn't have anything it was asked for, then it doesn't mean that the server doesn't have it too. The other things about the 3) I do not like: a) the point of changing book/cal APIs to not provide XML blobs of sources, but only UIDs was initiated and done by you, if I recall correctly, why to return back then? b) double commits? It might be about unnecessary overhead and double disk write/D-Bus noise too. I guess I got it right, because if it'll be only one commit and not two, then having a book factory creating the ESource, and still use it in the test/client code means that you still wait for a signal from the registry about source-added or whatever, which is basically no gain from the current state, only more complicated (D-Bus) API and giving a decision on the client whether to use one or the other function to open a book. Tristan's 2) can do all this transparently, without any API change on client side, maybe only a little change on the registry D-Bus API (adding function probably, I do not know). Anyway, I believe that keeping things simple is always a gain. Not that it's always possible, but why to confuse people by strange API? (Like those two 'remove' functions, and now proposed two opens.) Bye, Milan ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Regarding API breakage and lost test cases
On Tue, 2012-10-02 at 20:35 +0900, Tristan Van Berkom wrote: Ah, this is gold. I'll be able to readjust things with this. I've added a new section to the migration guide which covers the basics of creating a new local address book or calendar: https://live.gnome.org/Evolution/ESourceMigrationGuide#How_do_I_create_a_new_calendar_or_address_book.3F Hope this helps, Matthew Barnes ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Regarding API breakage and lost test cases
On 10/05/2012 09:00 PM, Matthew Barnes wrote: On Tue, 2012-10-02 at 20:35 +0900, Tristan Van Berkom wrote: Ah, this is gold. I'll be able to readjust things with this. I've added a new section to the migration guide which covers the basics of creating a new local address book or calendar: https://live.gnome.org/Evolution/ESourceMigrationGuide#How_do_I_create_a_new_calendar_or_address_book.3F Hi, Thanks Matthew for updating the migration guide. There are a couple of minor issues but overall this is working. a.) e_source_registry_commit_source_sync() seems not exactly very sync. I haven't looked into that in detail but surely the registry server needs to block on something else before sending the reply. Currently I have things working by adding a 20ms g_timeout_add() between e_source_registry_commit_source_sync() and the following call to e_source_registry_ref_source() (without out the timeout I get a 'no such source' type error). b.) It might also be worth noting in the guide/docs that all of these calls need to be made in the context of a running main loop (I was unable to get it working without a mainloop and with a basic call to usleep() for the added delay). However, the e-source-registry user facing APIs seem to make an attempt at doing this internally (i.e. it has it's own thread and mainloop presumably intended to handle the dbus messages as a convenience to the caller), so... probably just a minor detail or bug to fix there... Best Regards, -Tristan ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Regarding API breakage and lost test cases
On Fri, 2012-10-05 at 22:19 +0900, Tristan Van Berkom wrote: a.) e_source_registry_commit_source_sync() seems not exactly very sync. I haven't looked into that in detail but surely the registry server needs to block on something else before sending the reply. The issue is on the client side in ESourceRegistry. I had some pretty nasty code at one time to deal with this but must have yanked it while reworking the APIs. Even after the remote method call completes, ESourceRegistry will still have to stop and wait for an object-added signal from its internal GDBusObjectManagerClient, which is running in its own isolated thread. The object-added signal has the new GDBusObject needed to build the new ESource instance. And it can't complete on just any object-added signal -- it has to be the object-added signal corresponding to the scratch ESource that was just submitted. I'll see if I can restore that nastiness again in 3.7.x. However, the e-source-registry user facing APIs seem to make an attempt at doing this internally (i.e. it has it's own thread and mainloop presumably intended to handle the dbus messages as a convenience to the caller), so... probably just a minor detail or bug to fix there... The GDBusObjectManagerClient only emits signals from the GMainContext that was the thread-default at creation time. So it's created in its own dedicated thread to ensure its signal emissions cannot be inhibited by something pushing a different thread-default GMainContext. Otherwise deadlocks occur when an ESourceRegistry method has to stop and wait for a signal emission from the GDBusObjectManagerClient, such as in the case described above. Matthew Barnes ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Regarding API breakage and lost test cases
On Tue, 2012-10-02 at 13:30 +0900, Tristan Van Berkom wrote: I'm still trying to find my footing here, the migration guide and new documentation on ESourceRegistry don't seem to outline how a new addressbook is actually created. i.e. if I wanted to test a specific addressbook backend, I would need a way to create one and populate it inside some sandbox directory. The location of the database files is fixed for the file backend. It has to be in ~/.local/share/evolution and is derived from the UUID. I agree that an explicit path property for the file backend would be nice. I'm getting as far as: // Create scratch source e_source_new_with_uid(); // Add it to the registry e_source_registry_new(); e_source_registry_add_sources(); // Fetch the new source from the registry (no longer 'scratch') e_source_registry_ref_source(); // Create the book now that we have a valid source (... or so we // suspected) e_book_new (); Problems I'm encountering are: a.) It seems that e_source_registry_ref_source() only finds the newly created source the next time I startup the test app (so this may indicate a bug, that the local registry is not updated after adding a new source). Not sure about that one. To some extend the API depends on processing D-Bus events in the main event loop. Perhaps that's not happening in your test app? b.) e_book_new() complains that the new source has no backend: e-book.c:2944: cannot get book from factory: No backend name in source 'Unnamed' You need to add the right extensions to the new source: source = e_source_new(); ESourceBackend *backend = e_source_get_extension(source, E_SOURCE_EXTENSION_ADDRESS_BOOK); e_source_backend_set_backend_name(backend, local); I'm sure I've missed something ridiculous here, while observing the sources e-data-book-factory.c, it seems that if the source were given an addressbook extension with the intended backend name... that it would happily go ahead and create a book. But, while we do have: e_source_get_extension() e_source_has_extension() I'm not seeing: e_source_add_extension() _get_extension() adds the extension implicitly. -- Bye, Patrick Ohly -- patrick.o...@gmx.de http://www.estamos.de/ ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Regarding API breakage and lost test cases
On 10/02/2012 01:30 PM, Tristan Van Berkom wrote: On 10/01/2012 09:08 PM, Dan Vrátil wrote: On Monday 01 of October 2012 20:41:07 Tristan Van Berkom wrote: Hi, It's recently come to my attention that a vast API break[0] has been introduced in the last (3.6) cycle of EDS. Unfortunately this also breaks the vast majority of the test cases in EDS proper as well as the benchmarking tests[1] we've been using to track EDS performance over releases. After looking through about 1 year's worth of archives, I was only able to find a single thread[2] remotely related to this subject so, I'd like to at least raise the question, What is the rationale behind this API breakage in the 3.6 release ? Hi, check this thread [0], probably what you are looking for. And more practically speaking, is there any clear migration path towards the new APIs using the ESourceRegistry ? [1] Thanks Dan for your prompt reply yesterday. Those we're indeed what I was looking for. I'm still trying to find my footing here, the migration guide and new documentation on ESourceRegistry don't seem to outline how a new addressbook is actually created. FWIW, I have something in place that works, but it's a little dirty. Here's what can be done currently (but needs work to re-integrate into the tests suite): o Create a sandbox directory where EDS will be operating, lets call it TESTDIR o Create the file: $(TESTDIR)/evolution/sources/test-address-book.source o The contents of test-address-book.source should be as follows: [Data Source] DisplayName=Test Enabled=true Parent=local-stub [Address Book] BackendName=local Color=#becedd (NOTE: it should be the same as the EDS created file in data/sources/system-address-book.source, I just changed the DisplayName to be 'Test') o Here is one step I really need to get rid of: copy over the installed dbus .service files into /usr/share/dbus-1/ Ideally we should be running a dbus server that picks up the .service files that are not yet installed into your relocated build prefix (i.e. make check should run *before* make install) So this is just worse than bad, we use the system dbus to run the service over and copy the files into /usr :-/ o Now, setup the environment in which you will run the registry and addressbook services: export XDG_DATA_HOME=$(TESTDIR) export XDG_CACHE_HOME=$(TESTDIR) export XDG_CONFIG_HOME=$(TESTDIR) o Now everything should be setup to test the addressbook, run: /opt/devel/libexec/evolution-source-registry /opt/devel/libexec/evolution-addressbook-factory -r o Now assuming that you have fixed ebook-test-utils.c and client-test-utils.c, and uncommented the ebook tests in tests/libebook/Makefile.am, you can run tests and it will work. Another problem is that most of the tests in tests/libebook/client assume that they have created a book with 'new_temp_client()', so they will be successfully removing that book all the time, which means you need to comment out the calls to e_client_remove_sync() all over the place otherwise it will remove the handy file we've created test-address-book.source Attaching here quick and dirty patches to the ebook-test-utils.c and client-test-utils.c for reference. For now I think I'll leave the tests be, perhaps it will be better if we know what are the correct semantics for creating a book first. Cheers, -Tristan i.e. if I wanted to test a specific addressbook backend, I would need a way to create one and populate it inside some sandbox directory. I'm getting as far as: // Create scratch source e_source_new_with_uid(); // Add it to the registry e_source_registry_new(); e_source_registry_add_sources(); // Fetch the new source from the registry (no longer 'scratch') e_source_registry_ref_source(); // Create the book now that we have a valid source (... or so we // suspected) e_book_new (); Problems I'm encountering are: a.) It seems that e_source_registry_ref_source() only finds the newly created source the next time I startup the test app (so this may indicate a bug, that the local registry is not updated after adding a new source). b.) e_book_new() complains that the new source has no backend: e-book.c:2944: cannot get book from factory: No backend name in source 'Unnamed' I'm sure I've missed something ridiculous here, while observing the sources e-data-book-factory.c, it seems that if the source were given an addressbook extension with the intended backend name... that it would happily go ahead and create a book. But, while we do have: e_source_get_extension() e_source_has_extension() I'm not seeing: e_source_add_extension() Nor am I seeing any factory methods from the addressbook for creating an appropriate ESource derived object with an addressbook extension. Well, any
Re: [Evolution-hackers] Regarding API breakage and lost test cases
On 10/02/2012 08:29 PM, Patrick Ohly wrote: On Tue, 2012-10-02 at 13:30 +0900, Tristan Van Berkom wrote: I'm still trying to find my footing here, the migration guide and new documentation on ESourceRegistry don't seem to outline how a new addressbook is actually created. i.e. if I wanted to test a specific addressbook backend, I would need a way to create one and populate it inside some sandbox directory. The location of the database files is fixed for the file backend. It has to be in ~/.local/share/evolution and is derived from the UUID. I agree that an explicit path property for the file backend would be nice. I'm getting as far as: // Create scratch source e_source_new_with_uid(); // Add it to the registry e_source_registry_new(); e_source_registry_add_sources(); // Fetch the new source from the registry (no longer 'scratch') e_source_registry_ref_source(); // Create the book now that we have a valid source (... or so we // suspected) e_book_new (); Problems I'm encountering are: a.) It seems that e_source_registry_ref_source() only finds the newly created source the next time I startup the test app (so this may indicate a bug, that the local registry is not updated after adding a new source). Not sure about that one. To some extend the API depends on processing D-Bus events in the main event loop. Perhaps that's not happening in your test app? b.) e_book_new() complains that the new source has no backend: e-book.c:2944: cannot get book from factory: No backend name in source 'Unnamed' You need to add the right extensions to the new source: source = e_source_new(); ESourceBackend *backend = e_source_get_extension(source, E_SOURCE_EXTENSION_ADDRESS_BOOK); e_source_backend_set_backend_name(backend, local); I'm sure I've missed something ridiculous here, while observing the sources e-data-book-factory.c, it seems that if the source were given an addressbook extension with the intended backend name... that it would happily go ahead and create a book. But, while we do have: e_source_get_extension() e_source_has_extension() I'm not seeing: e_source_add_extension() _get_extension() adds the extension implicitly. Ah, this is gold. I'll be able to readjust things with this. Thanks a lot, -Tristan ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers
Re: [Evolution-hackers] Regarding API breakage and lost test cases
On 10/01/2012 09:08 PM, Dan Vrátil wrote: On Monday 01 of October 2012 20:41:07 Tristan Van Berkom wrote: Hi, It's recently come to my attention that a vast API break[0] has been introduced in the last (3.6) cycle of EDS. Unfortunately this also breaks the vast majority of the test cases in EDS proper as well as the benchmarking tests[1] we've been using to track EDS performance over releases. After looking through about 1 year's worth of archives, I was only able to find a single thread[2] remotely related to this subject so, I'd like to at least raise the question, What is the rationale behind this API breakage in the 3.6 release ? Hi, check this thread [0], probably what you are looking for. And more practically speaking, is there any clear migration path towards the new APIs using the ESourceRegistry ? [1] Thanks Dan for your prompt reply yesterday. Those we're indeed what I was looking for. I'm still trying to find my footing here, the migration guide and new documentation on ESourceRegistry don't seem to outline how a new addressbook is actually created. i.e. if I wanted to test a specific addressbook backend, I would need a way to create one and populate it inside some sandbox directory. I'm getting as far as: // Create scratch source e_source_new_with_uid(); // Add it to the registry e_source_registry_new(); e_source_registry_add_sources(); // Fetch the new source from the registry (no longer 'scratch') e_source_registry_ref_source(); // Create the book now that we have a valid source (... or so we // suspected) e_book_new (); Problems I'm encountering are: a.) It seems that e_source_registry_ref_source() only finds the newly created source the next time I startup the test app (so this may indicate a bug, that the local registry is not updated after adding a new source). b.) e_book_new() complains that the new source has no backend: e-book.c:2944: cannot get book from factory: No backend name in source 'Unnamed' I'm sure I've missed something ridiculous here, while observing the sources e-data-book-factory.c, it seems that if the source were given an addressbook extension with the intended backend name... that it would happily go ahead and create a book. But, while we do have: e_source_get_extension() e_source_has_extension() I'm not seeing: e_source_add_extension() Nor am I seeing any factory methods from the addressbook for creating an appropriate ESource derived object with an addressbook extension. Well, any insight into what detail I'm missing here would be great. Thanks, -Tristan I'd like to lend a hand in reviving the buried test cases, currently the trial-and-error approach is rather costly so any advice on the topic would be greatly appreciated. That would be great. Matthew will be back next week and will be able to give you more information then I :) Cheers, Dan [0] https://mail.gnome.org/archives/evolution-hackers/2012-May/msg00055.html [1] https://live.gnome.org/Evolution/ESourceMigrationGuide Best, -Tristan [0] http://git.gnome.org/browse/evolution-data-server/commit/?id=db4474f5463ac3f 1af12e182b20987e65a9b60c5 [1] http://taschenorakel.de/mathias/2012/06/20/performance-and-memory-usage-of-e volution-data-server/ [2] https://mail.gnome.org/archives/evolution-hackers/2011-September/msg00057.ht ml ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers ___ evolution-hackers mailing list evolution-hackers@gnome.org To change your list options or unsubscribe, visit ... https://mail.gnome.org/mailman/listinfo/evolution-hackers