Jamie McCracken wrote: > On Fri, 2008-05-02 at 11:58 +0200, Martyn Russell wrote: >> Jamie McCracken wrote: >>> On Mon, 2008-04-28 at 11:37 +0200, Philip Van Hoof wrote: >>>> On Mon, 2008-04-28 at 17:05 +0800, wendy wrote: >>>>> Hi Philip, >>>>> >>>>> I think we should take into account of using the "xesam-glib", which >>>>> will simplify the implementation of "xesam-support" on tracker. >>>> Hey Wendy, >>>> >>>> Xesam-glib has a few classes that provide an abstract GLib GObject. You >>>> can implement it and get everything DBus related for free. >>>> >>>> Which is nice, indeed. >>>> >>>> However. The DBus related things for the Xesam support for trackerd have >>>> already been done in a very similar way to how xesam-glib works. >>>> >>>> For now ain't the code actually doing things asynchronously because none >>>> of the existing trackerd infrastructure and code is prepared for >>>> asynchronous operations (there's insufficient locking and data isolating >>>> in place, SQLite wouldn't cope with multiple threads accessing it, etc). >>> sqlite 3.5 + is totally threadsafe and multiple threads can access it >>> safely >> That is good to know :) >> >>> though I dont think making it async is necessary? Surely that would >>> make the code harder to read and maintain? >> It isn't necessary, just like using threads isn't necessary. I thinking >> using threads is much harder to read and maintain than coding >> asynchronously. Understanding the execution process for a threaded >> application with mutexts blocking in various places, etc is a LOT harder >> to follow and understand than asynchronous programming. >> >>> to keep the tracker daemon resposive to handling search requests we will >>> need it to have threads with the main thread dedicated to handling dbus >>> calls (and shutdown) only (as it is in trunk) >> Making the daemon accept DBus calls asynchronously will not make it >> unresponsive. In my experience, a daemon handing 2 million telephony >> events a day on a busy switch can operate perfectly asynchronously, this >> sort of traffic is not what I would expect on a desktop at all either. I >> really have a hard time believing that the Tracker daemon would be >> unresponsive if we didn't use threads. Do you have evidence of this? >> >>> also as tracker is more increasingly used we will want multiple threads >>> handling requests >> Hmm? Why? > > if one client is asking for a huge result set which takes 5 seconds it > will block other clients for the same time period even if they only want > to ping it,
But this is all because the database API/our implementation is synchronous only. This effectively means we would need a thread in the indexer and in the daemon to do any database transactions, since I envisage the indexer having a DBus interface possibly (for shutting down and doing various non-database things too) - just an idea at this point. Also, have you done tests to see if there are any circumstances where the user has to wait 5 seconds or more? (I am just interested to be honest - most tests I have done here don't take that long at all). If they have to wait for that period of time for some database lookup in the first place, it means any other lookups will be delayed too - the only thing this really changes is how fast non-database requests respond (if at all since they can timeout). I have briefly looked at the SQLite3 API and it looks like an async approach could be written in Tracker using sqlite3_step() and throttling iterating the main loop occasionally. Also, if it is IO blocking, we can look into the patch that Mozilla uses for firefox to make IO access asynchronous. > if the daemon is totally async then we would: > > 1) lose any load balancing of queries so even non-db and non-index > requests can take a long time (and neither sqlite nor the index is > async) Depending on my assumption about our approach to using the SQLite3 API in an async or sync way, this is either a DB limitation or an implementation flaw which we can improve on. We should probably look into this some more and avoid the synchronous approach if we can do that. > 2) lose ability to shutdown promptly - force killing could corrupt index Yea, although with the new model we would only be reading in the tracker daemon not writing so there shouldn't be any corruption right? > 3) any client which used the sync dbus api (like nautilus and file > chooser) would be blocked totally What do you mean? You mean it wouldn't work for them? If so, that's not the case, DBus supports synchronous APIs for asynchronous interfaces. > with the index being threadsafe via the api I constructed (using > mutexes) and sqlite 3.5 being completely threadsafe there would be no > need for locks of any kind and none of the above would happen. For the > request daemon the threading is very simple too. I guess, ultimately if we HAVE no asynchronous approach to the database we have to use threads. I do wonder how we should approach this in the new libtracker-db internal library - since some backends will have async support and some won't. Some may even be async only. -- Regards, Martyn _______________________________________________ tracker-list mailing list [email protected] http://mail.gnome.org/mailman/listinfo/tracker-list
