[Zeitgeist] [Bug 695297] [NEW] unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken
Public bug reported: I'm talking about def _register_data_source_python_api(self, *args): mainloop = gobject.MainLoop() self.client.register_data_source(*args, reply_handler=lambda: mainloop.quit(), error_handler=lambda: fail(Error registering data-source)) mainloop.run() * In error_handler it should be self.fail(...) * register_data_source() has no reply_handler/error_handler arguments On the other hand, this method isn't used anywhere @Siegfried, I think you added this code recently, can you please fix it. Thanks so much ;) ** Affects: zeitgeist Importance: Undecided Status: New -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695297 Title: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken Status in Zeitgeist Framework: New Bug description: I'm talking about def _register_data_source_python_api(self, *args): mainloop = gobject.MainLoop() self.client.register_data_source(*args, reply_handler=lambda: mainloop.quit(), error_handler=lambda: fail(Error registering data-source)) mainloop.run() * In error_handler it should be self.fail(...) * register_data_source() has no reply_handler/error_handler arguments On the other hand, this method isn't used anywhere @Siegfried, I think you added this code recently, can you please fix it. Thanks so much ;) ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695311] [NEW] DataSourceRegistry is a huge performance bottleneck
Public bug reported: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ** Affects: zeitgeist Importance: Undecided Status: New -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695311 Title: DataSourceRegistry is a huge performance bottleneck Status in Zeitgeist Framework: New Bug description: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695311] Re: DataSourceRegistry is a huge performance bottleneck
** Attachment added: test_insert_events.py https://bugs.edge.launchpad.net/bugs/695311/+attachment/1778807/+files/test_insert_events.py -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695311 Title: DataSourceRegistry is a huge performance bottleneck Status in Zeitgeist Framework: New Bug description: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695087] Re: Invalid cache access
*** This bug is a duplicate of bug 598666 *** https://bugs.launchpad.net/bugs/598666 @Siegfried, what exactly are you benchmarking here? Eg. how many different (relevant) values are you using? How do the graphs look if you have some hundred different actors and mimetypes and a wide range of used manifestations and interpretations? -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695087 Title: Invalid cache access Status in Zeitgeist Framework: Triaged Bug description: ala...@alagos-desktop:~$ gnome-activity-journal ** Message: pygobject_register_sinkfunc is deprecated (GstObject) /usr/share/gnome-activity-journal/src/common.py:747: DeprecationWarning: object.__new__() takes no parameters GIO_FILES[subj] = object.__new__(classtype, *args, **kwargs) /usr/share/gnome-activity-journal/src/activity_widgets.py:303: GtkWarning: gtk_box_pack: assertion `child-parent == NULL' failed self.pack_end(hbox) Error from Zeitgeist engine: org.freedesktop.DBus.Python.KeyError: Traceback (most recent call last): File /usr/lib/pymodules/python2.6/dbus/service.py, line 702, in _message_cb retval = candidate_method(self, *args, **keywords) File /usr/share/zeitgeist/_zeitgeist/engine/remote.py, line 253, in FindEvents event_templates, storage_state, num_events, result_type, sender)) File /usr/share/zeitgeist/_zeitgeist/engine/main.py, line 395, in find_events return self._find_events(1, *args) File /usr/share/zeitgeist/_zeitgeist/engine/main.py, line 382, in _find_events result = self.get_events(ids=[row[0] for row in result], sender=sender) File /usr/share/zeitgeist/_zeitgeist/engine/main.py, line 187, in get_events event = self._get_event_from_row(row) File /usr/share/zeitgeist/_zeitgeist/engine/main.py, line 152, in _get_event_from_row setattr(event, field, getattr(self, _ + field).value(row[field])) File /usr/share/zeitgeist/_zeitgeist/engine/sql.py, line 430, in value return self._inv_dict[id] KeyError: 1 ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695297] Re: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken
** Branch linked: lp:zeitgeist -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695297 Title: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken Status in Zeitgeist Framework: Fix Released Bug description: I'm talking about def _register_data_source_python_api(self, *args): mainloop = gobject.MainLoop() self.client.register_data_source(*args, reply_handler=lambda: mainloop.quit(), error_handler=lambda: fail(Error registering data-source)) mainloop.run() * In error_handler it should be self.fail(...) * register_data_source() has no reply_handler/error_handler arguments On the other hand, this method isn't used anywhere @Siegfried, I think you added this code recently, can you please fix it. Thanks so much ;) ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695297] Re: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken
Yeah, removed that. Thanks for spotting. ** Changed in: zeitgeist Importance: Undecided = Low ** Changed in: zeitgeist Status: New = Fix Released ** Changed in: zeitgeist Milestone: None = 0.7.0 ** Changed in: zeitgeist Assignee: (unassigned) = Siegfried Gevatter (rainct) -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695297 Title: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken Status in Zeitgeist Framework: Fix Released Bug description: I'm talking about def _register_data_source_python_api(self, *args): mainloop = gobject.MainLoop() self.client.register_data_source(*args, reply_handler=lambda: mainloop.quit(), error_handler=lambda: fail(Error registering data-source)) mainloop.run() * In error_handler it should be self.fail(...) * register_data_source() has no reply_handler/error_handler arguments On the other hand, this method isn't used anywhere @Siegfried, I think you added this code recently, can you please fix it. Thanks so much ;) ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695363] [NEW] Python API doesn't expose DataSourceRegistry's enabled status
Public bug reported: def set_data_source_enabled_callback(self, enabled_callback): [...] # TODO: Make it possible to access the return value! ** Affects: zeitgeist Importance: Low Status: New ** Changed in: zeitgeist Importance: Undecided = Low ** Changed in: zeitgeist Milestone: None = 0.8.0 -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695363 Title: Python API doesn't expose DataSourceRegistry's enabled status Status in Zeitgeist Framework: New Bug description: def set_data_source_enabled_callback(self, enabled_callback): [...] # TODO: Make it possible to access the return value! ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695311] Re: DataSourceRegistry is a huge performance bottleneck
** Changed in: zeitgeist Importance: Undecided = Medium ** Changed in: zeitgeist Status: New = In Progress ** Changed in: zeitgeist Milestone: None = 0.7.0 ** Changed in: zeitgeist Assignee: (unassigned) = Siegfried Gevatter (rainct) -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695311 Title: DataSourceRegistry is a huge performance bottleneck Status in Zeitgeist Framework: In Progress Bug description: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Merge] lp:~zeitgeist/zeitgeist/bug695311 into lp:zeitgeist
Siegfried Gevatter has proposed merging lp:~zeitgeist/zeitgeist/bug695311 into
lp:zeitgeist.
Requested reviews:
Zeitgeist Framework Team (zeitgeist)
Related bugs:
#695311 DataSourceRegistry is a huge performance bottleneck
https://bugs.launchpad.net/bugs/695311
For more details, see:
https://code.launchpad.net/~zeitgeist/zeitgeist/bug695311/+merge/44850
--
https://code.launchpad.net/~zeitgeist/zeitgeist/bug695311/+merge/44850
Your team Zeitgeist Framework Team is requested to review the proposed merge of
lp:~zeitgeist/zeitgeist/bug695311 into lp:zeitgeist.
=== modified file '_zeitgeist/engine/extension.py'
--- _zeitgeist/engine/extension.py 2010-12-26 09:18:11 +
+++ _zeitgeist/engine/extension.py 2010-12-29 15:41:21 +
@@ -62,6 +62,15 @@
def __init__(self, engine):
self.engine = weakref.proxy(engine)
+ def unload(self):
+
+ This method gets called before Zeitgeist stops.
+
+ Execution of this method isn't guaranteed, and you shouldn't do
+ anything slow in there.
+
+ pass
+
def pre_insert_event(self, event, sender):
Hook applied to all events before they are inserted into the
=== modified file '_zeitgeist/engine/extensions/datasource_registry.py'
--- _zeitgeist/engine/extensions/datasource_registry.py 2010-10-19 13:54:12 +
+++ _zeitgeist/engine/extensions/datasource_registry.py 2010-12-29 15:41:21 +
@@ -72,17 +72,18 @@
dbus.service.Object.__init__(self, dbus.SessionBus(),
REGISTRY_DBUS_OBJECT_PATH)
+ self._registry = {}
if os.path.exists(DATA_FILE):
try:
-self._registry = [DataSource.from_list(
- datasource) for datasource in pickle.load(open(DATA_FILE))]
+with open(DATA_FILE) as data_file:
+ for datasource in pickle.load(data_file):
+ ds = DataSource.from_list(datasource)
+ self._registry[ds[DataSource.UniqueId]] = ds
log.debug(Loaded data-source data from %s % DATA_FILE)
except Exception, e:
log.warn(Failed to load data file %s: %s % (DATA_FILE, e))
-self._registry = []
else:
log.debug(No existing data-source data found.)
- self._registry = []
self._running = {}
# Connect to client disconnection signals
@@ -93,49 +94,46 @@
)
def _write_to_disk(self):
- data = [DataSource.get_plain(datasource) for datasource in self._registry]
+ data = [DataSource.get_plain(datasource) for datasource in
+ self._registry.itervalues()]
with open(DATA_FILE, w) as data_file:
pickle.dump(data, data_file)
#log.debug(Data-source registry update written to disk.)
- def _get_data_source(self, unique_id):
- for datasource in self._registry:
- if datasource.unique_id == unique_id:
-return datasource
-
def pre_insert_event(self, event, sender):
for (unique_id, bus_names) in self._running.iteritems():
if sender in bus_names:
-datasource = self._get_data_source(unique_id)
+datasource = self._registry[unique_id]
# Update LastSeen time
datasource.last_seen = get_timestamp_for_now()
-self._write_to_disk()
# Check whether the data-source is allowed to insert events
if not datasource.enabled:
return None
return event
+ def unload(self):
+ self._write_to_disk()
+
# PUBLIC
def register_data_source(self, unique_id, name, description, templates):
source = DataSource(str(unique_id), unicode(name), unicode(description),
map(Event.new_for_struct, templates))
- for datasource in self._registry:
- if datasource == source:
-datasource.update_from_data_source(source)
-self.DataSourceRegistered(datasource)
-return datasource.enabled
- self._registry.append(source)
+ if unique_id in self._registry:
+ datasource = self._registry[unique_id]
+ datasource.update_from_data_source(source)
+ else:
+ datasource = self._registry[unique_id] = source
self._write_to_disk()
- self.DataSourceRegistered(source)
- return True
+ self.DataSourceRegistered(datasource)
+ return datasource.enabled
# PUBLIC
def get_data_sources(self):
- return self._registry
+ return self._registry.values()
# PUBLIC
def set_data_source_enabled(self, unique_id, enabled):
- datasource = self._get_data_source(unique_id)
+ datasource = self._registry[unique_id]
if not datasource:
return False
if datasource.enabled != enabled:
@@ -244,11 +242,10 @@
return
uid = uid[0]
- datasource = self._get_data_source(uid)
+ datasource = self._registry[uid]
# Update LastSeen time
datasource.last_seen = get_timestamp_for_now()
- self._write_to_disk()
strid = %s (%s) % (uid, datasource.name)
log.debug(Client disconnected: %s % strid)
=== modified file 'test/remote-test.py'
--- test/remote-test.py 2010-12-29 13:31:10 +
+++ test/remote-test.py 2010-12-29 15:41:21 +
@@ -455,6 +455,7 @@
# Verify that they have been inserted correctly
datasources = list(self.client._registry.GetDataSources())
self.assertEquals(len(datasources), 2)
+ datasources.sort(key=lambda x:
[Zeitgeist] [Bug 695311] Re: DataSourceRegistry is a huge performance bottleneck
** Branch linked: lp:~zeitgeist/zeitgeist/bug695311 -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695311 Title: DataSourceRegistry is a huge performance bottleneck Status in Zeitgeist Framework: In Progress Bug description: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 691690] Re: DataSourceRegistry should have GetDataSourceForId method
(Uhm, I had send out an e-mail about this earlier but it looks like Launchpad lost it:) There's a typo in my first comment, what I meant is b) or c), not a) or b). That is, the solution I'd prefer is a). After that I'd choose b), since a GetDataSourceForId method by itself (without the signals) isn't really useful -- you need to know the IDs to ask for them. Thoughts? -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/691690 Title: DataSourceRegistry should have GetDataSourceForId method Status in Zeitgeist Framework: Confirmed Bug description: DataSourceRegistry should have a way to get a DataSource from it's id especially because the DataSourceEnabled signal only sends an id rather than a DataSource in the signal. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
Re: [Zeitgeist] [Merge] lp:~zeitgeist/zeitgeist/bug695311 into lp:zeitgeist
Review: Approve forgot the +1 -- https://code.launchpad.net/~zeitgeist/zeitgeist/bug695311/+merge/44850 Your team Zeitgeist Framework Team is subscribed to branch lp:~zeitgeist/zeitgeist/bug695311. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 691690] Re: DataSourceRegistry should have GetDataSourceForId method
Oh, I see what you meant. Yeah, b would be helpful I believe from our standpoint. Then the signals would be consistent. -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/691690 Title: DataSourceRegistry should have GetDataSourceForId method Status in Zeitgeist Framework: Confirmed Bug description: DataSourceRegistry should have a way to get a DataSource from it's id especially because the DataSourceEnabled signal only sends an id rather than a DataSource in the signal. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
