[Zeitgeist] [Bug 695297] [NEW] unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken
Public bug reported: I'm talking about def _register_data_source_python_api(self, *args): mainloop = gobject.MainLoop() self.client.register_data_source(*args, reply_handler=lambda: mainloop.quit(), error_handler=lambda: fail(Error registering data-source)) mainloop.run() * In error_handler it should be self.fail(...) * register_data_source() has no reply_handler/error_handler arguments On the other hand, this method isn't used anywhere @Siegfried, I think you added this code recently, can you please fix it. Thanks so much ;) ** Affects: zeitgeist Importance: Undecided Status: New -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695297 Title: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken Status in Zeitgeist Framework: New Bug description: I'm talking about def _register_data_source_python_api(self, *args): mainloop = gobject.MainLoop() self.client.register_data_source(*args, reply_handler=lambda: mainloop.quit(), error_handler=lambda: fail(Error registering data-source)) mainloop.run() * In error_handler it should be self.fail(...) * register_data_source() has no reply_handler/error_handler arguments On the other hand, this method isn't used anywhere @Siegfried, I think you added this code recently, can you please fix it. Thanks so much ;) ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695311] [NEW] DataSourceRegistry is a huge performance bottleneck
Public bug reported: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ** Affects: zeitgeist Importance: Undecided Status: New -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695311 Title: DataSourceRegistry is a huge performance bottleneck Status in Zeitgeist Framework: New Bug description: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695311] Re: DataSourceRegistry is a huge performance bottleneck
** Attachment added: test_insert_events.py https://bugs.edge.launchpad.net/bugs/695311/+attachment/1778807/+files/test_insert_events.py -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695311 Title: DataSourceRegistry is a huge performance bottleneck Status in Zeitgeist Framework: New Bug description: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695087] Re: Invalid cache access
*** This bug is a duplicate of bug 598666 *** https://bugs.launchpad.net/bugs/598666 @Siegfried, what exactly are you benchmarking here? Eg. how many different (relevant) values are you using? How do the graphs look if you have some hundred different actors and mimetypes and a wide range of used manifestations and interpretations? -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695087 Title: Invalid cache access Status in Zeitgeist Framework: Triaged Bug description: ala...@alagos-desktop:~$ gnome-activity-journal ** Message: pygobject_register_sinkfunc is deprecated (GstObject) /usr/share/gnome-activity-journal/src/common.py:747: DeprecationWarning: object.__new__() takes no parameters GIO_FILES[subj] = object.__new__(classtype, *args, **kwargs) /usr/share/gnome-activity-journal/src/activity_widgets.py:303: GtkWarning: gtk_box_pack: assertion `child-parent == NULL' failed self.pack_end(hbox) Error from Zeitgeist engine: org.freedesktop.DBus.Python.KeyError: Traceback (most recent call last): File /usr/lib/pymodules/python2.6/dbus/service.py, line 702, in _message_cb retval = candidate_method(self, *args, **keywords) File /usr/share/zeitgeist/_zeitgeist/engine/remote.py, line 253, in FindEvents event_templates, storage_state, num_events, result_type, sender)) File /usr/share/zeitgeist/_zeitgeist/engine/main.py, line 395, in find_events return self._find_events(1, *args) File /usr/share/zeitgeist/_zeitgeist/engine/main.py, line 382, in _find_events result = self.get_events(ids=[row[0] for row in result], sender=sender) File /usr/share/zeitgeist/_zeitgeist/engine/main.py, line 187, in get_events event = self._get_event_from_row(row) File /usr/share/zeitgeist/_zeitgeist/engine/main.py, line 152, in _get_event_from_row setattr(event, field, getattr(self, _ + field).value(row[field])) File /usr/share/zeitgeist/_zeitgeist/engine/sql.py, line 430, in value return self._inv_dict[id] KeyError: 1 ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695297] Re: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken
** Branch linked: lp:zeitgeist -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695297 Title: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken Status in Zeitgeist Framework: Fix Released Bug description: I'm talking about def _register_data_source_python_api(self, *args): mainloop = gobject.MainLoop() self.client.register_data_source(*args, reply_handler=lambda: mainloop.quit(), error_handler=lambda: fail(Error registering data-source)) mainloop.run() * In error_handler it should be self.fail(...) * register_data_source() has no reply_handler/error_handler arguments On the other hand, this method isn't used anywhere @Siegfried, I think you added this code recently, can you please fix it. Thanks so much ;) ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695297] Re: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken
Yeah, removed that. Thanks for spotting. ** Changed in: zeitgeist Importance: Undecided = Low ** Changed in: zeitgeist Status: New = Fix Released ** Changed in: zeitgeist Milestone: None = 0.7.0 ** Changed in: zeitgeist Assignee: (unassigned) = Siegfried Gevatter (rainct) -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695297 Title: unused helper ZeitgeistRemoteDataSourceRegistryTest._register_data_source_python_api() is broken Status in Zeitgeist Framework: Fix Released Bug description: I'm talking about def _register_data_source_python_api(self, *args): mainloop = gobject.MainLoop() self.client.register_data_source(*args, reply_handler=lambda: mainloop.quit(), error_handler=lambda: fail(Error registering data-source)) mainloop.run() * In error_handler it should be self.fail(...) * register_data_source() has no reply_handler/error_handler arguments On the other hand, this method isn't used anywhere @Siegfried, I think you added this code recently, can you please fix it. Thanks so much ;) ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695363] [NEW] Python API doesn't expose DataSourceRegistry's enabled status
Public bug reported: def set_data_source_enabled_callback(self, enabled_callback): [...] # TODO: Make it possible to access the return value! ** Affects: zeitgeist Importance: Low Status: New ** Changed in: zeitgeist Importance: Undecided = Low ** Changed in: zeitgeist Milestone: None = 0.8.0 -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695363 Title: Python API doesn't expose DataSourceRegistry's enabled status Status in Zeitgeist Framework: New Bug description: def set_data_source_enabled_callback(self, enabled_callback): [...] # TODO: Make it possible to access the return value! ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 695311] Re: DataSourceRegistry is a huge performance bottleneck
** Changed in: zeitgeist Importance: Undecided = Medium ** Changed in: zeitgeist Status: New = In Progress ** Changed in: zeitgeist Milestone: None = 0.7.0 ** Changed in: zeitgeist Assignee: (unassigned) = Siegfried Gevatter (rainct) -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695311 Title: DataSourceRegistry is a huge performance bottleneck Status in Zeitgeist Framework: In Progress Bug description: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Merge] lp:~zeitgeist/zeitgeist/bug695311 into lp:zeitgeist
Siegfried Gevatter has proposed merging lp:~zeitgeist/zeitgeist/bug695311 into lp:zeitgeist. Requested reviews: Zeitgeist Framework Team (zeitgeist) Related bugs: #695311 DataSourceRegistry is a huge performance bottleneck https://bugs.launchpad.net/bugs/695311 For more details, see: https://code.launchpad.net/~zeitgeist/zeitgeist/bug695311/+merge/44850 -- https://code.launchpad.net/~zeitgeist/zeitgeist/bug695311/+merge/44850 Your team Zeitgeist Framework Team is requested to review the proposed merge of lp:~zeitgeist/zeitgeist/bug695311 into lp:zeitgeist. === modified file '_zeitgeist/engine/extension.py' --- _zeitgeist/engine/extension.py 2010-12-26 09:18:11 + +++ _zeitgeist/engine/extension.py 2010-12-29 15:41:21 + @@ -62,6 +62,15 @@ def __init__(self, engine): self.engine = weakref.proxy(engine) + def unload(self): + + This method gets called before Zeitgeist stops. + + Execution of this method isn't guaranteed, and you shouldn't do + anything slow in there. + + pass + def pre_insert_event(self, event, sender): Hook applied to all events before they are inserted into the === modified file '_zeitgeist/engine/extensions/datasource_registry.py' --- _zeitgeist/engine/extensions/datasource_registry.py 2010-10-19 13:54:12 + +++ _zeitgeist/engine/extensions/datasource_registry.py 2010-12-29 15:41:21 + @@ -72,17 +72,18 @@ dbus.service.Object.__init__(self, dbus.SessionBus(), REGISTRY_DBUS_OBJECT_PATH) + self._registry = {} if os.path.exists(DATA_FILE): try: -self._registry = [DataSource.from_list( - datasource) for datasource in pickle.load(open(DATA_FILE))] +with open(DATA_FILE) as data_file: + for datasource in pickle.load(data_file): + ds = DataSource.from_list(datasource) + self._registry[ds[DataSource.UniqueId]] = ds log.debug(Loaded data-source data from %s % DATA_FILE) except Exception, e: log.warn(Failed to load data file %s: %s % (DATA_FILE, e)) -self._registry = [] else: log.debug(No existing data-source data found.) - self._registry = [] self._running = {} # Connect to client disconnection signals @@ -93,49 +94,46 @@ ) def _write_to_disk(self): - data = [DataSource.get_plain(datasource) for datasource in self._registry] + data = [DataSource.get_plain(datasource) for datasource in + self._registry.itervalues()] with open(DATA_FILE, w) as data_file: pickle.dump(data, data_file) #log.debug(Data-source registry update written to disk.) - def _get_data_source(self, unique_id): - for datasource in self._registry: - if datasource.unique_id == unique_id: -return datasource - def pre_insert_event(self, event, sender): for (unique_id, bus_names) in self._running.iteritems(): if sender in bus_names: -datasource = self._get_data_source(unique_id) +datasource = self._registry[unique_id] # Update LastSeen time datasource.last_seen = get_timestamp_for_now() -self._write_to_disk() # Check whether the data-source is allowed to insert events if not datasource.enabled: return None return event + def unload(self): + self._write_to_disk() + # PUBLIC def register_data_source(self, unique_id, name, description, templates): source = DataSource(str(unique_id), unicode(name), unicode(description), map(Event.new_for_struct, templates)) - for datasource in self._registry: - if datasource == source: -datasource.update_from_data_source(source) -self.DataSourceRegistered(datasource) -return datasource.enabled - self._registry.append(source) + if unique_id in self._registry: + datasource = self._registry[unique_id] + datasource.update_from_data_source(source) + else: + datasource = self._registry[unique_id] = source self._write_to_disk() - self.DataSourceRegistered(source) - return True + self.DataSourceRegistered(datasource) + return datasource.enabled # PUBLIC def get_data_sources(self): - return self._registry + return self._registry.values() # PUBLIC def set_data_source_enabled(self, unique_id, enabled): - datasource = self._get_data_source(unique_id) + datasource = self._registry[unique_id] if not datasource: return False if datasource.enabled != enabled: @@ -244,11 +242,10 @@ return uid = uid[0] - datasource = self._get_data_source(uid) + datasource = self._registry[uid] # Update LastSeen time datasource.last_seen = get_timestamp_for_now() - self._write_to_disk() strid = %s (%s) % (uid, datasource.name) log.debug(Client disconnected: %s % strid) === modified file 'test/remote-test.py' --- test/remote-test.py 2010-12-29 13:31:10 + +++ test/remote-test.py 2010-12-29 15:41:21 + @@ -455,6 +455,7 @@ # Verify that they have been inserted correctly datasources = list(self.client._registry.GetDataSources()) self.assertEquals(len(datasources), 2) + datasources.sort(key=lambda x:
[Zeitgeist] [Bug 695311] Re: DataSourceRegistry is a huge performance bottleneck
** Branch linked: lp:~zeitgeist/zeitgeist/bug695311 -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/695311 Title: DataSourceRegistry is a huge performance bottleneck Status in Zeitgeist Framework: In Progress Bug description: I was wondering today why the timings for inserting events when running our testsuite (or my benchmark scripts) are looking much better than the timings in a 'real' daemon instance. Some Data: * inserting 500 events at once in my benchmarks: ~0.09 sec * inserting the same 500 events at once in a 'real' zeitgeist session: ~0.185 sec You can reproduce this tests by running the attached script like ZEITGEIST_DEFAULT_EXTENSIONS=_zeitgeist.engine.extensions.datasource_registry.DataSourceRegistry PYTHONPATH=. python /tmp/test_insert_events.py PYTHONPATH=. python /tmp/test_insert_events.py and compare outputs. To simulate the scenario of multiple running instances of the same application, uncomment the lines 26+ in the script, you will get even worse results. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 691690] Re: DataSourceRegistry should have GetDataSourceForId method
(Uhm, I had send out an e-mail about this earlier but it looks like Launchpad lost it:) There's a typo in my first comment, what I meant is b) or c), not a) or b). That is, the solution I'd prefer is a). After that I'd choose b), since a GetDataSourceForId method by itself (without the signals) isn't really useful -- you need to know the IDs to ask for them. Thoughts? -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/691690 Title: DataSourceRegistry should have GetDataSourceForId method Status in Zeitgeist Framework: Confirmed Bug description: DataSourceRegistry should have a way to get a DataSource from it's id especially because the DataSourceEnabled signal only sends an id rather than a DataSource in the signal. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
Re: [Zeitgeist] [Merge] lp:~zeitgeist/zeitgeist/bug695311 into lp:zeitgeist
Review: Approve forgot the +1 -- https://code.launchpad.net/~zeitgeist/zeitgeist/bug695311/+merge/44850 Your team Zeitgeist Framework Team is subscribed to branch lp:~zeitgeist/zeitgeist/bug695311. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp
[Zeitgeist] [Bug 691690] Re: DataSourceRegistry should have GetDataSourceForId method
Oh, I see what you meant. Yeah, b would be helpful I believe from our standpoint. Then the signals would be consistent. -- You received this bug notification because you are a member of Zeitgeist Framework Team, which is subscribed to Zeitgeist Framework. https://bugs.launchpad.net/bugs/691690 Title: DataSourceRegistry should have GetDataSourceForId method Status in Zeitgeist Framework: Confirmed Bug description: DataSourceRegistry should have a way to get a DataSource from it's id especially because the DataSourceEnabled signal only sends an id rather than a DataSource in the signal. ___ Mailing list: https://launchpad.net/~zeitgeist Post to : zeitgeist@lists.launchpad.net Unsubscribe : https://launchpad.net/~zeitgeist More help : https://help.launchpad.net/ListHelp