[Zeitgeist] [Bug 691167] Re: The payload is sometimes mentioned as string and sometimes as array of bytes

2011-01-17 Thread Seif Lotfy 
** Changed in: zeitgeist
Milestone: 0.7.0 = None

-- 
You received this bug notification because you are a member of Zeitgeist
Framework Team, which is subscribed to Zeitgeist Framework.
https://bugs.launchpad.net/bugs/691167

Title:
  The payload is sometimes mentioned as string and sometimes as array of
  bytes

Status in Zeitgeist Framework:
  In Progress

Bug description:
  In the event serialization format, the third array is array of bytes
  or ay as dbus signature.

  When an event enters an extension, it looks like
  Event([dbus.Array([u'', u'1292500628312', u'', u'', 
u'application://foo.desktop'], signature=dbus.Signature('s')), [Subject([u'', 
u'', u'', u'', u'', u'', u''])], dbus.Array([], signature=dbus.Signature('y'))])

  which implies that payload is array of bytes

  Now look at _zeitgeist/engine/datamodel.py at line 58
  where you get the line
   popo.append(str(ev[2]))

  Really so when you do str() on dbus.Array([], signature=dbus.Signature('y'))
  you get dbus.Array([], signature=dbus.Signature('y')) instead of the 
contents of bytes converted to string

  Now when you call Event.get_plain on

  Event([dbus.Array([u'', u'1292500628312', u'', u'',
  u'application://foo.desktop'], signature=dbus.Signature('s')),
  [Subject([u'', u'', u'', u'', u'', u'', u''])], dbus.Array([],
  signature=dbus.Signature('y'))])

  you get

  [[u'', u'1292500628312', u'', u'', u'application://foo.desktop'],
  [[u'', u'', u'', u'', u'', u'', u'']], dbus.Array([],
  signature=dbus.Signature('y'))]

  Not what you expect



___
Mailing list: https://launchpad.net/~zeitgeist
Post to : zeitgeist@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zeitgeist
More help   : https://help.launchpad.net/ListHelp

[Zeitgeist] [Bug 632363] Re: Slow queries: SQL indexes not used

2011-01-17 Thread Seif Lotfy 
** Changed in: zeitgeist
Milestone: 0.7.0 = None

-- 
You received this bug notification because you are a member of Zeitgeist
Framework Team, which is subscribed to Zeitgeist Framework.
https://bugs.launchpad.net/bugs/632363

Title:
  Slow queries: SQL indexes not used

Status in Zeitgeist Framework:
  Confirmed

Bug description:
  THE PROBLEM:
  I am seeing query times around 200ms (and 150ms with my latest performance 
tweak in trunk), which surprised me as quite slow since my queries where quite 
simple. I had expected times around 1-2ms.

  Reading up on the sqlite documentation I see that the queries we
  generate are pretty far from optimized in an sqlite world [1]. The
  case is that when ever you use an OR sqlite will no longer use an
  index. Thus this query is NOT using the indexes:

SELECT * FROM event WHERE interpretation=1 OR interpretation=2

  But if we rewrite it using IN instead the indexes will be used:

SELECT * FROM event WHERE interpretation IN (1, 2)

  This also explains the case where Michal where seeing tremendously
  slow query times when searching for a big range of mimetypes.

  Looking in _zeitgeist.engine.sql.WhereClause.add_text_condition() I am
  also pretty sure we are not using the indexes for prefix queries (eg
  file://home/*).

  
  THE SOLUTION
  I *definitely* don't think we should panic and feverishly start rewriting our 
query compilation. Here's what I propose:

   1) Implement an envvar ZEITGEIST_DEBUG_QUERY_PLANS which will spit
  out all our SQL calls and the query plans for each of our calls. The
  query plan will tell us how the db is queried and which indexes are
  used if any. The query plan is obtained by prefixing the SQL statement
  with EXPLAIN QUERY PLAN.

   2) Collect some useful intelligence with this new tool, and generally
  learn more about how we can optimize sqlite queries. A big question
  here is how the event_view VIEW impacts the query plan.

   3) Write a new template - SQL compilation engine that generates SQL
  optimized for sqlite. We can actually be quite clever about grouping
  our OR statements into IN clauses - but it will be tricky to get
  right.

  NOTE: That this doesn't imply any change in the public API or event
  template system. That would be the wrong solution imho. Our current
  API is nice and simple by my standards. Let's keep it that way.

  [1]: See fx the section Using Indexes To Speed Searching in
  http://www.sqlite.org/vdbe.html



___
Mailing list: https://launchpad.net/~zeitgeist
Post to : zeitgeist@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zeitgeist
More help   : https://help.launchpad.net/ListHelp

[Zeitgeist] [Bug 686732] Re: add cache to get events

2011-01-17 Thread Seif Lotfy 
** Changed in: zeitgeist
   Status: In Progress = Fix Committed

-- 
You received this bug notification because you are a member of Zeitgeist
Framework Team, which is subscribed to Zeitgeist Framework.
https://bugs.launchpad.net/bugs/686732

Title:
  add cache to get events

Status in Zeitgeist Framework:
  Fix Committed

Bug description:
  Sometimes the same events get requested by several different apps.
  having a cache for the get_events cache can cut off some time, and
  enhance performance.



___
Mailing list: https://launchpad.net/~zeitgeist
Post to : zeitgeist@lists.launchpad.net
Unsubscribe : https://launchpad.net/~zeitgeist
More help   : https://help.launchpad.net/ListHelp