At 09:53 AM 10/9/2007 -1000, Brian Kirsch wrote:
We should have a meeting / discussion as to the best next steps. I
personally think the only way to get acceptable performance metrics
is to go one layer lower
and manually insert data in to our Repo / Berkeley DB. Whether this
is possible or not would require some consulting with Andi. But
basically the idea is to get as low level as
possible and bypass as much of the Python abstraction as we can. This
may even require writing some c code :)
Or better yet, reusing some:
http://pyinsci.blogspot.com/2007/07/fastest-python-database-interface.html
In fairness these numbers (up to 100,000 inserts in 3 seconds for
SQLite) are based on smaller record sizes than what we use, and fewer indexes.
However, this is precisely one of the key problems with our current
persistence strategy! That is, objects aren't very good at being
database records, precisely *because* they encapsulate quite a lot of
data into a single logical object. Databases, on the other hand, are
built around "excapsulation" -- splitting data into logically distinct chunks.
When using a database rather than an object store, you have many
options for tuning it, while leaving the front-end alone. However,
if we do this sort of thing in the repository, we end up creating yet
*another* layer of abstraction (i.e. performance drain) between us
and the data.
If in the repository, for example, we store an email as a single
Item, then its email-specific data must be accessible, even if we are
just listing it in the detail view, because it is a single item. To
improve this state of things in a database-based storage, we could
simply split triage/dashboard information to one table, and
email-specific data to another, without necessarily splitting the
Item itself. To do this in the repository, on the other hand, we
would have to actually have two different items.
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
Open Source Applications Foundation "chandler-dev" mailing list
http://lists.osafoundation.org/mailman/listinfo/chandler-dev
