I realize this question my be a bit off topic but I am trying to
determine the best structure for storing a large schema where some
attributes are lists or dictionaries. There are about 70 attributes in
the schema and I am trying to choose a structure that will not
necessarily have to hold a bunch of empty space. I have a base schema of
approximately 30 attributes and other that subclass from it with the
largest being about 70.
I had originally thought of RDF at the onset and built a datastore with
rdflib using a relational database. I chose this option because when
ZODB gets larger it takes plenty of RAM. Problem here is the number of
accesses to gather a complete object. It is pretty efficient from a
storage perspective since if an object does not have particular
attributes, you are not storing them and all items in the store are
unique. I was not duplicating one piece of data. But say you wanted to
present a page consisting of 20 items or do a search. Gathering this up
takes much time when you are hitting a disk how many times to gather up
just a single object so query times were unacceptable and loading rdf
from outside sources also took a very long time. The data store grows
into millions of records so you better have a pretty sweet rdb server
with lots of RAM also.
I had dismissed a relational database on its own since the data does not
lend to a row and I may want to add to the schema at some point in
time which could mean some pretty ugly business this way. But then I saw
the vertical example in the examples folder of SQLAlchemy that can do
something to create dynamic fields as necessary to potentially avoid
this kind of hassle.
The ZODB provides the flexibility and Generations could work well for
future updates so this looks very good but how efficient is it if 15% of
the attributes have data and 85% do not?
I have also been experimenting with hybrid pickle / rdb storage so that
the attributes that will receive the most attention are stored as fields
and the full record is stored as pickle that is unpickled for views and
In any case. Thought I'd ask since I am concerned about the efficiency
of storage and speed of access both. If rdf access was fast then it
would be great but this had not been the case. I just thought there may
be some other ideas on this or someone could advise on the efficiency of
ZODB when in some cases, uses will be selective about which attributes
are important to them.
Zope3-users mailing list