Kimbro Staken wrote:
[...]
I completely agree that we need to break the documents up more and I also agree with your reasons for needing to do so, however I'm very hesitant to go all the way to individual nodes. The are several reasons for this. One is that the current BTree is really slow, in fact it's faster to store things using the FSFiler then it is with the BTreeFiler. If we start breaking documents up at that granularity performance is really going to go through the floor. This is especially true if we store one node per page. A typical node will be only a few bytes to a few hundred bytes, but pages are supposed to be the same size as the OS page size (usually 4 or 8K). So we're talking about having to read large numbers of pages for relatively small documents.
Earlier in the year I pointed the members of this list to Lee Iverson's NODAL project. Lee was at SRI and now Univ. of British Columbia. I don't in any way claim to be a database expert, but one might note that the approach of NODAL is similar to that taken by ReiserFS, and performance of the latter is quite good -- it's optimized for use with many small documents.
For reference, Lee's NODAL project can be found at:
NODAL White Paper (at SRI) http://www.ai.sri.com/~leei/OHS/
NODAL on SourceForge.net: http://nodal.sourceforge.net/
I don't expect NODAL directly applies to our work with Xindice, but there may be some valuable ideas there that could inform it.
Murray
...................................................................... Murray Altheim <http://kmi.open.ac.uk/people/murray/> Knowledge Media Institute The Open University, Milton Keynes, Bucks, MK7 6AA, UK
If you're the first person in a new territory,
you're likely to get shot at.
-- ma