The second thing I'm exploring in this area, is a different way to partition an XML document into pieces. I've been exploring horizontal partitioning where all sibling nodes would be stored together in a single record. In a lot of ways this is pretty similar to how relational databases work, i.e. each set of sibling nodes represents a tuple within a relation, and the entire document is made up through joins between relations. In our case our relations will only have one tuple, but I'm thinking the general concept is the same. Relational databases don't store things at the attribute level, they store full tuples together as a unit. We'd still use the compression format for the sibling level and just extend it so that each node can find the proper BTree node with its children in it. Anyway, I've done a fair amount of exploratory work on this including some ideas for structural and content based indexes, but I need more time before I'm willing to say whether it's a good idea or not. So far I do kind of like it.
It sure sounds good. Can you point me out to papers and/or link describing this approach?
OTOH, while I'm not that versed into database core technology, I'd like to know (Tom?) if the research on the Xalan table model has been succesful and interesting or not. I took a look at the DTM, and it looks pretty cool to me. This would bring an heavy dependancy on Xalan, which might not be a Good Thing, but I'm wondering if we might leverage the hard research work that the Xalan team has done about topics that might be commons to both.
And again... I'm still making up my mind on it, but the few tests I did were encouraging: I've been playing with the Cocoon XMLCompiler/Interpreter stuff, and I'm wondering if that, coupled with a strong indexing scheme, might do the trick and solve the problem in a (maybe too) easy way. The idea here is to store a stream of compiled SAX events, which is nothing but a ByteArray (think of it as a binary and persistent representation of a SAX parsed document). Performance-wise this might ease the memory problems, since all you have to do is to "play" the events you need (I'm thinking about an indexing system that records the position of sensible events in the stream). Storage-wise it would probably be smaller than any DOM implementation (true, indexes might need to be bigger in this case). Network-wise we could gain something too, since the server would just send the ByteArray (or fragments of it) through the network and let the client "play" the events it receives (and by using simple utility classes we could easily convert to DOM, if needed). The biggest problem I see as of now is about modifications, since it won't be easy to modify the compiled stream, but maybe some research on it might bring a clean solution even on that side.
As I said, these are just very early stage RT, but do you see a reason to rule them out?
Ciao,
-- Gianugo Rabellino
