In general, I'm with Henrik on this. I'd rather see us get Elephant
to a reasonable degree of feature completeness before we start to add
more non-lisp datastore functionality. You can use postmodern for
licensing purposes and BDB for performance.
The answer to all of this, I think, is having a native lisp version
that has BDB's performance and no licensing restrictions. Then
supporting the other two becomes: Postmodern for a higher degree of
reliability as well as for distributed systems and BDB for legacy
reasons.
I have a pretty good idea in my head of what an all-lisp backend
requires and having one would lay to rest all of these discussions of
bringing up "yet another backend". Edi Weitz and I discussed
collaborating on this, but unfortunately he had some other projects
that took priority.
Is there a small critical mass of people out there that care enough
about this that they'd be willing to contribute to such a project? I
don't have the time to do it on my own, but if we broke it up into
small projects over the next handful of months, I don't think it's a
ton of work. I can put in a solid chunk of integration work in mid to
late April.
So what is involved?
The tricky problems I've discovered so far are:
- An efficient model of BTree-like storage for Elephant
1) BDB-like paged data + explicit page cache + operations over fields
2) Something more customized?
- Efficient pointer-based indexing (BTtree plus ptrs to data in main
BTree)
- Performing sorting and searching on serialized data rather than
having to
deserialize to sort as in the clsql backend (required to do BTree
insertions)
- Transaction/logging architecture; how to store transaction data,
track conflicts, etc.
(at lisp layer, in page cache ala BDB?)
multi-thread and multi-process safe?
- locking to enable transactions on all 3 platforms; multi-process safe?
(Is there a free library that has a C library that does this already,
I think having a simple library that compensates for some of the
missing
features in lisps is fine)
Some additional considerations:
- Do we add support for persistent heap garbage collection?
- Do we want to add supports for large persistent sets?
- Do we want a server mode for N:1 distributed transactions?
This is by no means a trivial design, but I think if we sketched out
the architecture there are a set of subsystems that could be made
somewhat independent:
- BTrees and disk storage
- Database maintenance ops: (reconstruct DB from log files, dump db,
optimize, etc)
- transaction support and logging
- low-level locking library
- online garbage collection
Cheers,
Ian
On Feb 13, 2008, at 10:11 AM, Henrik Hjelte wrote:
I had never heard of this project, but I it seems that Tokyo Cabinet
describes itself as fast, has transactions and can handle multiple
clients which is good. And it has a tcp/ip interface and protocol so
you wouldn't even need uffi/cffi to interface it from Lisp. Tokyo
cabinet seems to map to the bdb model good, so it should probably be
easier to do an interface than the sql interfaces. One observation
though, do we need yet another backend at this time, there are other
things to fix first on my personal wishlist.
Henrik
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
_______________________________________________
elephant-devel site list
elephant-devel@common-lisp.net
http://common-lisp.net/mailman/listinfo/elephant-devel
