one (perhaps insane) idea to make an all-lisp backend easier to implement was to leverage the underlying file system ala ZODB directory storage, since the file system is probably using B-trees anyways. there are fairly good architecture docs on
http://dirstorage.sourceforge.net/ tokyo cabinet looks good too. b On Feb 13, 2008 7:41 AM, Ian Eslick <[EMAIL PROTECTED]> wrote: > In general, I'm with Henrik on this. I'd rather see us get Elephant > to a reasonable degree of feature completeness before we start to add > more non-lisp datastore functionality. You can use postmodern for > licensing purposes and BDB for performance. > > The answer to all of this, I think, is having a native lisp version > that has BDB's performance and no licensing restrictions. Then > supporting the other two becomes: Postmodern for a higher degree of > reliability as well as for distributed systems and BDB for legacy > reasons. > > I have a pretty good idea in my head of what an all-lisp backend > requires and having one would lay to rest all of these discussions of > bringing up "yet another backend". Edi Weitz and I discussed > collaborating on this, but unfortunately he had some other projects > that took priority. > > Is there a small critical mass of people out there that care enough > about this that they'd be willing to contribute to such a project? I > don't have the time to do it on my own, but if we broke it up into > small projects over the next handful of months, I don't think it's a > ton of work. I can put in a solid chunk of integration work in mid to > late April. > > So what is involved? > > The tricky problems I've discovered so far are: > - An efficient model of BTree-like storage for Elephant > 1) BDB-like paged data + explicit page cache + operations over fields > 2) Something more customized? > - Efficient pointer-based indexing (BTtree plus ptrs to data in main > BTree) > - Performing sorting and searching on serialized data rather than > having to > deserialize to sort as in the clsql backend (required to do BTree > insertions) > - Transaction/logging architecture; how to store transaction data, > track conflicts, etc. > (at lisp layer, in page cache ala BDB?) > multi-thread and multi-process safe? > - locking to enable transactions on all 3 platforms; multi-process safe? > (Is there a free library that has a C library that does this already, > I think having a simple library that compensates for some of the > missing > features in lisps is fine) > > Some additional considerations: > - Do we add support for persistent heap garbage collection? > - Do we want to add supports for large persistent sets? > - Do we want a server mode for N:1 distributed transactions? > > This is by no means a trivial design, but I think if we sketched out > the architecture there are a set of subsystems that could be made > somewhat independent: > - BTrees and disk storage > - Database maintenance ops: (reconstruct DB from log files, dump db, > optimize, etc) > - transaction support and logging > - low-level locking library > - online garbage collection > > Cheers, > Ian > > > On Feb 13, 2008, at 10:11 AM, Henrik Hjelte wrote: > > > I had never heard of this project, but I it seems that Tokyo Cabinet > > describes itself as fast, has transactions and can handle multiple > > clients which is good. And it has a tcp/ip interface and protocol so > > you wouldn't even need uffi/cffi to interface it from Lisp. Tokyo > > cabinet seems to map to the bdb model good, so it should probably be > > easier to do an interface than the sql interfaces. One observation > > though, do we need yet another backend at this time, there are other > > things to fix first on my personal wishlist. > > > > Henrik > > _______________________________________________ > > elephant-devel site list > > elephant-devel@common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel@common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > _______________________________________________ elephant-devel site list elephant-devel@common-lisp.net http://common-lisp.net/mailman/listinfo/elephant-devel
