Emmanuel Lécharny wrote:
3) Switch to Mavibot
Mavibot is a labs I started last year, which implements a MVCC Btree in
memory. It's working, t's fast, but it's in memory only atm (although
the data are regularly flushed on disk).
We need to improve it so that we add those features :
- Make it backed with a physical layer that access data on disk, and
avoid storing everything in memory
- Add a raw load feature
- Add a global transaction system, cross trees.
The last feature is probably easier to implement, as we do have a
version associated with each new modification : if we retin the version
we started with, we can gather all the btree's revision, and validate
them all in one shot, or rollback them all in one shot (a revision is
visible only if it has been validated).
One very interestig aspect of Mavibot is that you can update the btrees
concurrently with searches : searches are done on older revisions, which
don't interract with the pages being added.
I think there is one to two more month of work on Mavibot to get it to
the level we need.
Proposal
--------
IMO, we should investigate JDBM4 asap. It might solve many of the issues
we have, at a low price. Improving the existing JDBM is just a waste of
our precious time.
I the long term, I would favor Mavibot for a couple of reasons :
- first, it's a MVCC base, so we can conduct concurrent searches and
modifications (although we can't do many modifications at the same time).
- second, we have a complete understanding of the code : we wrote it.
It's easier to understand the impact of the modifications we hve done on it.
- third, we can have a dedicated backend based on the current Mavibot
code, where all the data are in memory (that would work with a few tens
of thousands entries, and the performances will be blazing fast).
I'd like to know what's your opinion.
Is it a requirement to stick to pure java? We recently had someone inquire
about a java wrapper for our Lightning DB library. Developing that could get
you a big boost without as much development effort, assuming JNI is practical
for deployments.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
