Hi guys, just a mail to explain what we are currently working on wrt Mavibot.
Recently, we have had a few user's mail and issues filed for ApacheDS where Mavibot is being used. A couple of big issues has been fixed : typically, browsing a BTree was not possible in some corner cases, due to a bad implementation of a cursor. We also expect to use Mavibot more and more in the next few months. Fortress, for instance, which has a huge tests list, would benefit from the speedup. Now, a few big things has to be implemented. I'm currently working on the bulkLoader, which is workinhg when we have tuples with single values, but it fails when we inject tuples with multiple values, when we hit the value threashold (when we have more than 8 values, we don't store them in an array, but we use a sub-btree). It's a bit complex to implement, as we have to call the bulkloader inside the bulkloader, and be careful not to re-write pages that we already have written previously (this is what is currently happening atm, we create the sub-btree, which uses 3 pages, then we write the pages, and we update the previous sub-btree pages, which is a waste). At this point, I think an improvement would be to merge the BtreeHeader page and the BtreeInfo page, but taking care of not serializing the BtreeInfo over and over. That would save some space on the disk, especially for sub-btrees. Another option would be to avoid creating a btree-info for sub-btrees, as they inherit everything from their associated parent btree. The next big step is to finish the implementation of the free page management. This is really a must have. The first version should be a simple one, with a thread in charge of reclaiming the free pages when the oldest revision in used is released. This thread would then reclaim all the associated pages, and put them back in the free list, and goes up reclaiming the versions that are orlder than the new oldest version. Btw, this reclaimer thread must be the writer thread, as we don't want to add some locks between those two threads (we need to figure out what would be the best mechanism here). A better algorithm, where we would reclaim unused pages in the middle of used revision, has to be implemented later. Last, not least, we have to add a transaction mechanism that allows the updates to be done in batches, and in memory. Ie, we don't update the disk, we just keep the modified pages in memory, so that when a page has to be updated twice, we don't need to fetch it from the disk and write it back. A simple hashMap keeping the pages associated with their offset should be enough, but that means we should check on this hashmap before fetching the page on the disk. This feature brings to critical improvements : - first, it allows a set of updates made across many btrees to be seen as one single atomic operation. For the LDAP server, this is absolutely critical, as we always update many btrees when we update an entry. - second, this would speedup the updates, as the root pages, the headers and more important, the free paages and the record manager header will only be updated once, when they are updated once per update. I think the gain could be around 20%, minimum. Once those features are implemented, I'm quite sure that Mavibot 1.0 wil be in good shape. Thoughts ?
