hi Emmanuel, Thank you so much for the quick reply.
> check on https://lists.apache.org/thread.html/98d4a01b66caf0d39af4c437920283a5fa25ade75fe9f4262fa2bae1@%3Cdev.directory.apache.org%3E) I have read this report. Excellent job! Impressive results! Look forward to using Mavibot as production soon. > No progress so far, due to time compression (new born baby, day job..) Congratulations to you to have a baby! :) I will start to implement the LMDB backend from tomorrow immediately. And will post results to ApacheDS community when I complete and have performance test results. I am so glad to have the opportunity to contribute. > LBDM brings zero-copy, because it's based on Memory Mapped File, but OTOH you won't be able to benefit from it as much as you'd like, due to the need for serialization/deserialization. Basically, you will need to add an intermediary cache, otherwise you'll get degraded performances because when you fetch a page from disk, you will have to deserialize the whole content, even if you don't need more than just a key and value. Appreciate for this advice. I haven't been aware of the effect of this cache for LMDB before. I will test and use cache to optimize this. Thanks, Zack Liu ________________________________ From: Emmanuel Lécharny <[email protected]> Sent: Friday, March 10, 2017 4:32 PM To: Zack Liu Subject: Re: Could I know the progress of LMDB backend for ApacheDS and I want have a try on it Hi, Le 10/03/2017 à 16:38, Zack Liu a écrit : > Hi Emmanuel and Zheng, > > Recently I am using ApacheDS as the storage of my project but worried about > the DB backend of it. The index of JDBM may be corrupted somehow and it is > seems hard to resolve that issue. Another concern is that it is too slow to > turn on the syncOnWrite or facing the risk of loss of data when crashing or > power off if turn off the syncOnWrite option. Mavibot maybe a choice but it > is not for production yet currently. I have resumed teh work on Mavibot one month ago. Addition are now working, deletion is under work. This version will support transactions across multiple B-trees and operations (ie, you will be able to wait before committing the data to disk). Currently, I have conducted some tests where transactions are committed across 3 B-trees and committed after various number of ops (from 1 opeeration to 5000 batched operations : check on https://lists.apache.org/thread.html/98d4a01b66caf0d39af4c437920283a5fa25ade75fe9f4262fa2bae1@%3Cdev.directory.apache.org%3E) > > So I consider to implement the LMDB partition and index myself to provide a > stable DB backend. Good ! We would welcome such an implementation ! > I read this email threads > https://www.mail-archive.com/[email protected]/msg48554.html and know > that you guys want to have a try to implement LMDB backend for ApacheDS. So > how about the progress? I want to have a try on LMDB and it really helps if > there is some work I can leverage. No progress so far, due to time compression (new born baby, day job..) > > LMDB claims it supports “Multiple sub-databases may be created with > transactions covering all sub-databases”. It does. > So maybe the locks in the AbstractBTreePartition can be removed and use LMDB > transaction instead. Absolutely. This is also true for Mavibot, and it's also an absolute requirement for ApacheDS (and FTR it's the cause of the database corruption...) > It seem there’s a hope that LMDB may provides faster read/write speed and a > consistent DB backend. I would say it's a certainty. Mavibot will also be faster than JDBM : the tests we conducted 2 years ago with teh Mavibot partition shown that additions were 5 times faster, and searches were twice faster (from teh top of my head). LBDM brings zero-copy, because it's based on Memory Mapped File, but OTOH you won't be able to benefit from it as much as you'd like, due to the need for serialization/deserialization. Basically, you will need to add an intermediary cache, otherwise you'll get degraded performances because when you fetch a page from disk, you will have to deserialize the whole content, even if you don't need more than just a key and value. But that is just optimisation. Feel free to ask more if needed ! -- Emmanuel Lecharny Symas.com directory.apache.org
