Hi, I'm one of the contributors to JDMB3 serialization implementation. Actually earlier this year I made it much faster than before (2 orders of magnitude). And BTW, I'm also a contributor to Kryo and protostuff-runtime.
I find this discussion very interesting, so let me provide my two cents as well. First of all, I just want to mention that while working on improving JDBM's serialization, I extracted the serialization part of the JDBM into a dedicated serialization library, which I called Quickser. You can find it on GitHub: https://github.com/romix/quickser It is really very fast, often faster than Kryo and protostuff. Since Quickser contains only serialization-related stuff from JDBM/MapDB, it is easier to use it if you just want to add yet another serialization method to DM without any DB related functionality. It could even make sense, if MapDB would use Quickser for serialization instead of having both DB and serialization related functionality in one pot. @Jan: What do you think about it? I understand that you don't like external dependencies. But Quickser is not really external. It is more or less a copy of JDMBs serialization-related classes. On Wed, Nov 7, 2012 at 9:49 AM, Jan Kotek <[email protected]> wrote: > Hi, > > >> 1. DirectMemory could make good use of mapdb to serialize least >> frequently used items to disk and free memory >> 2. DirectMemory could implement a MapDB disk based store in addition >> to >> the bytebuffer and unsafe ones > > The only problem may be that MapDB currently does not support concurrent > transactions (it has only one single global transaction). > Not sure if it could be a problem. > > However it implements ConcurrentMap, so it is possible to swap items > atomically > > >> 3. MapDB could take advantage of DM's componentization approach to >> support multiple serializers (we believe each one has its advantages >> in >> different scenarios) > > MapDB already supports alternative serializers. User can supply their own on > Map (similar to table) creation. > I would love to integrate stuff from lightning serializer. > > >> 4. MapDB could use DM to write items to an off-heap before writing to >> disk (asynchronously) to improve speed > > Not sure it would be practical. MapDB already uses memory mapped files so > effect would be very similar. My tests shows that there is only 50% > performance difference between inMemory store and onDisk store. > > Currently MapDB has only heap based inMemory store. But implementing off > heap memory store is trivial and I will do it soon. This is very nice to know. Looking forward to see this feature. May be you should use DM for it? > >> 5. We could merge our serialization efforts (I believe lightning is >> very fast and worth to be considered) and provide an even better solution >> or two alternative implementations > > > 100% agree. I will check lightning sources and see if I could contribute my > stuff. MapDB serialization is very space-efficiency oriented and it can > contribute a lot. Well, having worked with JDBM's/MapDB's serialization, Kryo and protostuff, I would say that MapDB's serialization is space-efficient, but roughly at the same level as Kryo or a bit worse than latest versions of Kryo. IMHO, the biggest advantage of MapDB's serialization is its speed. It usually wins against highly optimized versions of Kryo and protostuff, even though they use Unsafe tricks and the like. To some extent this speed improvement can be probably attributed to the simplicity of MapDB's serialization implementation. It is not very feature rich, but very small and simple (just a few classes) and call stacks during serialization are usually also very short. Probably JIT is able to optimize and inline much better than in other more complex and universal frameworks. > My only condition is that lighting is distributed in separate JAR. I like > minimal dependencies. > > >> In both cases we would be open to contribution in different forms - just >> contributing patches or with you to join us and the ASF as module or >> subproject (the latter options have to undergo a formal vote by all >> project >> members, of course) as I strongly believe that merging efforts would bring >> to a better and more complete product. > > I would prefer MapDB to stay on GitHub. I find it more comfortable to use. > JDBM3 (previous version) nearly become ApacheDS subproject, but on last > moment I decided otherwise. I strongly agree with Jan here. JDBM/MapDB is used by most people as a DB or persistent map. Its serialization functionality is nice to have, but not the most important feature of it. At the same time, for DM such things like off-heap mgmt and serialization are the most important ones, but persistency is optional. Therefore, IMHO both project should remain independent and cooperate or make use of each other. But they should not be integrated into one "megaproject", which can do everything. -Roman
