Am 07.11.2012 21:37, schrieb Raffaele P. Guidi: > Well I don't quite get what you mean for serialization frameworks... kryo, > protostuff and lightning serialize pojos without any configuration and even > without the need to implement an interface such as Serializable. They just > work and are fast and general purpose.
Most purposes :-) Lightning cannot serialize everything as Java Serialization but 99% of all the standard POJOs you'll see in the normal Java environment. This is due to the case that Lightning was started as a serializer for highly-concurrent (Lightning is completely threadsafe) and high throughput clustered environments where only transfer object are used. > Ciao, > R > Il giorno 07/nov/2012 21:30, "Jan Kotek" <[email protected]> ha scritto: > >> Hi Roman, >> >> your patch saved me lot of headaches and was very welcomed. >> >> I am not using Quickser becouse serialization in MapDB is still evolving >> rapidly. For example I made some refactoring to make serialization less >> dependant on other JDBM classes. I also have plan to use some stuff from >> Kryo and Lighting (unsafe ops, bytecode generators). And Quickser did not >> had much updates since it forked. >> >> Obviously it is more comfortable for me if serialization framework stays >> inside MapDB. It is critical part since in database we care about long term >> persistence. But on other side I would love if somebody would took over >> this part. I have no problem with extracting serialization to separate >> project, but I need to see that this fork is active and can evolve on its >> own. >> >> I hoped I could use Lightning, Kryo or other framework developed as part >> of DirectMemory. But there seems to be conception difference. Kryo and >> Lightning seems to be more like 'serialization framework'; it has bunch of >> serializers (for numbers, dates...) and you should choose one which suits >> you best. >> >> But MapDB should 'just work' without additional configuration. So I need >> universal serialization; it should turn any object into bytes (similar to >> Java Serialization or XStream). Also I want it to mimic standard Java >> Serialization (Serializable marker interface, Externalizable, writeExternal >> methods... etc). >> >> So for now I will investigate if I can patch Lighting to support my needs. >> If not I will take parts I like and integrate it into MapDB. >> >> Jan >> >> On 07/11/12 09:30, Roman Levenstein wrote: >> >>> Hi, >>> >>> I'm one of the contributors to JDMB3 serialization implementation. >>> Actually earlier this year I made it much faster than before (2 orders >>> of magnitude). And BTW, I'm also a contributor to Kryo and >>> protostuff-runtime. >>> >>> I find this discussion very interesting, so let me provide my two cents >>> as well. >>> >>> First of all, I just want to mention that while working on improving >>> JDBM's serialization, I extracted the serialization part of the JDBM >>> into a dedicated serialization library, which I called Quickser. You >>> can find it on GitHub: >>> https://github.com/romix/**quickser<https://github.com/romix/quickser> >>> It is really very fast, often faster than Kryo and protostuff. Since >>> Quickser contains only serialization-related stuff from JDBM/MapDB, it >>> is easier to use it if you just want to add yet another serialization >>> method to DM without any DB related functionality. >>> >>> It could even make sense, if MapDB would use Quickser for >>> serialization instead of having both DB and serialization related >>> functionality in one pot. >>> >>> @Jan: What do you think about it? I understand that you don't like >>> external dependencies. But Quickser is not really external. It is more >>> or less a copy of JDMBs serialization-related classes. >>> >>> On Wed, Nov 7, 2012 at 9:49 AM, Jan Kotek <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> >>>> 1. DirectMemory could make good use of mapdb to serialize least >>>>> frequently used items to disk and free memory >>>>> 2. DirectMemory could implement a MapDB disk based store in >>>>> addition >>>>> to >>>>> the bytebuffer and unsafe ones >>>>> >>>> The only problem may be that MapDB currently does not support concurrent >>>> transactions (it has only one single global transaction). >>>> Not sure if it could be a problem. >>>> >>>> However it implements ConcurrentMap, so it is possible to swap items >>>> atomically >>>> >>>> >>>> 3. MapDB could take advantage of DM's componentization approach to >>>>> support multiple serializers (we believe each one has its >>>>> advantages >>>>> in >>>>> different scenarios) >>>>> >>>> MapDB already supports alternative serializers. User can supply their >>>> own on >>>> Map (similar to table) creation. >>>> I would love to integrate stuff from lightning serializer. >>>> >>>> >>>> 4. MapDB could use DM to write items to an off-heap before writing >>>>> to >>>>> disk (asynchronously) to improve speed >>>>> >>>> Not sure it would be practical. MapDB already uses memory mapped files so >>>> effect would be very similar. My tests shows that there is only 50% >>>> performance difference between inMemory store and onDisk store. >>>> >>>> Currently MapDB has only heap based inMemory store. But implementing off >>>> heap memory store is trivial and I will do it soon. >>>> >>> This is very nice to know. Looking forward to see this feature. May be >>> you should use DM for it? >>> >>> 5. We could merge our serialization efforts (I believe lightning is >>>>> very fast and worth to be considered) and provide an even better >>>>> solution >>>>> or two alternative implementations >>>>> >>>> 100% agree. I will check lightning sources and see if I could contribute >>>> my >>>> stuff. MapDB serialization is very space-efficiency oriented and it can >>>> contribute a lot. >>>> >>> Well, having worked with JDBM's/MapDB's serialization, Kryo and >>> protostuff, I would say that MapDB's serialization is space-efficient, >>> but roughly at the same level as Kryo or a bit worse than latest >>> versions of Kryo. >>> >>> IMHO, the biggest advantage of MapDB's serialization is its speed. It >>> usually wins against highly optimized versions of Kryo and protostuff, >>> even though they use Unsafe tricks and the like. To some extent this >>> speed improvement can be probably attributed to the simplicity of >>> MapDB's serialization implementation. It is not very feature rich, but >>> very small and simple (just a few classes) and call stacks during >>> serialization are usually also very short. Probably JIT is able to >>> optimize and inline much better than in other more complex and >>> universal frameworks. >>> >>> My only condition is that lighting is distributed in separate JAR. I like >>>> minimal dependencies. >>>> >>>> >>>> In both cases we would be open to contribution in different forms - just >>>>> contributing patches or with you to join us and the ASF as module or >>>>> subproject (the latter options have to undergo a formal vote by all >>>>> project >>>>> members, of course) as I strongly believe that merging efforts would >>>>> bring >>>>> to a better and more complete product. >>>>> >>>> I would prefer MapDB to stay on GitHub. I find it more comfortable to >>>> use. >>>> JDBM3 (previous version) nearly become ApacheDS subproject, but on last >>>> moment I decided otherwise. >>>> >>> I strongly agree with Jan here. JDBM/MapDB is used by most people as a >>> DB or persistent map. >>> Its serialization functionality is nice to have, but not the most >>> important feature of it. >>> At the same time, for DM such things like off-heap mgmt and >>> serialization are the most important ones, but persistency is >>> optional. >>> Therefore, IMHO both project should remain independent and cooperate >>> or make use of each other. But they should not be integrated into one >>> "megaproject", which can do everything. >>> >>> -Roman >>> >>>
