2012/9/29 Raffaele P. Guidi <[email protected]>: > What do you think about: > 1) implementing a lightning serialization module > 2) creating a serializer that directly works on a directmemory provider > ByteBuffer or (maybe better) Unsafe based Pointer?
Sounds good (perso I'd like we avoid hard dependency on Unsafe as maybe some use other jdks :-) ) > > Now I see lightning is apache licensed and this is fine but I don't think > it is published in any public maven repo, am I right? We could find a way > to deal with this; options vary from publishing lightning to the free > sonatype repo, joining the ASF (which is great anyhow!) and republishing > lightning code in DirectMemory becoming a commiter (which has to undergo a > PMC vote). At least for the moment he can provide a patch to be integrated in DM :-) > > I'd like to hear your and the team feelings on this. > > Thanks, > Raffaele > > On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <[email protected]> wrote: > >> Hey Raffaele, >> >> that's quite similar to what I did at work. We're developing Flash >> online games and using a customized AMF serialization. To support >> async writing of the clients event results I added serialization of >> the components / entities to the players zone calculation and stored >> the pre-serialized bytestream directly to the off-heap (using >> direct-ring-cache implementation). When the client requests the >> results (using long-polling) I just write the pre-serialized data to >> the right position to deserialize it by standard ways on Flash side. >> >> So yeah an seriliaztion to off-heap algorithm would be fine. You can >> avoid using byte arrays and minimalize GC. >> >> Cheers >> Chris >> >> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi: >> > Nice to hear back from you! Yes, I was thinking about creating a new >> memory >> > storage implementation using Unsafe (and I did it, recently) and also, as >> > DirectMemory relies heavily on serialization (and supports many of them, >> > protostuff, protobuf, msgpack and of course standard serialization), >> about >> > creating a simple embedded serializer leveraging the same techniques you >> > used (Unsafe and bytecode generation). >> > The idea with embedding is avoiding to serialize in a byte array and then >> > moving the byte array to off-heap memory (via Unsafe or ByteBuffers), and >> > serializing directly into a "managed" off-heap buffer, thus further >> > optimizing heap utilization (less GC). >> > >> > What do you think about it? Does it make any sense to you? >> > >> > Ciao, >> > R >> > >> > On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <[email protected]> wrote: >> > >> >> Hey guys, >> >> >> >> Raffaele found out about a project of mine (Lightning) a few weeks >> >> ago. Lightning is a heavy Unsafe and Bytecode generation using >> >> Serializer implementation. He told me that he was interested in >> >> adding something similar to DirectMemory and I would be glad to help >> >> out in this. >> >> >> >> Another project I started a few days ago, since it was needed for >> >> work is DirectRingCache. The name not really meets to actual >> >> implementation since it's not yet a ring buffer using cache. I used >> >> this for a pre-serialization simple bytestream cache with >> >> self-growing buffers. It could be nice to have DirectMemory having >> >> raw "buffers" to write to or to read from. >> >> >> >> Here are the links from the projects: >> >> https://github.com/noctarius/Lightning >> >> https://github.com/noctarius/direct-ring-cache >> >> >> >> It would be nice to help out since I really like the idea of >> >> DirectMemory and since direct-ring-cache is some kind of reinventing >> >> the wheel. >> >> >> >> Cheers >> >> Noctarius (Chris) >> >> >> > >> >> >> -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy
