2012/9/29 Raffaele P. Guidi <[email protected]>: > OK, deal, at least for me ;-) I propose you rename the packages, produce a > patch for this and the new serializer module (should be simple enough > starting from an existing one) and, in the meanwhile, apply for ASF > membership. Is IP clearance needed? I guess yes. > After this we will come up with a formal vote regarding Noctarius (is this > your real name?!) allowance in the project team. +1 from me too. IMHO ip clearance is not needed. We will just receive a patch from a contributor (all the code has been done by him AFICS in the git repo). If that's not the case ip clearance is needed.
At least as it's a huge patch we need a cla from him. @Noctarius see http://www.apache.org/licenses/ and in section "Contributor License Agreements" the link to cla from to send to secretary@. Once this is done attach your patch to a jira issue https://issues.apache.org/jira/browse/DIRECTMEMORY > > Good times are gonna come :-) > > Thanks, > R > Il giorno 29/set/2012 17:58, "Olivier Lamy" <[email protected]> ha scritto: > >> 2012/9/29 Raffaele P. Guidi <[email protected]>: >> > Well we already have a NIO ready interface allowing direct access to DMs >> > managed bytebuffers but I think this is just half way to what could be >> > achieved optimally blending serialization and memory allocation together. >> > >> > Lightning as a module is of course a good idea and it could easily evolve >> > as a subproject (for the more experienced asf members: is it a feasible >> > way?). >> Nothing prevent to have >> http://svn.apache.org/repos/asf/directmemory/lightning/trunk >> with a package like: o.a.d.lightning >> That will be a subproject. >> > >> > Ciao, >> > R >> > Il giorno 29/set/2012 17:44, "Noctarius" <[email protected]> ha scritto: >> > >> >> Hey guys, >> >> >> >> ok there's no lightning binary available since lightning wasn't >> >> ready yet for releasing. >> >> >> >> For being the only developer it would be no problem to contribute >> >> the sourcebase for DirectMemory but I'm not sure yet if it wouldn't >> >> be better to seperate it to be available without using DirectMemory >> >> itself. I started it as a serializer for cluster synchronization, >> >> but it would be cool to contribute lightning as a subproject to >> >> DirectMemory :-) >> >> >> >> About the second project I would love to see a public available >> >> buffer API directly in DirectMemory so that project would be nearly >> >> needless :-) The only difference I think is the allocation strategy >> >> my implementation is using against the one DirectMemory has, but I'm >> >> pretty sure the allocation is extensible ;-) >> >> >> >> Like I said, for both projects I'm the only dev so there would be no >> >> IP problem. So if it's ok to you to not include lightning directly >> >> in DM I would be glad to contribute to the Apache Foundation :-) >> >> >> >> Cheers Chris >> >> >> >> >> >> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi: >> >> > Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe >> >> > storage: it's an opt-in feature that can be set with the fluent API >> (and >> >> > soon through the conference file). >> >> > >> >> > Ciao, >> >> > R >> >> > Il giorno 29/set/2012 16:45, "Olivier Lamy" <[email protected]> ha >> >> scritto: >> >> > >> >> >> 2012/9/29 Raffaele P. Guidi <[email protected]>: >> >> >>>>> At least for the moment he can provide a patch to be integrated >> in DM >> >> >> :-) >> >> >>> >> >> >>> Sure, but as lightning is not in any public mvn repo should its >> code be >> >> >>> re-published in our svn? Or what? >> >> >> @Apache we don't care about binaries, only sources are important ! (a >> >> >> bit theorical for sure but that's it :-) ). >> >> >> So if Noctarius was the only guy who participate in lightning. He can >> >> >> just provide a patch we could integrate as a new dm module (note: the >> >> >> patch must not contains any more copyright and all sources must have >> >> >> ASF licenses). >> >> >> >> >> >> "Copyright 2012 the original author or authors." must be removed. >> >> >> And BTW package must be changed :-) (com.github is not acceptable >> @asf >> >> >> :-) )(@Noctarius are you working for github ? :-) ) >> >> >> >> >> >> And having him as a committer will be only a matter of voting (we >> have >> >> >> a great chair who take cares of administrative stuff :P ) >> >> >> >> >> >> If some others have participated in the project, we must pass tru an >> >> >> ip clearance mechanism >> >> >> (http://incubator.apache.org/ip-clearance/index.html) and all >> >> >> contributors to lightning must provide a cla. (It it's the case I can >> >> >> help) >> >> >> >> >> >>> >> >> >>>>> perso I'd like we avoid hard dependency on Unsafe as maybe some >> use >> >> >>> other jdks :-) >> >> >>> >> >> >>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and >> >> >>> JRockit - and I believe that it is more than enough. Also keep in >> mind >> >> >> that >> >> >>> we already have an alternative Unsafe based memory storage - and >> >> although >> >> >>> it's not thoroughly tested for performance it dramaticaly simplifies >> >> >> code, >> >> >>> I have great expectations about it. >> >> >> Me too :-). Yup definitely more simple and faster ! >> >> >> But we must provide a switch off configuration mechanism if some >> users >> >> >> don't want to use that (because Unsafe is just "Unsafe" :-) ) >> >> >> And sorry I didn't have a look yet at your changes with using Unsafe. >> >> >>> >> >> >>> Cheers, >> >> >>> R >> >> >>> >> >> >>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <[email protected]> >> >> wrote: >> >> >>> >> >> >>>> 2012/9/29 Raffaele P. Guidi <[email protected]>: >> >> >>>>> What do you think about: >> >> >>>>> 1) implementing a lightning serialization module >> >> >>>>> 2) creating a serializer that directly works on a directmemory >> >> >> provider >> >> >>>>> ByteBuffer or (maybe better) Unsafe based Pointer? >> >> >>>> >> >> >>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as >> >> >>>> maybe some use other jdks :-) ) >> >> >>>>> >> >> >>>>> Now I see lightning is apache licensed and this is fine but I >> don't >> >> >> think >> >> >>>>> it is published in any public maven repo, am I right? We could >> find a >> >> >> way >> >> >>>>> to deal with this; options vary from publishing lightning to the >> free >> >> >>>>> sonatype repo, joining the ASF (which is great anyhow!) and >> >> >> republishing >> >> >>>>> lightning code in DirectMemory becoming a commiter (which has to >> >> >> undergo >> >> >>>> a >> >> >>>>> PMC vote). >> >> >>>> >> >> >>>> At least for the moment he can provide a patch to be integrated in >> DM >> >> >> :-) >> >> >>>> >> >> >>>>> >> >> >>>>> I'd like to hear your and the team feelings on this. >> >> >>>>> >> >> >>>>> Thanks, >> >> >>>>> Raffaele >> >> >>>>> >> >> >>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <[email protected]> >> wrote: >> >> >>>>> >> >> >>>>>> Hey Raffaele, >> >> >>>>>> >> >> >>>>>> that's quite similar to what I did at work. We're developing >> Flash >> >> >>>>>> online games and using a customized AMF serialization. To support >> >> >>>>>> async writing of the clients event results I added serialization >> of >> >> >>>>>> the components / entities to the players zone calculation and >> stored >> >> >>>>>> the pre-serialized bytestream directly to the off-heap (using >> >> >>>>>> direct-ring-cache implementation). When the client requests the >> >> >>>>>> results (using long-polling) I just write the pre-serialized >> data to >> >> >>>>>> the right position to deserialize it by standard ways on Flash >> side. >> >> >>>>>> >> >> >>>>>> So yeah an seriliaztion to off-heap algorithm would be fine. You >> can >> >> >>>>>> avoid using byte arrays and minimalize GC. >> >> >>>>>> >> >> >>>>>> Cheers >> >> >>>>>> Chris >> >> >>>>>> >> >> >>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi: >> >> >>>>>>> Nice to hear back from you! Yes, I was thinking about creating a >> >> >> new >> >> >>>>>> memory >> >> >>>>>>> storage implementation using Unsafe (and I did it, recently) and >> >> >>>> also, as >> >> >>>>>>> DirectMemory relies heavily on serialization (and supports many >> of >> >> >>>> them, >> >> >>>>>>> protostuff, protobuf, msgpack and of course standard >> >> >> serialization), >> >> >>>>>> about >> >> >>>>>>> creating a simple embedded serializer leveraging the same >> >> >> techniques >> >> >>>> you >> >> >>>>>>> used (Unsafe and bytecode generation). >> >> >>>>>>> The idea with embedding is avoiding to serialize in a byte array >> >> >> and >> >> >>>> then >> >> >>>>>>> moving the byte array to off-heap memory (via Unsafe or >> >> >> ByteBuffers), >> >> >>>> and >> >> >>>>>>> serializing directly into a "managed" off-heap buffer, thus >> further >> >> >>>>>>> optimizing heap utilization (less GC). >> >> >>>>>>> >> >> >>>>>>> What do you think about it? Does it make any sense to you? >> >> >>>>>>> >> >> >>>>>>> Ciao, >> >> >>>>>>> R >> >> >>>>>>> >> >> >>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <[email protected]> >> >> >> wrote: >> >> >>>>>>> >> >> >>>>>>>> Hey guys, >> >> >>>>>>>> >> >> >>>>>>>> Raffaele found out about a project of mine (Lightning) a few >> weeks >> >> >>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode generation using >> >> >>>>>>>> Serializer implementation. He told me that he was interested in >> >> >>>>>>>> adding something similar to DirectMemory and I would be glad to >> >> >> help >> >> >>>>>>>> out in this. >> >> >>>>>>>> >> >> >>>>>>>> Another project I started a few days ago, since it was needed >> for >> >> >>>>>>>> work is DirectRingCache. The name not really meets to actual >> >> >>>>>>>> implementation since it's not yet a ring buffer using cache. I >> >> >> used >> >> >>>>>>>> this for a pre-serialization simple bytestream cache with >> >> >>>>>>>> self-growing buffers. It could be nice to have DirectMemory >> having >> >> >>>>>>>> raw "buffers" to write to or to read from. >> >> >>>>>>>> >> >> >>>>>>>> Here are the links from the projects: >> >> >>>>>>>> https://github.com/noctarius/Lightning >> >> >>>>>>>> https://github.com/noctarius/direct-ring-cache >> >> >>>>>>>> >> >> >>>>>>>> It would be nice to help out since I really like the idea of >> >> >>>>>>>> DirectMemory and since direct-ring-cache is some kind of >> >> >> reinventing >> >> >>>>>>>> the wheel. >> >> >>>>>>>> >> >> >>>>>>>> Cheers >> >> >>>>>>>> Noctarius (Chris) >> >> >>>>>>>> >> >> >>>>>>> >> >> >>>>>> >> >> >>>>>> >> >> >>>>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>>> -- >> >> >>>> Olivier Lamy >> >> >>>> Talend: http://coders.talend.com >> >> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy >> >> >>>> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Olivier Lamy >> >> >> Talend: http://coders.talend.com >> >> >> http://twitter.com/olamy | http://linkedin.com/in/olamy >> >> >> >> >> > >> >> >> >> >> >> -- >> Olivier Lamy >> Talend: http://coders.talend.com >> http://twitter.com/olamy | http://linkedin.com/in/olamy >> -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy
