Thanks Olivier for carify, I'll take a look in it tomorrow but there's just one question left (for now ;)): What is that vote for becoming a committer? What if the vote will be negative? Until now I just used Apache stuff, was never interested in being part of it so I guess it can be negative for any reason, can't it?
Am 29.09.2012 18:56, schrieb Olivier Lamy: > 2012/9/29 Noctarius <[email protected]>: >> Nope my real name is Christoph Engelbert, but Noctarius is >> the all time nick :) >> >> Renaming the package should be no problem, is it >> "org.apache.directmemory.lightning" or what would it be? > fine for me >> Then there needs to be a change in the license header as >> Olivier mentioned, that means just remove the first sentence >> or is there anything more to do (maybe it's easiest thing to >> just copy the header from DM file ;))? > yup use same header as DM >> >> The CLA is just a form to clarify that the source can be >> contributed to the Apache Foundation? > yup correct. >> >> The final step will be attaching the patch in form of a huge >> diff file? > yes >> >> And what is the way to apply for a membership? Never thought >> about how to do that. > Read here http://apache.org/foundation/how-it-works.html and > here http://apache.org/foundation/getinvolved.html . And feel > free to ask any questions :-) >> >> Chris >> >> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi: >>> OK, deal, at least for me ;-) I propose you rename the >>> packages, produce a patch for this and the new serializer >>> module (should be simple enough starting from an existing >>> one) and, in the meanwhile, apply for ASF membership. Is >>> IP clearance needed? I guess yes. After this we will come >>> up with a formal vote regarding Noctarius (is this your >>> real name?!) allowance in the project team. >>> >>> Good times are gonna come :-) >>> >>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy" >>> <[email protected]> ha scritto: >>> >>>> 2012/9/29 Raffaele P. Guidi >>>> <[email protected]>: >>>>> Well we already have a NIO ready interface allowing >>>>> direct access to DMs managed bytebuffers but I think >>>>> this is just half way to what could be achieved >>>>> optimally blending serialization and memory allocation >>>>> together. >>>>> >>>>> Lightning as a module is of course a good idea and it >>>>> could easily evolve as a subproject (for the more >>>>> experienced asf members: is it a feasible way?). >>>> Nothing prevent to have >>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk >>>> >>>> with a package like: o.a.d.lightning That will be a >>>> subproject. >>>>> >>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius" >>>>> <[email protected]> ha scritto: >>>>> >>>>>> Hey guys, >>>>>> >>>>>> ok there's no lightning binary available since >>>>>> lightning wasn't ready yet for releasing. >>>>>> >>>>>> For being the only developer it would be no problem >>>>>> to contribute the sourcebase for DirectMemory but I'm >>>>>> not sure yet if it wouldn't be better to seperate it >>>>>> to be available without using DirectMemory itself. I >>>>>> started it as a serializer for cluster >>>>>> synchronization, but it would be cool to contribute >>>>>> lightning as a subproject to DirectMemory :-) >>>>>> >>>>>> About the second project I would love to see a >>>>>> public available buffer API directly in DirectMemory >>>>>> so that project would be nearly needless :-) The only >>>>>> difference I think is the allocation strategy my >>>>>> implementation is using against the one DirectMemory >>>>>> has, but I'm pretty sure the allocation is extensible >>>>>> ;-) >>>>>> >>>>>> Like I said, for both projects I'm the only dev so >>>>>> there would be no IP problem. So if it's ok to you to >>>>>> not include lightning directly in DM I would be glad >>>>>> to contribute to the Apache Foundation :-) >>>>>> >>>>>> Cheers Chris >>>>>> >>>>>> >>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi: >>>>>>> Ok so it's up to noctarius - your move! ;-) >>>>>>> Regarding the new unsafe storage: it's an opt-in >>>>>>> feature that can be set with the fluent API >>>> (and >>>>>>> soon through the conference file). >>>>>>> >>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier >>>>>>> Lamy" <[email protected]> ha >>>>>> scritto: >>>>>>> >>>>>>>> 2012/9/29 Raffaele P. Guidi >>>>>>>> <[email protected]>: >>>>>>>>>>> At least for the moment he can provide a >>>>>>>>>>> patch to be integrated >>>> in DM >>>>>>>> :-) >>>>>>>>> >>>>>>>>> Sure, but as lightning is not in any public >>>>>>>>> mvn repo should its >>>> code be >>>>>>>>> re-published in our svn? Or what? >>>>>>>> @Apache we don't care about binaries, only >>>>>>>> sources are important ! (a bit theorical for sure >>>>>>>> but that's it :-) ). So if Noctarius was the only >>>>>>>> guy who participate in lightning. He can just >>>>>>>> provide a patch we could integrate as a new dm >>>>>>>> module (note: the patch must not contains any >>>>>>>> more copyright and all sources must have ASF >>>>>>>> licenses). >>>>>>>> >>>>>>>> "Copyright 2012 the original author or authors." >>>>>>>> must be removed. And BTW package must be changed >>>>>>>> :-) (com.github is not acceptable >>>> @asf >>>>>>>> :-) )(@Noctarius are you working for github ? :-) >>>>>>>> ) >>>>>>>> >>>>>>>> And having him as a committer will be only a >>>>>>>> matter of voting (we >>>> have >>>>>>>> a great chair who take cares of administrative >>>>>>>> stuff :P ) >>>>>>>> >>>>>>>> If some others have participated in the project, >>>>>>>> we must pass tru an ip clearance mechanism >>>>>>>> (http://incubator.apache.org/ip-clearance/index.html) >>>>>>>> >>>>>>>> and all contributors to lightning must provide a cla. >>>>>>>> (It it's the case I can help) >>>>>>>> >>>>>>>>> >>>>>>>>>>> perso I'd like we avoid hard dependency on >>>>>>>>>>> Unsafe as maybe some >>>> use >>>>>>>>> other jdks :-) >>>>>>>>> >>>>>>>>> Well, I believe Unsafe is supported by Sun >>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I >>>>>>>>> believe that it is more than enough. Also keep >>>>>>>>> in >>>> mind >>>>>>>> that >>>>>>>>> we already have an alternative Unsafe based >>>>>>>>> memory storage - and >>>>>> although >>>>>>>>> it's not thoroughly tested for performance it >>>>>>>>> dramaticaly simplifies >>>>>>>> code, >>>>>>>>> I have great expectations about it. >>>>>>>> Me too :-). Yup definitely more simple and faster >>>>>>>> ! But we must provide a switch off configuration >>>>>>>> mechanism if some >>>> users >>>>>>>> don't want to use that (because Unsafe is just >>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet >>>>>>>> at your changes with using Unsafe. >>>>>>>>> >>>>>>>>> Cheers, R >>>>>>>>> >>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy >>>>>>>>> <[email protected]> >>>>>> wrote: >>>>>>>>> >>>>>>>>>> 2012/9/29 Raffaele P. Guidi >>>>>>>>>> <[email protected]>: >>>>>>>>>>> What do you think about: 1) implementing a >>>>>>>>>>> lightning serialization module 2) creating >>>>>>>>>>> a serializer that directly works on a >>>>>>>>>>> directmemory >>>>>>>> provider >>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based >>>>>>>>>>> Pointer? >>>>>>>>>> >>>>>>>>>> Sounds good (perso I'd like we avoid hard >>>>>>>>>> dependency on Unsafe as maybe some use other >>>>>>>>>> jdks :-) ) >>>>>>>>>>> >>>>>>>>>>> Now I see lightning is apache licensed and >>>>>>>>>>> this is fine but I >>>> don't >>>>>>>> think >>>>>>>>>>> it is published in any public maven repo, >>>>>>>>>>> am I right? We could >>>> find a >>>>>>>> way >>>>>>>>>>> to deal with this; options vary from >>>>>>>>>>> publishing lightning to the >>>> free >>>>>>>>>>> sonatype repo, joining the ASF (which is >>>>>>>>>>> great anyhow!) and >>>>>>>> republishing >>>>>>>>>>> lightning code in DirectMemory becoming a >>>>>>>>>>> commiter (which has to >>>>>>>> undergo >>>>>>>>>> a >>>>>>>>>>> PMC vote). >>>>>>>>>> >>>>>>>>>> At least for the moment he can provide a >>>>>>>>>> patch to be integrated in >>>> DM >>>>>>>> :-) >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> I'd like to hear your and the team feelings >>>>>>>>>>> on this. >>>>>>>>>>> >>>>>>>>>>> Thanks, Raffaele >>>>>>>>>>> >>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius >>>>>>>>>>> <[email protected]> >>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hey Raffaele, >>>>>>>>>>>> >>>>>>>>>>>> that's quite similar to what I did at >>>>>>>>>>>> work. We're developing >>>> Flash >>>>>>>>>>>> online games and using a customized AMF >>>>>>>>>>>> serialization. To support async writing >>>>>>>>>>>> of the clients event results I added >>>>>>>>>>>> serialization >>>> of >>>>>>>>>>>> the components / entities to the players >>>>>>>>>>>> zone calculation and >>>> stored >>>>>>>>>>>> the pre-serialized bytestream directly to >>>>>>>>>>>> the off-heap (using direct-ring-cache >>>>>>>>>>>> implementation). When the client >>>>>>>>>>>> requests the results (using long-polling) >>>>>>>>>>>> I just write the pre-serialized >>>> data to >>>>>>>>>>>> the right position to deserialize it by >>>>>>>>>>>> standard ways on Flash >>>> side. >>>>>>>>>>>> >>>>>>>>>>>> So yeah an seriliaztion to off-heap >>>>>>>>>>>> algorithm would be fine. You >>>> can >>>>>>>>>>>> avoid using byte arrays and minimalize >>>>>>>>>>>> GC. >>>>>>>>>>>> >>>>>>>>>>>> Cheers Chris >>>>>>>>>>>> >>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. >>>>>>>>>>>> Guidi: >>>>>>>>>>>>> Nice to hear back from you! Yes, I was >>>>>>>>>>>>> thinking about creating a >>>>>>>> new >>>>>>>>>>>> memory >>>>>>>>>>>>> storage implementation using Unsafe >>>>>>>>>>>>> (and I did it, recently) and >>>>>>>>>> also, as >>>>>>>>>>>>> DirectMemory relies heavily on >>>>>>>>>>>>> serialization (and supports many >>>> of >>>>>>>>>> them, >>>>>>>>>>>>> protostuff, protobuf, msgpack and of >>>>>>>>>>>>> course standard >>>>>>>> serialization), >>>>>>>>>>>> about >>>>>>>>>>>>> creating a simple embedded serializer >>>>>>>>>>>>> leveraging the same >>>>>>>> techniques >>>>>>>>>> you >>>>>>>>>>>>> used (Unsafe and bytecode generation). >>>>>>>>>>>>> The idea with embedding is avoiding to >>>>>>>>>>>>> serialize in a byte array >>>>>>>> and >>>>>>>>>> then >>>>>>>>>>>>> moving the byte array to off-heap >>>>>>>>>>>>> memory (via Unsafe or >>>>>>>> ByteBuffers), >>>>>>>>>> and >>>>>>>>>>>>> serializing directly into a "managed" >>>>>>>>>>>>> off-heap buffer, thus >>>> further >>>>>>>>>>>>> optimizing heap utilization (less GC). >>>>>>>>>>>>> >>>>>>>>>>>>> What do you think about it? Does it >>>>>>>>>>>>> make any sense to you? >>>>>>>>>>>>> >>>>>>>>>>>>> Ciao, R >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, >>>>>>>>>>>>> Noctarius <[email protected]> >>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hey guys, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Raffaele found out about a project >>>>>>>>>>>>>> of mine (Lightning) a few >>>> weeks >>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and >>>>>>>>>>>>>> Bytecode generation using Serializer >>>>>>>>>>>>>> implementation. He told me that he >>>>>>>>>>>>>> was interested in adding something >>>>>>>>>>>>>> similar to DirectMemory and I would >>>>>>>>>>>>>> be glad to >>>>>>>> help >>>>>>>>>>>>>> out in this. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Another project I started a few days >>>>>>>>>>>>>> ago, since it was needed >>>> for >>>>>>>>>>>>>> work is DirectRingCache. The name >>>>>>>>>>>>>> not really meets to actual >>>>>>>>>>>>>> implementation since it's not yet a >>>>>>>>>>>>>> ring buffer using cache. I >>>>>>>> used >>>>>>>>>>>>>> this for a pre-serialization simple >>>>>>>>>>>>>> bytestream cache with self-growing >>>>>>>>>>>>>> buffers. It could be nice to have >>>>>>>>>>>>>> DirectMemory >>>> having >>>>>>>>>>>>>> raw "buffers" to write to or to read >>>>>>>>>>>>>> from. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here are the links from the >>>>>>>>>>>>>> projects: >>>>>>>>>>>>>> https://github.com/noctarius/Lightning >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://github.com/noctarius/direct-ring-cache >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >> It would be nice to help out since I really like the idea of >>>>>>>>>>>>>> DirectMemory and since >>>>>>>>>>>>>> direct-ring-cache is some kind of >>>>>>>> reinventing >>>>>>>>>>>>>> the wheel. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers Noctarius (Chris) >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- Olivier Lamy Talend: >>>>>>>>>> http://coders.talend.com >>>>>>>>>> http://twitter.com/olamy | >>>>>>>>>> http://linkedin.com/in/olamy >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com >>>>>>>> http://twitter.com/olamy | >>>>>>>> http://linkedin.com/in/olamy >>>>>>>> >>>>>>> >>>>>> >>>> >>>> >>>> >>>> -- Olivier Lamy Talend: http://coders.talend.com >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy >>>> >>> > > >
