2012/9/29 Noctarius <[email protected]>: > Thanks Olivier for carify, I'll take a look in it tomorrow but > there's just one question left (for now ;)): > What is that vote for becoming a committer? What if the vote will > be negative? The vote is on private list (pmc list for privacy reasons and possible negative stuff being on public lists) > Until now I just used Apache stuff, was never interested in being > part of it so I guess it can be negative for any reason, can't it? I don't see why it could be negative but suspens .... :-) > > Am 29.09.2012 18:56, schrieb Olivier Lamy: >> 2012/9/29 Noctarius <[email protected]>: >>> Nope my real name is Christoph Engelbert, but Noctarius is >>> the all time nick :) >>> >>> Renaming the package should be no problem, is it >>> "org.apache.directmemory.lightning" or what would it be? >> fine for me >>> Then there needs to be a change in the license header as >>> Olivier mentioned, that means just remove the first sentence >>> or is there anything more to do (maybe it's easiest thing to >>> just copy the header from DM file ;))? >> yup use same header as DM >>> >>> The CLA is just a form to clarify that the source can be >>> contributed to the Apache Foundation? >> yup correct. >>> >>> The final step will be attaching the patch in form of a huge >>> diff file? >> yes >>> >>> And what is the way to apply for a membership? Never thought >>> about how to do that. >> Read here http://apache.org/foundation/how-it-works.html and >> here http://apache.org/foundation/getinvolved.html . And feel >> free to ask any questions :-) >>> >>> Chris >>> >>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi: >>>> OK, deal, at least for me ;-) I propose you rename the >>>> packages, produce a patch for this and the new serializer >>>> module (should be simple enough starting from an existing >>>> one) and, in the meanwhile, apply for ASF membership. Is >>>> IP clearance needed? I guess yes. After this we will come >>>> up with a formal vote regarding Noctarius (is this your >>>> real name?!) allowance in the project team. >>>> >>>> Good times are gonna come :-) >>>> >>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy" >>>> <[email protected]> ha scritto: >>>> >>>>> 2012/9/29 Raffaele P. Guidi >>>>> <[email protected]>: >>>>>> Well we already have a NIO ready interface allowing >>>>>> direct access to DMs managed bytebuffers but I think >>>>>> this is just half way to what could be achieved >>>>>> optimally blending serialization and memory allocation >>>>>> together. >>>>>> >>>>>> Lightning as a module is of course a good idea and it >>>>>> could easily evolve as a subproject (for the more >>>>>> experienced asf members: is it a feasible way?). >>>>> Nothing prevent to have >>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk >>>>> >>>>> > with a package like: o.a.d.lightning That will be a >>>>> subproject. >>>>>> >>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius" >>>>>> <[email protected]> ha scritto: >>>>>> >>>>>>> Hey guys, >>>>>>> >>>>>>> ok there's no lightning binary available since >>>>>>> lightning wasn't ready yet for releasing. >>>>>>> >>>>>>> For being the only developer it would be no problem >>>>>>> to contribute the sourcebase for DirectMemory but I'm >>>>>>> not sure yet if it wouldn't be better to seperate it >>>>>>> to be available without using DirectMemory itself. I >>>>>>> started it as a serializer for cluster >>>>>>> synchronization, but it would be cool to contribute >>>>>>> lightning as a subproject to DirectMemory :-) >>>>>>> >>>>>>> About the second project I would love to see a >>>>>>> public available buffer API directly in DirectMemory >>>>>>> so that project would be nearly needless :-) The only >>>>>>> difference I think is the allocation strategy my >>>>>>> implementation is using against the one DirectMemory >>>>>>> has, but I'm pretty sure the allocation is extensible >>>>>>> ;-) >>>>>>> >>>>>>> Like I said, for both projects I'm the only dev so >>>>>>> there would be no IP problem. So if it's ok to you to >>>>>>> not include lightning directly in DM I would be glad >>>>>>> to contribute to the Apache Foundation :-) >>>>>>> >>>>>>> Cheers Chris >>>>>>> >>>>>>> >>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi: >>>>>>>> Ok so it's up to noctarius - your move! ;-) >>>>>>>> Regarding the new unsafe storage: it's an opt-in >>>>>>>> feature that can be set with the fluent API >>>>> (and >>>>>>>> soon through the conference file). >>>>>>>> >>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier >>>>>>>> Lamy" <[email protected]> ha >>>>>>> scritto: >>>>>>>> >>>>>>>>> 2012/9/29 Raffaele P. Guidi >>>>>>>>> <[email protected]>: >>>>>>>>>>>> At least for the moment he can provide a >>>>>>>>>>>> patch to be integrated >>>>> in DM >>>>>>>>> :-) >>>>>>>>>> >>>>>>>>>> Sure, but as lightning is not in any public >>>>>>>>>> mvn repo should its >>>>> code be >>>>>>>>>> re-published in our svn? Or what? >>>>>>>>> @Apache we don't care about binaries, only >>>>>>>>> sources are important ! (a bit theorical for sure >>>>>>>>> but that's it :-) ). So if Noctarius was the only >>>>>>>>> guy who participate in lightning. He can just >>>>>>>>> provide a patch we could integrate as a new dm >>>>>>>>> module (note: the patch must not contains any >>>>>>>>> more copyright and all sources must have ASF >>>>>>>>> licenses). >>>>>>>>> >>>>>>>>> "Copyright 2012 the original author or authors." >>>>>>>>> must be removed. And BTW package must be changed >>>>>>>>> :-) (com.github is not acceptable >>>>> @asf >>>>>>>>> :-) )(@Noctarius are you working for github ? :-) >>>>>>>>> ) >>>>>>>>> >>>>>>>>> And having him as a committer will be only a >>>>>>>>> matter of voting (we >>>>> have >>>>>>>>> a great chair who take cares of administrative >>>>>>>>> stuff :P ) >>>>>>>>> >>>>>>>>> If some others have participated in the project, >>>>>>>>> we must pass tru an ip clearance mechanism >>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html) >>>>>>>>> >>>>>>>>> > and all contributors to lightning must provide a cla. >>>>>>>>> (It it's the case I can help) >>>>>>>>> >>>>>>>>>> >>>>>>>>>>>> perso I'd like we avoid hard dependency on >>>>>>>>>>>> Unsafe as maybe some >>>>> use >>>>>>>>>> other jdks :-) >>>>>>>>>> >>>>>>>>>> Well, I believe Unsafe is supported by Sun >>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I >>>>>>>>>> believe that it is more than enough. Also keep >>>>>>>>>> in >>>>> mind >>>>>>>>> that >>>>>>>>>> we already have an alternative Unsafe based >>>>>>>>>> memory storage - and >>>>>>> although >>>>>>>>>> it's not thoroughly tested for performance it >>>>>>>>>> dramaticaly simplifies >>>>>>>>> code, >>>>>>>>>> I have great expectations about it. >>>>>>>>> Me too :-). Yup definitely more simple and faster >>>>>>>>> ! But we must provide a switch off configuration >>>>>>>>> mechanism if some >>>>> users >>>>>>>>> don't want to use that (because Unsafe is just >>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look yet >>>>>>>>> at your changes with using Unsafe. >>>>>>>>>> >>>>>>>>>> Cheers, R >>>>>>>>>> >>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy >>>>>>>>>> <[email protected]> >>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> 2012/9/29 Raffaele P. Guidi >>>>>>>>>>> <[email protected]>: >>>>>>>>>>>> What do you think about: 1) implementing a >>>>>>>>>>>> lightning serialization module 2) creating >>>>>>>>>>>> a serializer that directly works on a >>>>>>>>>>>> directmemory >>>>>>>>> provider >>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based >>>>>>>>>>>> Pointer? >>>>>>>>>>> >>>>>>>>>>> Sounds good (perso I'd like we avoid hard >>>>>>>>>>> dependency on Unsafe as maybe some use other >>>>>>>>>>> jdks :-) ) >>>>>>>>>>>> >>>>>>>>>>>> Now I see lightning is apache licensed and >>>>>>>>>>>> this is fine but I >>>>> don't >>>>>>>>> think >>>>>>>>>>>> it is published in any public maven repo, >>>>>>>>>>>> am I right? We could >>>>> find a >>>>>>>>> way >>>>>>>>>>>> to deal with this; options vary from >>>>>>>>>>>> publishing lightning to the >>>>> free >>>>>>>>>>>> sonatype repo, joining the ASF (which is >>>>>>>>>>>> great anyhow!) and >>>>>>>>> republishing >>>>>>>>>>>> lightning code in DirectMemory becoming a >>>>>>>>>>>> commiter (which has to >>>>>>>>> undergo >>>>>>>>>>> a >>>>>>>>>>>> PMC vote). >>>>>>>>>>> >>>>>>>>>>> At least for the moment he can provide a >>>>>>>>>>> patch to be integrated in >>>>> DM >>>>>>>>> :-) >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I'd like to hear your and the team feelings >>>>>>>>>>>> on this. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, Raffaele >>>>>>>>>>>> >>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius >>>>>>>>>>>> <[email protected]> >>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hey Raffaele, >>>>>>>>>>>>> >>>>>>>>>>>>> that's quite similar to what I did at >>>>>>>>>>>>> work. We're developing >>>>> Flash >>>>>>>>>>>>> online games and using a customized AMF >>>>>>>>>>>>> serialization. To support async writing >>>>>>>>>>>>> of the clients event results I added >>>>>>>>>>>>> serialization >>>>> of >>>>>>>>>>>>> the components / entities to the players >>>>>>>>>>>>> zone calculation and >>>>> stored >>>>>>>>>>>>> the pre-serialized bytestream directly to >>>>>>>>>>>>> the off-heap (using direct-ring-cache >>>>>>>>>>>>> implementation). When the client >>>>>>>>>>>>> requests the results (using long-polling) >>>>>>>>>>>>> I just write the pre-serialized >>>>> data to >>>>>>>>>>>>> the right position to deserialize it by >>>>>>>>>>>>> standard ways on Flash >>>>> side. >>>>>>>>>>>>> >>>>>>>>>>>>> So yeah an seriliaztion to off-heap >>>>>>>>>>>>> algorithm would be fine. You >>>>> can >>>>>>>>>>>>> avoid using byte arrays and minimalize >>>>>>>>>>>>> GC. >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers Chris >>>>>>>>>>>>> >>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. >>>>>>>>>>>>> Guidi: >>>>>>>>>>>>>> Nice to hear back from you! Yes, I was >>>>>>>>>>>>>> thinking about creating a >>>>>>>>> new >>>>>>>>>>>>> memory >>>>>>>>>>>>>> storage implementation using Unsafe >>>>>>>>>>>>>> (and I did it, recently) and >>>>>>>>>>> also, as >>>>>>>>>>>>>> DirectMemory relies heavily on >>>>>>>>>>>>>> serialization (and supports many >>>>> of >>>>>>>>>>> them, >>>>>>>>>>>>>> protostuff, protobuf, msgpack and of >>>>>>>>>>>>>> course standard >>>>>>>>> serialization), >>>>>>>>>>>>> about >>>>>>>>>>>>>> creating a simple embedded serializer >>>>>>>>>>>>>> leveraging the same >>>>>>>>> techniques >>>>>>>>>>> you >>>>>>>>>>>>>> used (Unsafe and bytecode generation). >>>>>>>>>>>>>> The idea with embedding is avoiding to >>>>>>>>>>>>>> serialize in a byte array >>>>>>>>> and >>>>>>>>>>> then >>>>>>>>>>>>>> moving the byte array to off-heap >>>>>>>>>>>>>> memory (via Unsafe or >>>>>>>>> ByteBuffers), >>>>>>>>>>> and >>>>>>>>>>>>>> serializing directly into a "managed" >>>>>>>>>>>>>> off-heap buffer, thus >>>>> further >>>>>>>>>>>>>> optimizing heap utilization (less GC). >>>>>>>>>>>>>> >>>>>>>>>>>>>> What do you think about it? Does it >>>>>>>>>>>>>> make any sense to you? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ciao, R >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, >>>>>>>>>>>>>> Noctarius <[email protected]> >>>>>>>>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hey guys, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Raffaele found out about a project >>>>>>>>>>>>>>> of mine (Lightning) a few >>>>> weeks >>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe and >>>>>>>>>>>>>>> Bytecode generation using Serializer >>>>>>>>>>>>>>> implementation. He told me that he >>>>>>>>>>>>>>> was interested in adding something >>>>>>>>>>>>>>> similar to DirectMemory and I would >>>>>>>>>>>>>>> be glad to >>>>>>>>> help >>>>>>>>>>>>>>> out in this. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Another project I started a few days >>>>>>>>>>>>>>> ago, since it was needed >>>>> for >>>>>>>>>>>>>>> work is DirectRingCache. The name >>>>>>>>>>>>>>> not really meets to actual >>>>>>>>>>>>>>> implementation since it's not yet a >>>>>>>>>>>>>>> ring buffer using cache. I >>>>>>>>> used >>>>>>>>>>>>>>> this for a pre-serialization simple >>>>>>>>>>>>>>> bytestream cache with self-growing >>>>>>>>>>>>>>> buffers. It could be nice to have >>>>>>>>>>>>>>> DirectMemory >>>>> having >>>>>>>>>>>>>>> raw "buffers" to write to or to read >>>>>>>>>>>>>>> from. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Here are the links from the >>>>>>>>>>>>>>> projects: >>>>>>>>>>>>>>> https://github.com/noctarius/Lightning >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> > https://github.com/noctarius/direct-ring-cache >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>> It would be nice to help out since I really like the idea of >>>>>>>>>>>>>>> DirectMemory and since >>>>>>>>>>>>>>> direct-ring-cache is some kind of >>>>>>>>> reinventing >>>>>>>>>>>>>>> the wheel. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers Noctarius (Chris) >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- Olivier Lamy Talend: >>>>>>>>>>> http://coders.talend.com >>>>>>>>>>> http://twitter.com/olamy | >>>>>>>>>>> http://linkedin.com/in/olamy >>>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- Olivier Lamy Talend: http://coders.talend.com >>>>>>>>> http://twitter.com/olamy | >>>>>>>>> http://linkedin.com/in/olamy >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>>> >>>>> >>>>> -- Olivier Lamy Talend: http://coders.talend.com >>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy >>>>> >>>> >> >> >>
-- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy
