2012/9/29 Noctarius <[email protected]>: > PS: Links below won't work for me "connection cancelled". Is there > a problem with the servers? I noticed too network issues for people in eu :-( > > Am 29.09.2012 19:00, schrieb Noctarius: >> Thanks Olivier for carify, I'll take a look in it tomorrow but >> there's just one question left (for now ;)): What is that vote >> for becoming a committer? What if the vote will be negative? >> Until now I just used Apache stuff, was never interested in >> being part of it so I guess it can be negative for any reason, >> can't it? >> >> Am 29.09.2012 18:56, schrieb Olivier Lamy: >>> 2012/9/29 Noctarius <[email protected]>: >>>> Nope my real name is Christoph Engelbert, but Noctarius is >>>> the all time nick :) >>>> >>>> Renaming the package should be no problem, is it >>>> "org.apache.directmemory.lightning" or what would it be? >>> fine for me >>>> Then there needs to be a change in the license header as >>>> Olivier mentioned, that means just remove the first >>>> sentence or is there anything more to do (maybe it's >>>> easiest thing to just copy the header from DM file ;))? >>> yup use same header as DM >>>> >>>> The CLA is just a form to clarify that the source can be >>>> contributed to the Apache Foundation? >>> yup correct. >>>> >>>> The final step will be attaching the patch in form of a >>>> huge diff file? >>> yes >>>> >>>> And what is the way to apply for a membership? Never >>>> thought about how to do that. >>> Read here http://apache.org/foundation/how-it-works.html and >>> here http://apache.org/foundation/getinvolved.html . And >>> feel free to ask any questions :-) >>>> >>>> Chris >>>> >>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi: >>>>> OK, deal, at least for me ;-) I propose you rename the >>>>> packages, produce a patch for this and the new serializer >>>>> module (should be simple enough starting from an >>>>> existing one) and, in the meanwhile, apply for ASF >>>>> membership. Is IP clearance needed? I guess yes. After >>>>> this we will come up with a formal vote regarding >>>>> Noctarius (is this your real name?!) allowance in the >>>>> project team. >>>>> >>>>> Good times are gonna come :-) >>>>> >>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier Lamy" >>>>> <[email protected]> ha scritto: >>>>> >>>>>> 2012/9/29 Raffaele P. Guidi >>>>>> <[email protected]>: >>>>>>> Well we already have a NIO ready interface allowing >>>>>>> direct access to DMs managed bytebuffers but I think >>>>>>> this is just half way to what could be achieved >>>>>>> optimally blending serialization and memory >>>>>>> allocation together. >>>>>>> >>>>>>> Lightning as a module is of course a good idea and >>>>>>> it could easily evolve as a subproject (for the more >>>>>>> experienced asf members: is it a feasible way?). >>>>>> Nothing prevent to have >>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk >>>>>> >>>>>> >> >>>>>> > with a package like: o.a.d.lightning That will be a >>>>>> subproject. >>>>>>> >>>>>>> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius" >>>>>>> <[email protected]> ha scritto: >>>>>>> >>>>>>>> Hey guys, >>>>>>>> >>>>>>>> ok there's no lightning binary available since >>>>>>>> lightning wasn't ready yet for releasing. >>>>>>>> >>>>>>>> For being the only developer it would be no >>>>>>>> problem to contribute the sourcebase for >>>>>>>> DirectMemory but I'm not sure yet if it wouldn't be >>>>>>>> better to seperate it to be available without using >>>>>>>> DirectMemory itself. I started it as a serializer >>>>>>>> for cluster synchronization, but it would be cool >>>>>>>> to contribute lightning as a subproject to >>>>>>>> DirectMemory :-) >>>>>>>> >>>>>>>> About the second project I would love to see a >>>>>>>> public available buffer API directly in >>>>>>>> DirectMemory so that project would be nearly >>>>>>>> needless :-) The only difference I think is the >>>>>>>> allocation strategy my implementation is using >>>>>>>> against the one DirectMemory has, but I'm pretty >>>>>>>> sure the allocation is extensible ;-) >>>>>>>> >>>>>>>> Like I said, for both projects I'm the only dev so >>>>>>>> there would be no IP problem. So if it's ok to you >>>>>>>> to not include lightning directly in DM I would be >>>>>>>> glad to contribute to the Apache Foundation :-) >>>>>>>> >>>>>>>> Cheers Chris >>>>>>>> >>>>>>>> >>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi: >>>>>>>>> Ok so it's up to noctarius - your move! ;-) >>>>>>>>> Regarding the new unsafe storage: it's an opt-in >>>>>>>>> feature that can be set with the fluent API >>>>>> (and >>>>>>>>> soon through the conference file). >>>>>>>>> >>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier >>>>>>>>> Lamy" <[email protected]> ha >>>>>>>> scritto: >>>>>>>>> >>>>>>>>>> 2012/9/29 Raffaele P. Guidi >>>>>>>>>> <[email protected]>: >>>>>>>>>>>>> At least for the moment he can provide a >>>>>>>>>>>>> patch to be integrated >>>>>> in DM >>>>>>>>>> :-) >>>>>>>>>>> >>>>>>>>>>> Sure, but as lightning is not in any public >>>>>>>>>>> mvn repo should its >>>>>> code be >>>>>>>>>>> re-published in our svn? Or what? >>>>>>>>>> @Apache we don't care about binaries, only >>>>>>>>>> sources are important ! (a bit theorical for >>>>>>>>>> sure but that's it :-) ). So if Noctarius was >>>>>>>>>> the only guy who participate in lightning. He >>>>>>>>>> can just provide a patch we could integrate as >>>>>>>>>> a new dm module (note: the patch must not >>>>>>>>>> contains any more copyright and all sources >>>>>>>>>> must have ASF licenses). >>>>>>>>>> >>>>>>>>>> "Copyright 2012 the original author or >>>>>>>>>> authors." must be removed. And BTW package must >>>>>>>>>> be changed :-) (com.github is not acceptable >>>>>> @asf >>>>>>>>>> :-) )(@Noctarius are you working for github ? >>>>>>>>>> :-) ) >>>>>>>>>> >>>>>>>>>> And having him as a committer will be only a >>>>>>>>>> matter of voting (we >>>>>> have >>>>>>>>>> a great chair who take cares of administrative >>>>>>>>>> stuff :P ) >>>>>>>>>> >>>>>>>>>> If some others have participated in the >>>>>>>>>> project, we must pass tru an ip clearance >>>>>>>>>> mechanism >>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html) >>>>>>>>>> >>>>>>>>>> >> >>>>>>>>>> > and all contributors to lightning must provide a cla. >>>>>>>>>> (It it's the case I can help) >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>>> perso I'd like we avoid hard dependency >>>>>>>>>>>>> on Unsafe as maybe some >>>>>> use >>>>>>>>>>> other jdks :-) >>>>>>>>>>> >>>>>>>>>>> Well, I believe Unsafe is supported by Sun >>>>>>>>>>> JDK, OpenJDK, IBM JDK and JRockit - and I >>>>>>>>>>> believe that it is more than enough. Also >>>>>>>>>>> keep in >>>>>> mind >>>>>>>>>> that >>>>>>>>>>> we already have an alternative Unsafe based >>>>>>>>>>> memory storage - and >>>>>>>> although >>>>>>>>>>> it's not thoroughly tested for performance it >>>>>>>>>>> dramaticaly simplifies >>>>>>>>>> code, >>>>>>>>>>> I have great expectations about it. >>>>>>>>>> Me too :-). Yup definitely more simple and >>>>>>>>>> faster ! But we must provide a switch off >>>>>>>>>> configuration mechanism if some >>>>>> users >>>>>>>>>> don't want to use that (because Unsafe is just >>>>>>>>>> "Unsafe" :-) ) And sorry I didn't have a look >>>>>>>>>> yet at your changes with using Unsafe. >>>>>>>>>>> >>>>>>>>>>> Cheers, R >>>>>>>>>>> >>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy >>>>>>>>>>> <[email protected]> >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi >>>>>>>>>>>> <[email protected]>: >>>>>>>>>>>>> What do you think about: 1) implementing >>>>>>>>>>>>> a lightning serialization module 2) >>>>>>>>>>>>> creating a serializer that directly works >>>>>>>>>>>>> on a directmemory >>>>>>>>>> provider >>>>>>>>>>>>> ByteBuffer or (maybe better) Unsafe based >>>>>>>>>>>>> Pointer? >>>>>>>>>>>> >>>>>>>>>>>> Sounds good (perso I'd like we avoid hard >>>>>>>>>>>> dependency on Unsafe as maybe some use >>>>>>>>>>>> other jdks :-) ) >>>>>>>>>>>>> >>>>>>>>>>>>> Now I see lightning is apache licensed >>>>>>>>>>>>> and this is fine but I >>>>>> don't >>>>>>>>>> think >>>>>>>>>>>>> it is published in any public maven >>>>>>>>>>>>> repo, am I right? We could >>>>>> find a >>>>>>>>>> way >>>>>>>>>>>>> to deal with this; options vary from >>>>>>>>>>>>> publishing lightning to the >>>>>> free >>>>>>>>>>>>> sonatype repo, joining the ASF (which >>>>>>>>>>>>> is great anyhow!) and >>>>>>>>>> republishing >>>>>>>>>>>>> lightning code in DirectMemory becoming a >>>>>>>>>>>>> commiter (which has to >>>>>>>>>> undergo >>>>>>>>>>>> a >>>>>>>>>>>>> PMC vote). >>>>>>>>>>>> >>>>>>>>>>>> At least for the moment he can provide a >>>>>>>>>>>> patch to be integrated in >>>>>> DM >>>>>>>>>> :-) >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I'd like to hear your and the team >>>>>>>>>>>>> feelings on this. >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, Raffaele >>>>>>>>>>>>> >>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, >>>>>>>>>>>>> Noctarius <[email protected]> >>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hey Raffaele, >>>>>>>>>>>>>> >>>>>>>>>>>>>> that's quite similar to what I did at >>>>>>>>>>>>>> work. We're developing >>>>>> Flash >>>>>>>>>>>>>> online games and using a customized AMF >>>>>>>>>>>>>> serialization. To support async >>>>>>>>>>>>>> writing of the clients event results I >>>>>>>>>>>>>> added serialization >>>>>> of >>>>>>>>>>>>>> the components / entities to the >>>>>>>>>>>>>> players zone calculation and >>>>>> stored >>>>>>>>>>>>>> the pre-serialized bytestream directly >>>>>>>>>>>>>> to the off-heap (using >>>>>>>>>>>>>> direct-ring-cache implementation). When >>>>>>>>>>>>>> the client requests the results (using >>>>>>>>>>>>>> long-polling) I just write the >>>>>>>>>>>>>> pre-serialized >>>>>> data to >>>>>>>>>>>>>> the right position to deserialize it by >>>>>>>>>>>>>> standard ways on Flash >>>>>> side. >>>>>>>>>>>>>> >>>>>>>>>>>>>> So yeah an seriliaztion to off-heap >>>>>>>>>>>>>> algorithm would be fine. You >>>>>> can >>>>>>>>>>>>>> avoid using byte arrays and minimalize >>>>>>>>>>>>>> GC. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers Chris >>>>>>>>>>>>>> >>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele >>>>>>>>>>>>>> P. Guidi: >>>>>>>>>>>>>>> Nice to hear back from you! Yes, I >>>>>>>>>>>>>>> was thinking about creating a >>>>>>>>>> new >>>>>>>>>>>>>> memory >>>>>>>>>>>>>>> storage implementation using Unsafe >>>>>>>>>>>>>>> (and I did it, recently) and >>>>>>>>>>>> also, as >>>>>>>>>>>>>>> DirectMemory relies heavily on >>>>>>>>>>>>>>> serialization (and supports many >>>>>> of >>>>>>>>>>>> them, >>>>>>>>>>>>>>> protostuff, protobuf, msgpack and of >>>>>>>>>>>>>>> course standard >>>>>>>>>> serialization), >>>>>>>>>>>>>> about >>>>>>>>>>>>>>> creating a simple embedded serializer >>>>>>>>>>>>>>> leveraging the same >>>>>>>>>> techniques >>>>>>>>>>>> you >>>>>>>>>>>>>>> used (Unsafe and bytecode >>>>>>>>>>>>>>> generation). The idea with embedding >>>>>>>>>>>>>>> is avoiding to serialize in a byte >>>>>>>>>>>>>>> array >>>>>>>>>> and >>>>>>>>>>>> then >>>>>>>>>>>>>>> moving the byte array to off-heap >>>>>>>>>>>>>>> memory (via Unsafe or >>>>>>>>>> ByteBuffers), >>>>>>>>>>>> and >>>>>>>>>>>>>>> serializing directly into a "managed" >>>>>>>>>>>>>>> off-heap buffer, thus >>>>>> further >>>>>>>>>>>>>>> optimizing heap utilization (less >>>>>>>>>>>>>>> GC). >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> What do you think about it? Does it >>>>>>>>>>>>>>> make any sense to you? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ciao, R >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, >>>>>>>>>>>>>>> Noctarius <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hey guys, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Raffaele found out about a project >>>>>>>>>>>>>>>> of mine (Lightning) a few >>>>>> weeks >>>>>>>>>>>>>>>> ago. Lightning is a heavy Unsafe >>>>>>>>>>>>>>>> and Bytecode generation using >>>>>>>>>>>>>>>> Serializer implementation. He told >>>>>>>>>>>>>>>> me that he was interested in adding >>>>>>>>>>>>>>>> something similar to DirectMemory >>>>>>>>>>>>>>>> and I would be glad to >>>>>>>>>> help >>>>>>>>>>>>>>>> out in this. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Another project I started a few >>>>>>>>>>>>>>>> days ago, since it was needed >>>>>> for >>>>>>>>>>>>>>>> work is DirectRingCache. The name >>>>>>>>>>>>>>>> not really meets to actual >>>>>>>>>>>>>>>> implementation since it's not yet >>>>>>>>>>>>>>>> a ring buffer using cache. I >>>>>>>>>> used >>>>>>>>>>>>>>>> this for a pre-serialization simple >>>>>>>>>>>>>>>> bytestream cache with self-growing >>>>>>>>>>>>>>>> buffers. It could be nice to have >>>>>>>>>>>>>>>> DirectMemory >>>>>> having >>>>>>>>>>>>>>>> raw "buffers" to write to or to >>>>>>>>>>>>>>>> read from. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Here are the links from the >>>>>>>>>>>>>>>> projects: >>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >> >>>>>>>>>>>>>>>> > https://github.com/noctarius/direct-ring-cache >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>> It would be nice to help out since I really like the idea >>>> of >>>>>>>>>>>>>>>> DirectMemory and since >>>>>>>>>>>>>>>> direct-ring-cache is some kind of >>>>>>>>>> reinventing >>>>>>>>>>>>>>>> the wheel. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Cheers Noctarius (Chris) >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- Olivier Lamy Talend: >>>>>>>>>>>> http://coders.talend.com >>>>>>>>>>>> http://twitter.com/olamy | >>>>>>>>>>>> http://linkedin.com/in/olamy >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- Olivier Lamy Talend: >>>>>>>>>> http://coders.talend.com >>>>>>>>>> http://twitter.com/olamy | >>>>>>>>>> http://linkedin.com/in/olamy >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- Olivier Lamy Talend: http://coders.talend.com >>>>>> http://twitter.com/olamy | >>>>>> http://linkedin.com/in/olamy >>>>>> >>>>> >>> >>> >>> >
-- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy
