If that is possible there's no problem from my side. It would be an honor to give the Apache Foundation something instead of only taking code :-)
PS: I'm pretty sure there's a way to make lightning again a little bit faster by removing boxing on primitives by using invokedynamic. I started it but left the idea behind to finish the public api first. Am 29.09.2012 17:58, schrieb Olivier Lamy: > 2012/9/29 Raffaele P. Guidi <[email protected]>: >> Well we already have a NIO ready interface allowing direct >> access to DMs managed bytebuffers but I think this is just >> half way to what could be achieved optimally blending >> serialization and memory allocation together. >> >> Lightning as a module is of course a good idea and it could >> easily evolve as a subproject (for the more experienced asf >> members: is it a feasible way?). > Nothing prevent to have > http://svn.apache.org/repos/asf/directmemory/lightning/trunk > with a package like: o.a.d.lightning That will be a > subproject. >> >> Ciao, R Il giorno 29/set/2012 17:44, "Noctarius" >> <[email protected]> ha scritto: >> >>> Hey guys, >>> >>> ok there's no lightning binary available since lightning >>> wasn't ready yet for releasing. >>> >>> For being the only developer it would be no problem to >>> contribute the sourcebase for DirectMemory but I'm not sure >>> yet if it wouldn't be better to seperate it to be available >>> without using DirectMemory itself. I started it as a >>> serializer for cluster synchronization, but it would be >>> cool to contribute lightning as a subproject to >>> DirectMemory :-) >>> >>> About the second project I would love to see a public >>> available buffer API directly in DirectMemory so that >>> project would be nearly needless :-) The only difference I >>> think is the allocation strategy my implementation is using >>> against the one DirectMemory has, but I'm pretty sure the >>> allocation is extensible ;-) >>> >>> Like I said, for both projects I'm the only dev so there >>> would be no IP problem. So if it's ok to you to not include >>> lightning directly in DM I would be glad to contribute to >>> the Apache Foundation :-) >>> >>> Cheers Chris >>> >>> >>> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi: >>>> Ok so it's up to noctarius - your move! ;-) Regarding the >>>> new unsafe storage: it's an opt-in feature that can be >>>> set with the fluent API (and soon through the conference >>>> file). >>>> >>>> Ciao, R Il giorno 29/set/2012 16:45, "Olivier Lamy" >>>> <[email protected]> ha >>> scritto: >>>> >>>>> 2012/9/29 Raffaele P. Guidi >>>>> <[email protected]>: >>>>>>>> At least for the moment he can provide a patch to >>>>>>>> be integrated in DM >>>>> :-) >>>>>> >>>>>> Sure, but as lightning is not in any public mvn repo >>>>>> should its code be re-published in our svn? Or what? >>>>> @Apache we don't care about binaries, only sources are >>>>> important ! (a bit theorical for sure but that's it :-) >>>>> ). So if Noctarius was the only guy who participate in >>>>> lightning. He can just provide a patch we could >>>>> integrate as a new dm module (note: the patch must not >>>>> contains any more copyright and all sources must have >>>>> ASF licenses). >>>>> >>>>> "Copyright 2012 the original author or authors." must >>>>> be removed. And BTW package must be changed :-) >>>>> (com.github is not acceptable @asf :-) )(@Noctarius are >>>>> you working for github ? :-) ) >>>>> >>>>> And having him as a committer will be only a matter of >>>>> voting (we have a great chair who take cares of >>>>> administrative stuff :P ) >>>>> >>>>> If some others have participated in the project, we >>>>> must pass tru an ip clearance mechanism >>>>> (http://incubator.apache.org/ip-clearance/index.html) >>>>> and all contributors to lightning must provide a cla. >>>>> (It it's the case I can help) >>>>> >>>>>> >>>>>>>> perso I'd like we avoid hard dependency on Unsafe >>>>>>>> as maybe some use >>>>>> other jdks :-) >>>>>> >>>>>> Well, I believe Unsafe is supported by Sun JDK, >>>>>> OpenJDK, IBM JDK and JRockit - and I believe that it >>>>>> is more than enough. Also keep in mind >>>>> that >>>>>> we already have an alternative Unsafe based memory >>>>>> storage - and >>> although >>>>>> it's not thoroughly tested for performance it >>>>>> dramaticaly simplifies >>>>> code, >>>>>> I have great expectations about it. >>>>> Me too :-). Yup definitely more simple and faster ! But >>>>> we must provide a switch off configuration mechanism if >>>>> some users don't want to use that (because Unsafe is >>>>> just "Unsafe" :-) ) And sorry I didn't have a look yet >>>>> at your changes with using Unsafe. >>>>>> >>>>>> Cheers, R >>>>>> >>>>>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy >>>>>> <[email protected]> >>> wrote: >>>>>> >>>>>>> 2012/9/29 Raffaele P. Guidi >>>>>>> <[email protected]>: >>>>>>>> What do you think about: 1) implementing a >>>>>>>> lightning serialization module 2) creating a >>>>>>>> serializer that directly works on a directmemory >>>>> provider >>>>>>>> ByteBuffer or (maybe better) Unsafe based >>>>>>>> Pointer? >>>>>>> >>>>>>> Sounds good (perso I'd like we avoid hard >>>>>>> dependency on Unsafe as maybe some use other jdks >>>>>>> :-) ) >>>>>>>> >>>>>>>> Now I see lightning is apache licensed and this >>>>>>>> is fine but I don't >>>>> think >>>>>>>> it is published in any public maven repo, am I >>>>>>>> right? We could find a >>>>> way >>>>>>>> to deal with this; options vary from publishing >>>>>>>> lightning to the free sonatype repo, joining the >>>>>>>> ASF (which is great anyhow!) and >>>>> republishing >>>>>>>> lightning code in DirectMemory becoming a >>>>>>>> commiter (which has to >>>>> undergo >>>>>>> a >>>>>>>> PMC vote). >>>>>>> >>>>>>> At least for the moment he can provide a patch to >>>>>>> be integrated in DM >>>>> :-) >>>>>>> >>>>>>>> >>>>>>>> I'd like to hear your and the team feelings on >>>>>>>> this. >>>>>>>> >>>>>>>> Thanks, Raffaele >>>>>>>> >>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius >>>>>>>> <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hey Raffaele, >>>>>>>>> >>>>>>>>> that's quite similar to what I did at work. >>>>>>>>> We're developing Flash online games and using a >>>>>>>>> customized AMF serialization. To support async >>>>>>>>> writing of the clients event results I added >>>>>>>>> serialization of the components / entities to >>>>>>>>> the players zone calculation and stored the >>>>>>>>> pre-serialized bytestream directly to the >>>>>>>>> off-heap (using direct-ring-cache >>>>>>>>> implementation). When the client requests the >>>>>>>>> results (using long-polling) I just write the >>>>>>>>> pre-serialized data to the right position to >>>>>>>>> deserialize it by standard ways on Flash side. >>>>>>>>> >>>>>>>>> So yeah an seriliaztion to off-heap algorithm >>>>>>>>> would be fine. You can avoid using byte arrays >>>>>>>>> and minimalize GC. >>>>>>>>> >>>>>>>>> Cheers Chris >>>>>>>>> >>>>>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. >>>>>>>>> Guidi: >>>>>>>>>> Nice to hear back from you! Yes, I was >>>>>>>>>> thinking about creating a >>>>> new >>>>>>>>> memory >>>>>>>>>> storage implementation using Unsafe (and I >>>>>>>>>> did it, recently) and >>>>>>> also, as >>>>>>>>>> DirectMemory relies heavily on serialization >>>>>>>>>> (and supports many of >>>>>>> them, >>>>>>>>>> protostuff, protobuf, msgpack and of course >>>>>>>>>> standard >>>>> serialization), >>>>>>>>> about >>>>>>>>>> creating a simple embedded serializer >>>>>>>>>> leveraging the same >>>>> techniques >>>>>>> you >>>>>>>>>> used (Unsafe and bytecode generation). The >>>>>>>>>> idea with embedding is avoiding to serialize >>>>>>>>>> in a byte array >>>>> and >>>>>>> then >>>>>>>>>> moving the byte array to off-heap memory (via >>>>>>>>>> Unsafe or >>>>> ByteBuffers), >>>>>>> and >>>>>>>>>> serializing directly into a "managed" >>>>>>>>>> off-heap buffer, thus further optimizing heap >>>>>>>>>> utilization (less GC). >>>>>>>>>> >>>>>>>>>> What do you think about it? Does it make any >>>>>>>>>> sense to you? >>>>>>>>>> >>>>>>>>>> Ciao, R >>>>>>>>>> >>>>>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius >>>>>>>>>> <[email protected]> >>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hey guys, >>>>>>>>>>> >>>>>>>>>>> Raffaele found out about a project of mine >>>>>>>>>>> (Lightning) a few weeks ago. Lightning is a >>>>>>>>>>> heavy Unsafe and Bytecode generation using >>>>>>>>>>> Serializer implementation. He told me that >>>>>>>>>>> he was interested in adding something >>>>>>>>>>> similar to DirectMemory and I would be glad >>>>>>>>>>> to >>>>> help >>>>>>>>>>> out in this. >>>>>>>>>>> >>>>>>>>>>> Another project I started a few days ago, >>>>>>>>>>> since it was needed for work is >>>>>>>>>>> DirectRingCache. The name not really meets >>>>>>>>>>> to actual implementation since it's not yet >>>>>>>>>>> a ring buffer using cache. I >>>>> used >>>>>>>>>>> this for a pre-serialization simple >>>>>>>>>>> bytestream cache with self-growing buffers. >>>>>>>>>>> It could be nice to have DirectMemory >>>>>>>>>>> having raw "buffers" to write to or to read >>>>>>>>>>> from. >>>>>>>>>>> >>>>>>>>>>> Here are the links from the projects: >>>>>>>>>>> https://github.com/noctarius/Lightning >>>>>>>>>>> https://github.com/noctarius/direct-ring-cache >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> It would be nice to help out since I really like the idea of >>>>>>>>>>> DirectMemory and since direct-ring-cache is >>>>>>>>>>> some kind of >>>>> reinventing >>>>>>>>>>> the wheel. >>>>>>>>>>> >>>>>>>>>>> Cheers Noctarius (Chris) >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- Olivier Lamy Talend: http://coders.talend.com >>>>>>> http://twitter.com/olamy | >>>>>>> http://linkedin.com/in/olamy >>>>>>> >>>>> >>>>> >>>>> >>>>> -- Olivier Lamy Talend: http://coders.talend.com >>>>> http://twitter.com/olamy | >>>>> http://linkedin.com/in/olamy >>>>> >>>> >>> > > >
