Well we already have a NIO ready interface allowing direct access to DMs
managed bytebuffers but I think this is just half way to what could be
achieved optimally blending serialization and memory allocation together.
Lightning as a module is of course a good idea and it could easily evolve
as a subproject (for the more experienced asf members: is it a feasible
way?).
Ciao,
R
Il giorno 29/set/2012 17:44, "Noctarius" <[email protected]> ha scritto:
> Hey guys,
>
> ok there's no lightning binary available since lightning wasn't
> ready yet for releasing.
>
> For being the only developer it would be no problem to contribute
> the sourcebase for DirectMemory but I'm not sure yet if it wouldn't
> be better to seperate it to be available without using DirectMemory
> itself. I started it as a serializer for cluster synchronization,
> but it would be cool to contribute lightning as a subproject to
> DirectMemory :-)
>
> About the second project I would love to see a public available
> buffer API directly in DirectMemory so that project would be nearly
> needless :-) The only difference I think is the allocation strategy
> my implementation is using against the one DirectMemory has, but I'm
> pretty sure the allocation is extensible ;-)
>
> Like I said, for both projects I'm the only dev so there would be no
> IP problem. So if it's ok to you to not include lightning directly
> in DM I would be glad to contribute to the Apache Foundation :-)
>
> Cheers Chris
>
>
> Am 29.09.2012 16:53, schrieb Raffaele P. Guidi:
> > Ok so it's up to noctarius - your move! ;-) Regarding the new unsafe
> > storage: it's an opt-in feature that can be set with the fluent API (and
> > soon through the conference file).
> >
> > Ciao,
> > R
> > Il giorno 29/set/2012 16:45, "Olivier Lamy" <[email protected]> ha
> scritto:
> >
> >> 2012/9/29 Raffaele P. Guidi <[email protected]>:
> >>>>> At least for the moment he can provide a patch to be integrated in DM
> >> :-)
> >>>
> >>> Sure, but as lightning is not in any public mvn repo should its code be
> >>> re-published in our svn? Or what?
> >> @Apache we don't care about binaries, only sources are important ! (a
> >> bit theorical for sure but that's it :-) ).
> >> So if Noctarius was the only guy who participate in lightning. He can
> >> just provide a patch we could integrate as a new dm module (note: the
> >> patch must not contains any more copyright and all sources must have
> >> ASF licenses).
> >>
> >> "Copyright 2012 the original author or authors." must be removed.
> >> And BTW package must be changed :-) (com.github is not acceptable @asf
> >> :-) )(@Noctarius are you working for github ? :-) )
> >>
> >> And having him as a committer will be only a matter of voting (we have
> >> a great chair who take cares of administrative stuff :P )
> >>
> >> If some others have participated in the project, we must pass tru an
> >> ip clearance mechanism
> >> (http://incubator.apache.org/ip-clearance/index.html) and all
> >> contributors to lightning must provide a cla. (It it's the case I can
> >> help)
> >>
> >>>
> >>>>> perso I'd like we avoid hard dependency on Unsafe as maybe some use
> >>> other jdks :-)
> >>>
> >>> Well, I believe Unsafe is supported by Sun JDK, OpenJDK, IBM JDK and
> >>> JRockit - and I believe that it is more than enough. Also keep in mind
> >> that
> >>> we already have an alternative Unsafe based memory storage - and
> although
> >>> it's not thoroughly tested for performance it dramaticaly simplifies
> >> code,
> >>> I have great expectations about it.
> >> Me too :-). Yup definitely more simple and faster !
> >> But we must provide a switch off configuration mechanism if some users
> >> don't want to use that (because Unsafe is just "Unsafe" :-) )
> >> And sorry I didn't have a look yet at your changes with using Unsafe.
> >>>
> >>> Cheers,
> >>> R
> >>>
> >>> On Sat, Sep 29, 2012 at 4:03 PM, Olivier Lamy <[email protected]>
> wrote:
> >>>
> >>>> 2012/9/29 Raffaele P. Guidi <[email protected]>:
> >>>>> What do you think about:
> >>>>> 1) implementing a lightning serialization module
> >>>>> 2) creating a serializer that directly works on a directmemory
> >> provider
> >>>>> ByteBuffer or (maybe better) Unsafe based Pointer?
> >>>>
> >>>> Sounds good (perso I'd like we avoid hard dependency on Unsafe as
> >>>> maybe some use other jdks :-) )
> >>>>>
> >>>>> Now I see lightning is apache licensed and this is fine but I don't
> >> think
> >>>>> it is published in any public maven repo, am I right? We could find a
> >> way
> >>>>> to deal with this; options vary from publishing lightning to the free
> >>>>> sonatype repo, joining the ASF (which is great anyhow!) and
> >> republishing
> >>>>> lightning code in DirectMemory becoming a commiter (which has to
> >> undergo
> >>>> a
> >>>>> PMC vote).
> >>>>
> >>>> At least for the moment he can provide a patch to be integrated in DM
> >> :-)
> >>>>
> >>>>>
> >>>>> I'd like to hear your and the team feelings on this.
> >>>>>
> >>>>> Thanks,
> >>>>> Raffaele
> >>>>>
> >>>>> On Sat, Sep 29, 2012 at 3:27 PM, Noctarius <[email protected]> wrote:
> >>>>>
> >>>>>> Hey Raffaele,
> >>>>>>
> >>>>>> that's quite similar to what I did at work. We're developing Flash
> >>>>>> online games and using a customized AMF serialization. To support
> >>>>>> async writing of the clients event results I added serialization of
> >>>>>> the components / entities to the players zone calculation and stored
> >>>>>> the pre-serialized bytestream directly to the off-heap (using
> >>>>>> direct-ring-cache implementation). When the client requests the
> >>>>>> results (using long-polling) I just write the pre-serialized data to
> >>>>>> the right position to deserialize it by standard ways on Flash side.
> >>>>>>
> >>>>>> So yeah an seriliaztion to off-heap algorithm would be fine. You can
> >>>>>> avoid using byte arrays and minimalize GC.
> >>>>>>
> >>>>>> Cheers
> >>>>>> Chris
> >>>>>>
> >>>>>> Am 29.09.2012 15:02, schrieb Raffaele P. Guidi:
> >>>>>>> Nice to hear back from you! Yes, I was thinking about creating a
> >> new
> >>>>>> memory
> >>>>>>> storage implementation using Unsafe (and I did it, recently) and
> >>>> also, as
> >>>>>>> DirectMemory relies heavily on serialization (and supports many of
> >>>> them,
> >>>>>>> protostuff, protobuf, msgpack and of course standard
> >> serialization),
> >>>>>> about
> >>>>>>> creating a simple embedded serializer leveraging the same
> >> techniques
> >>>> you
> >>>>>>> used (Unsafe and bytecode generation).
> >>>>>>> The idea with embedding is avoiding to serialize in a byte array
> >> and
> >>>> then
> >>>>>>> moving the byte array to off-heap memory (via Unsafe or
> >> ByteBuffers),
> >>>> and
> >>>>>>> serializing directly into a "managed" off-heap buffer, thus further
> >>>>>>> optimizing heap utilization (less GC).
> >>>>>>>
> >>>>>>> What do you think about it? Does it make any sense to you?
> >>>>>>>
> >>>>>>> Ciao,
> >>>>>>> R
> >>>>>>>
> >>>>>>> On Sat, Sep 29, 2012 at 2:40 PM, Noctarius <[email protected]>
> >> wrote:
> >>>>>>>
> >>>>>>>> Hey guys,
> >>>>>>>>
> >>>>>>>> Raffaele found out about a project of mine (Lightning) a few weeks
> >>>>>>>> ago. Lightning is a heavy Unsafe and Bytecode generation using
> >>>>>>>> Serializer implementation. He told me that he was interested in
> >>>>>>>> adding something similar to DirectMemory and I would be glad to
> >> help
> >>>>>>>> out in this.
> >>>>>>>>
> >>>>>>>> Another project I started a few days ago, since it was needed for
> >>>>>>>> work is DirectRingCache. The name not really meets to actual
> >>>>>>>> implementation since it's not yet a ring buffer using cache. I
> >> used
> >>>>>>>> this for a pre-serialization simple bytestream cache with
> >>>>>>>> self-growing buffers. It could be nice to have DirectMemory having
> >>>>>>>> raw "buffers" to write to or to read from.
> >>>>>>>>
> >>>>>>>> Here are the links from the projects:
> >>>>>>>> https://github.com/noctarius/Lightning
> >>>>>>>> https://github.com/noctarius/direct-ring-cache
> >>>>>>>>
> >>>>>>>> It would be nice to help out since I really like the idea of
> >>>>>>>> DirectMemory and since direct-ring-cache is some kind of
> >> reinventing
> >>>>>>>> the wheel.
> >>>>>>>>
> >>>>>>>> Cheers
> >>>>>>>> Noctarius (Chris)
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Olivier Lamy
> >>>> Talend: http://coders.talend.com
> >>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>>>
> >>
> >>
> >>
> >> --
> >> Olivier Lamy
> >> Talend: http://coders.talend.com
> >> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >>
> >
>
