Cool. Hope to see those things in trunk and to hear from you soon.
Cheers,
R
Il giorno 29/set/2012 22:10, "Roman Levenstein" <[email protected]> ha
scritto:
> Hi,
>
>
>
> On Sat, Sep 29, 2012 at 9:56 PM, Raffaele P. Guidi
> <[email protected]> wrote:
> > Really nice ideas - we already have an Unsafe based storage (it is rather
> > new) and the idea was exactly to combine the benefits of it with the ones
> > of unsafe serialization. Said that keep in mind that classic DirectMemory
> > is in any case way faster than just using ByteBuffers because it
> > pre-allocates large chunks of memory and then "slices" them for single,
> > smaller items.
>
> Yes, pre-allocation is good. But you should really see the performance
> benefits of writing/reading off-heap memory using Unsafe :-)
> It is way faster, especially for arrays of primitive types. The link I
> mentioned contained some preliminary figures based on my own
> benchmarks.
> Therefore I think even Lightning (or any other serialization lib)
> could benefit from using Unsafe-based off-heap Input/Output streams.
> BTW, my streams support both DirectBuffer-based and
> Unsafe.allocateMemory-based (i.e. pre-allocated) off-heap memory
> access. Therefore, it is a nice fit with DirectMemory.
>
> > And - a Kryo serializer is a rather good idea. Do you feel like working
> on
> > it? Serializers are actually one of the easiest thing to contribute on in
> > DM - and they add immediate value as each one is probably a better fit
> than
> > others in different scenarios.
>
> Yes, I think I could work on it. But first, I'd like to land these
> improvements in Kryo's trunk so that I don't need to work on a branch
> or a fork of Kryo.
>
> Ciao,
> Roman
>
> > Il giorno 29/set/2012 21:16, "Roman Levenstein" <[email protected]> ha
> > scritto:
> >
> >> Hi,
> >>
> >> I'm just wondering if lightning uses Unsafe only for serializers (i.e.
> >> for reading/writing fields of objects) or also for the input/output
> >> streams used to read/write to/from off-heap memory? Using
> >> DirectBuffers "as is" for accessing off-heap memory is actually rather
> >> slow.
> >>
> >> Let me share some experiences with you:
> >> I have implemented a similar feature for Kryo. It is not in the trunk
> >> yet, as it is being reviewed by Nate, the author of Kryo.
> >> One of the things that I noticed is that using Unsafe only for
> >> serializers is nice, but does not provide much improvement.
> >> At the same time, using Unsafe for input/output streams and
> >> reading/writing directly from/into off-heap memory provides
> >> significant performance benefits.
> >> My patch for Kryo implements both features using Unsafe. If you want
> >> to see more details about the implementation and findings, please have
> >> a look at:
> >> http://code.google.com/p/kryo/issues/detail?id=75
> >>
> >> I hope it will land in Kryo trunk soon. Once it is there, I think it
> >> could be interesting to add a Kryo-based serialization to
> >> DirectMemory. What do you think?
> >> But independent of this, feel free to take any ideas or even the
> >> implementation from the patch. The off-heap memory streams
> >> implementation is rather self-contained and could be reused with other
> >> frameworks besides Kryo.
> >>
> >> Regards,
> >> Roman
> >>
> >> On Sat, Sep 29, 2012 at 8:46 PM, Noctarius <[email protected]> wrote:
> >> > Actually all dependencies should be AL2 or BSD licensed :-)
> >> >
> >> > Am 29.09.2012 20:42, schrieb Raffaele P. Guidi:
> >> >>>> Hehe well that really sounds like a nice bunch of people.
> >> >>
> >> >> Indeed they are (I'm a newbie as well and try to do my best)
> >> >>
> >> >>>> If lightning will be a sub-part (sub-project) of DM, do I
> >> >>>> need to write
> >> >> an project purposal?
> >> >>
> >> >> Nope, not needed for a sub-project
> >> >>
> >> >>>> Do I need to make any changes to the pom.xml like adding a
> >> >> special parent pom or anything like that?
> >> >>
> >> >> Not for the serializer - just have to take a look at project
> >> >> dependencies - or, better, at their licenses - are they
> >> >> compatible with the ASL 2.0? i.e. a GPL'd library is not a good
> >> >> fit and should be replaced with an apache licensed (or BSD, or
> >> >> MIT...) one if possible. For the integration module is a
> >> >> separate story - you should start off copying one of the other
> >> >> serializers and reusing the same pom and directory structure.
> >> >>
> >> >> Pleased to meet you, Chris :)
> >> >>
> >> >> Ciao, R
> >> >>
> >> >> On Sat, Sep 29, 2012 at 8:24 PM, Noctarius <[email protected]>
> >> >> wrote:
> >> >>
> >> >>> Hehe well that really sounds like a nice bunch of people.
> >> >>>
> >> >>> Ok to be true I couldn't wait until tomorrow and started
> >> >>> already reading the links. From what I was reading: If
> >> >>> lightning will be a sub-part (sub-project) of DM, do I need
> >> >>> to write an project purposal?
> >> >>>
> >> >>> Do I need to make any changes to the pom.xml like adding a
> >> >>> special parent pom or anything like that?
> >> >>>
> >> >>> In general: there are a lot things to know :-)
> >> >>>
> >> >>> Am 29.09.2012 19:59, schrieb Raffaele P. Guidi:
> >> >>>> Negative part of ASF membership? You get together with a
> >> >>>> lot of geeky, talented people with a fixation for software
> >> >>>> and open source. Oh wait but this is actually nice! :-D Il
> >> >>>> giorno 29/set/2012 19:05, "Olivier Lamy" <[email protected]>
> >> >>>> ha
> >> >>> scritto:
> >> >>>>
> >> >>>>> 2012/9/29 Noctarius <[email protected]>:
> >> >>>>>> Thanks Olivier for carify, I'll take a look in it
> >> >>>>>> tomorrow but there's just one question left (for now
> >> >>>>>> ;)): What is that vote for becoming a committer? What
> >> >>>>>> if the vote will be negative?
> >> >>>>> The vote is on private list (pmc list for privacy reasons
> >> >>>>> and possible negative stuff being on public lists)
> >> >>>>>> Until now I just used Apache stuff, was never
> >> >>>>>> interested in being part of it so I guess it can be
> >> >>>>>> negative for any reason, can't it?
> >> >>>>> I don't see why it could be negative but suspens ....
> >> >>>>> :-)
> >> >>>>>>
> >> >>>>>> Am 29.09.2012 18:56, schrieb Olivier Lamy:
> >> >>>>>>> 2012/9/29 Noctarius <[email protected]>:
> >> >>>>>>>> Nope my real name is Christoph Engelbert, but
> >> >>>>>>>> Noctarius is the all time nick :)
> >> >>>>>>>>
> >> >>>>>>>> Renaming the package should be no problem, is it
> >> >>>>>>>> "org.apache.directmemory.lightning" or what would
> >> >>>>>>>> it be?
> >> >>>>>>> fine for me
> >> >>>>>>>> Then there needs to be a change in the license
> >> >>>>>>>> header as Olivier mentioned, that means just remove
> >> >>>>>>>> the first sentence or is there anything more to do
> >> >>>>>>>> (maybe it's easiest thing to just copy the header
> >> >>>>>>>> from DM file ;))?
> >> >>>>>>> yup use same header as DM
> >> >>>>>>>>
> >> >>>>>>>> The CLA is just a form to clarify that the source
> >> >>>>>>>> can be contributed to the Apache Foundation?
> >> >>>>>>> yup correct.
> >> >>>>>>>>
> >> >>>>>>>> The final step will be attaching the patch in form
> >> >>>>>>>> of a huge diff file?
> >> >>>>>>> yes
> >> >>>>>>>>
> >> >>>>>>>> And what is the way to apply for a membership?
> >> >>>>>>>> Never thought about how to do that.
> >> >>>>>>> Read here
> >> >>>>>>> http://apache.org/foundation/how-it-works.html and
> >> >>>>>>> here http://apache.org/foundation/getinvolved.html .
> >> >>>>>>> And feel free to ask any questions :-)
> >> >>>>>>>>
> >> >>>>>>>> Chris
> >> >>>>>>>>
> >> >>>>>>>> Am 29.09.2012 18:23, schrieb Raffaele P. Guidi:
> >> >>>>>>>>> OK, deal, at least for me ;-) I propose you
> >> >>>>>>>>> rename the packages, produce a patch for this and
> >> >>>>>>>>> the new serializer module (should be simple
> >> >>>>>>>>> enough starting from an existing one) and, in the
> >> >>>>>>>>> meanwhile, apply for ASF membership. Is IP
> >> >>>>>>>>> clearance needed? I guess yes. After this we will
> >> >>>>>>>>> come up with a formal vote regarding Noctarius
> >> >>>>>>>>> (is this your real name?!) allowance in the
> >> >>>>>>>>> project team.
> >> >>>>>>>>>
> >> >>>>>>>>> Good times are gonna come :-)
> >> >>>>>>>>>
> >> >>>>>>>>> Thanks, R Il giorno 29/set/2012 17:58, "Olivier
> >> >>>>>>>>> Lamy" <[email protected]> ha scritto:
> >> >>>>>>>>>
> >> >>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >> >>>>>>>>>> <[email protected]>:
> >> >>>>>>>>>>> Well we already have a NIO ready interface
> >> >>>>>>>>>>> allowing direct access to DMs managed
> >> >>>>>>>>>>> bytebuffers but I think this is just half way
> >> >>>>>>>>>>> to what could be achieved optimally blending
> >> >>>>>>>>>>> serialization and memory allocation
> >> >>>>>>>>>>> together.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Lightning as a module is of course a good
> >> >>>>>>>>>>> idea and it could easily evolve as a
> >> >>>>>>>>>>> subproject (for the more experienced asf
> >> >>>>>>>>>>> members: is it a feasible way?).
> >> >>>>>>>>>> Nothing prevent to have
> >> >>>>>>>>>> http://svn.apache.org/repos/asf/directmemory/lightning/trunk
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>
> >> >>>>>>>>>>
> >> > with a package like: o.a.d.lightning That will be a
> >> >>>>>>>>>> subproject.
> >> >>>>>>>>>>>
> >> >>>>>>>>>>> Ciao, R Il giorno 29/set/2012 17:44,
> >> >>>>>>>>>>> "Noctarius" <[email protected]> ha scritto:
> >> >>>>>>>>>>>
> >> >>>>>>>>>>>> Hey guys,
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> ok there's no lightning binary available
> >> >>>>>>>>>>>> since lightning wasn't ready yet for
> >> >>>>>>>>>>>> releasing.
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> For being the only developer it would be no
> >> >>>>>>>>>>>> problem to contribute the sourcebase for
> >> >>>>>>>>>>>> DirectMemory but I'm not sure yet if it
> >> >>>>>>>>>>>> wouldn't be better to seperate it to be
> >> >>>>>>>>>>>> available without using DirectMemory
> >> >>>>>>>>>>>> itself. I started it as a serializer for
> >> >>>>>>>>>>>> cluster synchronization, but it would be
> >> >>>>>>>>>>>> cool to contribute lightning as a
> >> >>>>>>>>>>>> subproject to DirectMemory :-)
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> About the second project I would love to
> >> >>>>>>>>>>>> see a public available buffer API directly
> >> >>>>>>>>>>>> in DirectMemory so that project would be
> >> >>>>>>>>>>>> nearly needless :-) The only difference I
> >> >>>>>>>>>>>> think is the allocation strategy my
> >> >>>>>>>>>>>> implementation is using against the one
> >> >>>>>>>>>>>> DirectMemory has, but I'm pretty sure the
> >> >>>>>>>>>>>> allocation is extensible ;-)
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Like I said, for both projects I'm the only
> >> >>>>>>>>>>>> dev so there would be no IP problem. So if
> >> >>>>>>>>>>>> it's ok to you to not include lightning
> >> >>>>>>>>>>>> directly in DM I would be glad to
> >> >>>>>>>>>>>> contribute to the Apache Foundation :-)
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Cheers Chris
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>>> Am 29.09.2012 16:53, schrieb Raffaele P.
> >> >>>>>>>>>>>> Guidi:
> >> >>>>>>>>>>>>> Ok so it's up to noctarius - your move!
> >> >>>>>>>>>>>>> ;-) Regarding the new unsafe storage:
> >> >>>>>>>>>>>>> it's an opt-in feature that can be set
> >> >>>>>>>>>>>>> with the fluent API
> >> >>>>>>>>>> (and
> >> >>>>>>>>>>>>> soon through the conference file).
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>> Ciao, R Il giorno 29/set/2012 16:45,
> >> >>>>>>>>>>>>> "Olivier Lamy" <[email protected]> ha
> >> >>>>>>>>>>>> scritto:
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >> >>>>>>>>>>>>>> <[email protected]>:
> >> >>>>>>>>>>>>>>>>> At least for the moment he can
> >> >>>>>>>>>>>>>>>>> provide a patch to be integrated
> >> >>>>>>>>>> in DM
> >> >>>>>>>>>>>>>> :-)
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Sure, but as lightning is not in any
> >> >>>>>>>>>>>>>>> public mvn repo should its
> >> >>>>>>>>>> code be
> >> >>>>>>>>>>>>>>> re-published in our svn? Or what?
> >> >>>>>>>>>>>>>> @Apache we don't care about binaries,
> >> >>>>>>>>>>>>>> only sources are important ! (a bit
> >> >>>>>>>>>>>>>> theorical for sure but that's it :-) ).
> >> >>>>>>>>>>>>>> So if Noctarius was the only guy who
> >> >>>>>>>>>>>>>> participate in lightning. He can just
> >> >>>>>>>>>>>>>> provide a patch we could integrate as a
> >> >>>>>>>>>>>>>> new dm module (note: the patch must not
> >> >>>>>>>>>>>>>> contains any more copyright and all
> >> >>>>>>>>>>>>>> sources must have ASF licenses).
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> "Copyright 2012 the original author or
> >> >>>>>>>>>>>>>> authors." must be removed. And BTW
> >> >>>>>>>>>>>>>> package must be changed :-) (com.github
> >> >>>>>>>>>>>>>> is not acceptable
> >> >>>>>>>>>> @asf
> >> >>>>>>>>>>>>>> :-) )(@Noctarius are you working for
> >> >>>>>>>>>>>>>> github ? :-) )
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> And having him as a committer will be
> >> >>>>>>>>>>>>>> only a matter of voting (we
> >> >>>>>>>>>> have
> >> >>>>>>>>>>>>>> a great chair who take cares of
> >> >>>>>>>>>>>>>> administrative stuff :P )
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> If some others have participated in the
> >> >>>>>>>>>>>>>> project, we must pass tru an ip
> >> >>>>>>>>>>>>>> clearance mechanism
> >> >>>>>>>>>>>>>> (http://incubator.apache.org/ip-clearance/index.html)
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>
> >> >>>>>>>>>>>>>>
> >> > and all contributors to lightning must provide a cla.
> >> >>>>>>>>>>>>>> (It it's the case I can help)
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> perso I'd like we avoid hard
> >> >>>>>>>>>>>>>>>>> dependency on Unsafe as maybe
> >> >>>>>>>>>>>>>>>>> some
> >> >>>>>>>>>> use
> >> >>>>>>>>>>>>>>> other jdks :-)
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Well, I believe Unsafe is supported
> >> >>>>>>>>>>>>>>> by Sun JDK, OpenJDK, IBM JDK and
> >> >>>>>>>>>>>>>>> JRockit - and I believe that it is
> >> >>>>>>>>>>>>>>> more than enough. Also keep in
> >> >>>>>>>>>> mind
> >> >>>>>>>>>>>>>> that
> >> >>>>>>>>>>>>>>> we already have an alternative Unsafe
> >> >>>>>>>>>>>>>>> based memory storage - and
> >> >>>>>>>>>>>> although
> >> >>>>>>>>>>>>>>> it's not thoroughly tested for
> >> >>>>>>>>>>>>>>> performance it dramaticaly
> >> >>>>>>>>>>>>>>> simplifies
> >> >>>>>>>>>>>>>> code,
> >> >>>>>>>>>>>>>>> I have great expectations about it.
> >> >>>>>>>>>>>>>> Me too :-). Yup definitely more simple
> >> >>>>>>>>>>>>>> and faster ! But we must provide a
> >> >>>>>>>>>>>>>> switch off configuration mechanism if
> >> >>>>>>>>>>>>>> some
> >> >>>>>>>>>> users
> >> >>>>>>>>>>>>>> don't want to use that (because Unsafe
> >> >>>>>>>>>>>>>> is just "Unsafe" :-) ) And sorry I
> >> >>>>>>>>>>>>>> didn't have a look yet at your changes
> >> >>>>>>>>>>>>>> with using Unsafe.
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> Cheers, R
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 4:03 PM,
> >> >>>>>>>>>>>>>>> Olivier Lamy <[email protected]>
> >> >>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> 2012/9/29 Raffaele P. Guidi
> >> >>>>>>>>>>>>>>>> <[email protected]>:
> >> >>>>>>>>>>>>>>>>> What do you think about: 1)
> >> >>>>>>>>>>>>>>>>> implementing a lightning
> >> >>>>>>>>>>>>>>>>> serialization module 2) creating
> >> >>>>>>>>>>>>>>>>> a serializer that directly works
> >> >>>>>>>>>>>>>>>>> on a directmemory
> >> >>>>>>>>>>>>>> provider
> >> >>>>>>>>>>>>>>>>> ByteBuffer or (maybe better)
> >> >>>>>>>>>>>>>>>>> Unsafe based Pointer?
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> Sounds good (perso I'd like we
> >> >>>>>>>>>>>>>>>> avoid hard dependency on Unsafe as
> >> >>>>>>>>>>>>>>>> maybe some use other jdks :-) )
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Now I see lightning is apache
> >> >>>>>>>>>>>>>>>>> licensed and this is fine but I
> >> >>>>>>>>>> don't
> >> >>>>>>>>>>>>>> think
> >> >>>>>>>>>>>>>>>>> it is published in any public
> >> >>>>>>>>>>>>>>>>> maven repo, am I right? We could
> >> >>>>>>>>>> find a
> >> >>>>>>>>>>>>>> way
> >> >>>>>>>>>>>>>>>>> to deal with this; options vary
> >> >>>>>>>>>>>>>>>>> from publishing lightning to the
> >> >>>>>>>>>> free
> >> >>>>>>>>>>>>>>>>> sonatype repo, joining the ASF
> >> >>>>>>>>>>>>>>>>> (which is great anyhow!) and
> >> >>>>>>>>>>>>>> republishing
> >> >>>>>>>>>>>>>>>>> lightning code in DirectMemory
> >> >>>>>>>>>>>>>>>>> becoming a commiter (which has
> >> >>>>>>>>>>>>>>>>> to
> >> >>>>>>>>>>>>>> undergo
> >> >>>>>>>>>>>>>>>> a
> >> >>>>>>>>>>>>>>>>> PMC vote).
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> At least for the moment he can
> >> >>>>>>>>>>>>>>>> provide a patch to be integrated
> >> >>>>>>>>>>>>>>>> in
> >> >>>>>>>>>> DM
> >> >>>>>>>>>>>>>> :-)
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> I'd like to hear your and the
> >> >>>>>>>>>>>>>>>>> team feelings on this.
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> Thanks, Raffaele
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 3:27 PM,
> >> >>>>>>>>>>>>>>>>> Noctarius <[email protected]>
> >> >>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Hey Raffaele,
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> that's quite similar to what I
> >> >>>>>>>>>>>>>>>>>> did at work. We're developing
> >> >>>>>>>>>> Flash
> >> >>>>>>>>>>>>>>>>>> online games and using a
> >> >>>>>>>>>>>>>>>>>> customized AMF serialization.
> >> >>>>>>>>>>>>>>>>>> To support async writing of the
> >> >>>>>>>>>>>>>>>>>> clients event results I added
> >> >>>>>>>>>>>>>>>>>> serialization
> >> >>>>>>>>>> of
> >> >>>>>>>>>>>>>>>>>> the components / entities to
> >> >>>>>>>>>>>>>>>>>> the players zone calculation
> >> >>>>>>>>>>>>>>>>>> and
> >> >>>>>>>>>> stored
> >> >>>>>>>>>>>>>>>>>> the pre-serialized bytestream
> >> >>>>>>>>>>>>>>>>>> directly to the off-heap (using
> >> >>>>>>>>>>>>>>>>>> direct-ring-cache
> >> >>>>>>>>>>>>>>>>>> implementation). When the
> >> >>>>>>>>>>>>>>>>>> client requests the results
> >> >>>>>>>>>>>>>>>>>> (using long-polling) I just
> >> >>>>>>>>>>>>>>>>>> write the pre-serialized
> >> >>>>>>>>>> data to
> >> >>>>>>>>>>>>>>>>>> the right position to
> >> >>>>>>>>>>>>>>>>>> deserialize it by standard ways
> >> >>>>>>>>>>>>>>>>>> on Flash
> >> >>>>>>>>>> side.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> So yeah an seriliaztion to
> >> >>>>>>>>>>>>>>>>>> off-heap algorithm would be
> >> >>>>>>>>>>>>>>>>>> fine. You
> >> >>>>>>>>>> can
> >> >>>>>>>>>>>>>>>>>> avoid using byte arrays and
> >> >>>>>>>>>>>>>>>>>> minimalize GC.
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Cheers Chris
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>> Am 29.09.2012 15:02, schrieb
> >> >>>>>>>>>>>>>>>>>> Raffaele P. Guidi:
> >> >>>>>>>>>>>>>>>>>>> Nice to hear back from you!
> >> >>>>>>>>>>>>>>>>>>> Yes, I was thinking about
> >> >>>>>>>>>>>>>>>>>>> creating a
> >> >>>>>>>>>>>>>> new
> >> >>>>>>>>>>>>>>>>>> memory
> >> >>>>>>>>>>>>>>>>>>> storage implementation using
> >> >>>>>>>>>>>>>>>>>>> Unsafe (and I did it,
> >> >>>>>>>>>>>>>>>>>>> recently) and
> >> >>>>>>>>>>>>>>>> also, as
> >> >>>>>>>>>>>>>>>>>>> DirectMemory relies heavily
> >> >>>>>>>>>>>>>>>>>>> on serialization (and
> >> >>>>>>>>>>>>>>>>>>> supports many
> >> >>>>>>>>>> of
> >> >>>>>>>>>>>>>>>> them,
> >> >>>>>>>>>>>>>>>>>>> protostuff, protobuf, msgpack
> >> >>>>>>>>>>>>>>>>>>> and of course standard
> >> >>>>>>>>>>>>>> serialization),
> >> >>>>>>>>>>>>>>>>>> about
> >> >>>>>>>>>>>>>>>>>>> creating a simple embedded
> >> >>>>>>>>>>>>>>>>>>> serializer leveraging the
> >> >>>>>>>>>>>>>>>>>>> same
> >> >>>>>>>>>>>>>> techniques
> >> >>>>>>>>>>>>>>>> you
> >> >>>>>>>>>>>>>>>>>>> used (Unsafe and bytecode
> >> >>>>>>>>>>>>>>>>>>> generation). The idea with
> >> >>>>>>>>>>>>>>>>>>> embedding is avoiding to
> >> >>>>>>>>>>>>>>>>>>> serialize in a byte array
> >> >>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>> then
> >> >>>>>>>>>>>>>>>>>>> moving the byte array to
> >> >>>>>>>>>>>>>>>>>>> off-heap memory (via Unsafe
> >> >>>>>>>>>>>>>>>>>>> or
> >> >>>>>>>>>>>>>> ByteBuffers),
> >> >>>>>>>>>>>>>>>> and
> >> >>>>>>>>>>>>>>>>>>> serializing directly into a
> >> >>>>>>>>>>>>>>>>>>> "managed" off-heap buffer,
> >> >>>>>>>>>>>>>>>>>>> thus
> >> >>>>>>>>>> further
> >> >>>>>>>>>>>>>>>>>>> optimizing heap utilization
> >> >>>>>>>>>>>>>>>>>>> (less GC).
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> What do you think about it?
> >> >>>>>>>>>>>>>>>>>>> Does it make any sense to
> >> >>>>>>>>>>>>>>>>>>> you?
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> Ciao, R
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>> On Sat, Sep 29, 2012 at 2:40
> >> >>>>>>>>>>>>>>>>>>> PM, Noctarius
> >> >>>>>>>>>>>>>>>>>>> <[email protected]>
> >> >>>>>>>>>>>>>> wrote:
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Hey guys,
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Raffaele found out about a
> >> >>>>>>>>>>>>>>>>>>>> project of mine (Lightning)
> >> >>>>>>>>>>>>>>>>>>>> a few
> >> >>>>>>>>>> weeks
> >> >>>>>>>>>>>>>>>>>>>> ago. Lightning is a heavy
> >> >>>>>>>>>>>>>>>>>>>> Unsafe and Bytecode
> >> >>>>>>>>>>>>>>>>>>>> generation using
> >> >>>>>>>>>>>>>>>>>>>> Serializer implementation.
> >> >>>>>>>>>>>>>>>>>>>> He told me that he was
> >> >>>>>>>>>>>>>>>>>>>> interested in adding
> >> >>>>>>>>>>>>>>>>>>>> something similar to
> >> >>>>>>>>>>>>>>>>>>>> DirectMemory and I would be
> >> >>>>>>>>>>>>>>>>>>>> glad to
> >> >>>>>>>>>>>>>> help
> >> >>>>>>>>>>>>>>>>>>>> out in this.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Another project I started a
> >> >>>>>>>>>>>>>>>>>>>> few days ago, since it was
> >> >>>>>>>>>>>>>>>>>>>> needed
> >> >>>>>>>>>> for
> >> >>>>>>>>>>>>>>>>>>>> work is DirectRingCache.
> >> >>>>>>>>>>>>>>>>>>>> The name not really meets
> >> >>>>>>>>>>>>>>>>>>>> to actual implementation
> >> >>>>>>>>>>>>>>>>>>>> since it's not yet a ring
> >> >>>>>>>>>>>>>>>>>>>> buffer using cache. I
> >> >>>>>>>>>>>>>> used
> >> >>>>>>>>>>>>>>>>>>>> this for a
> >> >>>>>>>>>>>>>>>>>>>> pre-serialization simple
> >> >>>>>>>>>>>>>>>>>>>> bytestream cache with
> >> >>>>>>>>>>>>>>>>>>>> self-growing buffers. It
> >> >>>>>>>>>>>>>>>>>>>> could be nice to have
> >> >>>>>>>>>>>>>>>>>>>> DirectMemory
> >> >>>>>>>>>> having
> >> >>>>>>>>>>>>>>>>>>>> raw "buffers" to write to
> >> >>>>>>>>>>>>>>>>>>>> or to read from.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Here are the links from
> >> >>>>>>>>>>>>>>>>>>>> the projects:
> >> >>>>>>>>>>>>>>>>>>>> https://github.com/noctarius/Lightning
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> > https://github.com/noctarius/direct-ring-cache
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>> It would be nice to help out since I really like
> >> >>>>>>>> the idea of
> >> >>>>>>>>>>>>>>>>>>>> DirectMemory and since
> >> >>>>>>>>>>>>>>>>>>>> direct-ring-cache is some
> >> >>>>>>>>>>>>>>>>>>>> kind of
> >> >>>>>>>>>>>>>> reinventing
> >> >>>>>>>>>>>>>>>>>>>> the wheel.
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>> Cheers Noctarius (Chris)
> >> >>>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >> >>>>>>>>>>>>>>>> http://coders.talend.com
> >> >>>>>>>>>>>>>>>> http://twitter.com/olamy |
> >> >>>>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >> >>>>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>> -- Olivier Lamy Talend:
> >> >>>>>>>>>>>>>> http://coders.talend.com
> >> >>>>>>>>>>>>>> http://twitter.com/olamy |
> >> >>>>>>>>>>>>>> http://linkedin.com/in/olamy
> >> >>>>>>>>>>>>>>
> >> >>>>>>>>>>>>>
> >> >>>>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>>
> >> >>>>>>>>>> -- Olivier Lamy Talend:
> >> >>>>>>>>>> http://coders.talend.com
> >> >>>>>>>>>> http://twitter.com/olamy |
> >> >>>>>>>>>> http://linkedin.com/in/olamy
> >> >>>>>>>>>>
> >> >>>>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> -- Olivier Lamy Talend: http://coders.talend.com
> >> >>>>> http://twitter.com/olamy | http://linkedin.com/in/olamy
> >> >>>>>
> >> >>>>
> >> >>>
> >> >>
> >> >
> >>
>
