Am 22.07.2013 11:32, schrieb Tommaso Teofili: > thanks a lot Christoph for your detailed explanation, from my point of view > it'd be nice to have this behavior configurable (basically that would plug > in and out the new buffer backed) so that users could choose in a simpler > way, do you think that'd be possible ?
It is exchangable. The old implementations without the buffer backend are still there and available that why I said we need to clarify that one implementation is behaving totally different :-) > Regarding having an hangout I don't think it'd be easy, probably it's > better to keep sharing our concerns ideas here on the ML, however if you > could make something like a documentation / 2 mins tutorial page that would > help a lot I think. I'll try to :-) > > I wonder also if it'd make sense to have a DM 0.2 release before committing > the new backend in order to have that be part of a separate additional > release. > What do you think? > > Thanks a lot for your effort, > Tommaso > > > 2013/7/21 Christoph Engelbert <[email protected]> > >> Am 21.07.2013 13:39, schrieb Christoph Engelbert: >>> Hey guys >>> What I forgot to mention is that we definately have to explain the >>> parameters when creating the MemoryManager since for the new buffer >>> backend they are different to the original ones. I use the >>> concurrencylevel which is already used by guava to decide on the >>> partition count and use the buffercount as the count of slices >>> inside of an partition. >>> >>> A bit more on the internals of the new buffer framework: >>> >>> There are a number of partitions which are sliced into multiple >>> fragments, all of the same size (a divisor of the partition size). >>> Partition 1 (fragment size for example 128 bytes) >>> +------------------------------------------ ----------+ >>> | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | >>> +-----------------------------------------------------+ >>> | ... | ... | ... | ... | ... | ... | ... | ... | ... | >>> +-----------------------------------------------------+ >>> | ... | ... | ... | ... | ... | ... | ... | ... | ... | >>> +-----------------------------------------------------+ >>> >>> Partition 2 >>> +------------------------------------------ ----------+ >>> | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | >>> +-----------------------------------------------------+ >>> | ... | ... | ... | ... | ... | ... | ... | ... | ... | >>> +-----------------------------------------------------+ >>> | ... | ... | ... | ... | ... | ... | ... | ... | ... | >>> +-----------------------------------------------------+ >>> >>> If you request a new PartitionBuffer you have the option to give it >>> a base size, if not given a single fragment will be requested. >>> >>> PartitionBufferPool.getPartitionBuffer() => ParitionBuffer1 [[P1,1]] >>> PartitionBufferPool.getPartitionBuffer(256) => ParitionBuffer2 >>> [[P2,1], [P2,2]] >>> >>> All PartitionBuffers are auto growing as long as there are free >>> fragments available so if you try to write beyond the current max >>> capacity in behind the buffer requests a new fragment to write to: >>> >>> PartitionBuffer1.writeByte(...) -> [[P1,1], [P1,1]] >> Oops should be: >> >> PartitionBuffer1.writeByte(...) -> [[P1,1], [P1,2]] >> >> >>> Fragments don't to be received from the same partition, this heavily >>> depends on the choosing selection strategy (like cpu local, thread >>> local, round robin) and if the current partition has some more free >>> fragments. Depending on the selection strategy different best >>> practices for partition count are available. >>> >>> What is important to note for the new buffer backend, the memory >>> usage is higher for small pieces of data or datasize little bit >>> bigger than multiple of fragment size. So in example you'll waste >>> lots of memory if you would store values of 129 bytes (2 fragments >>> with second fragment only used for one byte). This is a part that is >>> best for user to configure correctly but the most important part is, >>> this is totally different behavior than for old MemoryManagers which >>> sliced to the correct size. >>> >>> I guess that's it so far, any questions? I may would appreciate a >>> hangout or something like that to do some further explanation and to >>> get opinions, ideas, review. >>> >>> Chris >>> >>> Am 17.07.2013 19:29, schrieb Raffaele P. Guidi: >>>> Great ! :) >>>> Il giorno 17/lug/2013 12:41, "Christoph Engelbert" < >> [email protected]> >>>> ha scritto: >>
