thanks a lot Christoph for your detailed explanation, from my point of view it'd be nice to have this behavior configurable (basically that would plug in and out the new buffer backed) so that users could choose in a simpler way, do you think that'd be possible ? Regarding having an hangout I don't think it'd be easy, probably it's better to keep sharing our concerns ideas here on the ML, however if you could make something like a documentation / 2 mins tutorial page that would help a lot I think.
I wonder also if it'd make sense to have a DM 0.2 release before committing the new backend in order to have that be part of a separate additional release. What do you think? Thanks a lot for your effort, Tommaso 2013/7/21 Christoph Engelbert <[email protected]> > Am 21.07.2013 13:39, schrieb Christoph Engelbert: > > Hey guys > > What I forgot to mention is that we definately have to explain the > > parameters when creating the MemoryManager since for the new buffer > > backend they are different to the original ones. I use the > > concurrencylevel which is already used by guava to decide on the > > partition count and use the buffercount as the count of slices > > inside of an partition. > > > > A bit more on the internals of the new buffer framework: > > > > There are a number of partitions which are sliced into multiple > > fragments, all of the same size (a divisor of the partition size). > > Partition 1 (fragment size for example 128 bytes) > > +------------------------------------------ ----------+ > > | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | > > +-----------------------------------------------------+ > > | ... | ... | ... | ... | ... | ... | ... | ... | ... | > > +-----------------------------------------------------+ > > | ... | ... | ... | ... | ... | ... | ... | ... | ... | > > +-----------------------------------------------------+ > > > > Partition 2 > > +------------------------------------------ ----------+ > > | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | > > +-----------------------------------------------------+ > > | ... | ... | ... | ... | ... | ... | ... | ... | ... | > > +-----------------------------------------------------+ > > | ... | ... | ... | ... | ... | ... | ... | ... | ... | > > +-----------------------------------------------------+ > > > > If you request a new PartitionBuffer you have the option to give it > > a base size, if not given a single fragment will be requested. > > > > PartitionBufferPool.getPartitionBuffer() => ParitionBuffer1 [[P1,1]] > > PartitionBufferPool.getPartitionBuffer(256) => ParitionBuffer2 > > [[P2,1], [P2,2]] > > > > All PartitionBuffers are auto growing as long as there are free > > fragments available so if you try to write beyond the current max > > capacity in behind the buffer requests a new fragment to write to: > > > > PartitionBuffer1.writeByte(...) -> [[P1,1], [P1,1]] > Oops should be: > > PartitionBuffer1.writeByte(...) -> [[P1,1], [P1,2]] > > > > > > Fragments don't to be received from the same partition, this heavily > > depends on the choosing selection strategy (like cpu local, thread > > local, round robin) and if the current partition has some more free > > fragments. Depending on the selection strategy different best > > practices for partition count are available. > > > > What is important to note for the new buffer backend, the memory > > usage is higher for small pieces of data or datasize little bit > > bigger than multiple of fragment size. So in example you'll waste > > lots of memory if you would store values of 129 bytes (2 fragments > > with second fragment only used for one byte). This is a part that is > > best for user to configure correctly but the most important part is, > > this is totally different behavior than for old MemoryManagers which > > sliced to the correct size. > > > > I guess that's it so far, any questions? I may would appreciate a > > hangout or something like that to do some further explanation and to > > get opinions, ideas, review. > > > > Chris > > > > Am 17.07.2013 19:29, schrieb Raffaele P. Guidi: > >> Great ! :) > >> Il giorno 17/lug/2013 12:41, "Christoph Engelbert" < > [email protected]> > >> ha scritto: > >
