Am 21.07.2013 13:39, schrieb Christoph Engelbert: > Hey guys > What I forgot to mention is that we definately have to explain the > parameters when creating the MemoryManager since for the new buffer > backend they are different to the original ones. I use the > concurrencylevel which is already used by guava to decide on the > partition count and use the buffercount as the count of slices > inside of an partition. > > A bit more on the internals of the new buffer framework: > > There are a number of partitions which are sliced into multiple > fragments, all of the same size (a divisor of the partition size). > Partition 1 (fragment size for example 128 bytes) > +------------------------------------------ ----------+ > | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | > +-----------------------------------------------------+ > | ... | ... | ... | ... | ... | ... | ... | ... | ... | > +-----------------------------------------------------+ > | ... | ... | ... | ... | ... | ... | ... | ... | ... | > +-----------------------------------------------------+ > > Partition 2 > +------------------------------------------ ----------+ > | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | > +-----------------------------------------------------+ > | ... | ... | ... | ... | ... | ... | ... | ... | ... | > +-----------------------------------------------------+ > | ... | ... | ... | ... | ... | ... | ... | ... | ... | > +-----------------------------------------------------+ > > If you request a new PartitionBuffer you have the option to give it > a base size, if not given a single fragment will be requested. > > PartitionBufferPool.getPartitionBuffer() => ParitionBuffer1 [[P1,1]] > PartitionBufferPool.getPartitionBuffer(256) => ParitionBuffer2 > [[P2,1], [P2,2]] > > All PartitionBuffers are auto growing as long as there are free > fragments available so if you try to write beyond the current max > capacity in behind the buffer requests a new fragment to write to: > > PartitionBuffer1.writeByte(...) -> [[P1,1], [P1,1]] Oops should be:
PartitionBuffer1.writeByte(...) -> [[P1,1], [P1,2]] > > Fragments don't to be received from the same partition, this heavily > depends on the choosing selection strategy (like cpu local, thread > local, round robin) and if the current partition has some more free > fragments. Depending on the selection strategy different best > practices for partition count are available. > > What is important to note for the new buffer backend, the memory > usage is higher for small pieces of data or datasize little bit > bigger than multiple of fragment size. So in example you'll waste > lots of memory if you would store values of 129 bytes (2 fragments > with second fragment only used for one byte). This is a part that is > best for user to configure correctly but the most important part is, > this is totally different behavior than for old MemoryManagers which > sliced to the correct size. > > I guess that's it so far, any questions? I may would appreciate a > hangout or something like that to do some further explanation and to > get opinions, ideas, review. > > Chris > > Am 17.07.2013 19:29, schrieb Raffaele P. Guidi: >> Great ! :) >> Il giorno 17/lug/2013 12:41, "Christoph Engelbert" <[email protected]> >> ha scritto:
