Hey guys What I forgot to mention is that we definately have to explain the parameters when creating the MemoryManager since for the new buffer backend they are different to the original ones. I use the concurrencylevel which is already used by guava to decide on the partition count and use the buffercount as the count of slices inside of an partition.
A bit more on the internals of the new buffer framework: There are a number of partitions which are sliced into multiple fragments, all of the same size (a divisor of the partition size). Partition 1 (fragment size for example 128 bytes) +------------------------------------------ ----------+ | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | +-----------------------------------------------------+ | ... | ... | ... | ... | ... | ... | ... | ... | ... | +-----------------------------------------------------+ | ... | ... | ... | ... | ... | ... | ... | ... | ... | +-----------------------------------------------------+ Partition 2 +------------------------------------------ ----------+ | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | +-----------------------------------------------------+ | ... | ... | ... | ... | ... | ... | ... | ... | ... | +-----------------------------------------------------+ | ... | ... | ... | ... | ... | ... | ... | ... | ... | +-----------------------------------------------------+ If you request a new PartitionBuffer you have the option to give it a base size, if not given a single fragment will be requested. PartitionBufferPool.getPartitionBuffer() => ParitionBuffer1 [[P1,1]] PartitionBufferPool.getPartitionBuffer(256) => ParitionBuffer2 [[P2,1], [P2,2]] All PartitionBuffers are auto growing as long as there are free fragments available so if you try to write beyond the current max capacity in behind the buffer requests a new fragment to write to: PartitionBuffer1.writeByte(...) -> [[P1,1], [P1,1]] Fragments don't to be received from the same partition, this heavily depends on the choosing selection strategy (like cpu local, thread local, round robin) and if the current partition has some more free fragments. Depending on the selection strategy different best practices for partition count are available. What is important to note for the new buffer backend, the memory usage is higher for small pieces of data or datasize little bit bigger than multiple of fragment size. So in example you'll waste lots of memory if you would store values of 129 bytes (2 fragments with second fragment only used for one byte). This is a part that is best for user to configure correctly but the most important part is, this is totally different behavior than for old MemoryManagers which sliced to the correct size. I guess that's it so far, any questions? I may would appreciate a hangout or something like that to do some further explanation and to get opinions, ideas, review. Chris Am 17.07.2013 19:29, schrieb Raffaele P. Guidi: > Great ! :) > Il giorno 17/lug/2013 12:41, "Christoph Engelbert" <[email protected]> > ha scritto:
