[go-nuts] Heap fragmentation due to protobuffs serialization/de-serialization

[Dave Cheney]

It looks ok to me, the svg.released number is growing which suggests that 
fragmentation is not an issue. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Heap fragmentation due to protobuffs serialization/de-serialization

[Ian Lance Taylor]

On Mon, Nov 28, 2016 at 8:00 AM, Devanand Reddy
 wrote:
>
> We are using protocol buffers extensively in our application. We have a
> suspect that over the time, due to excessive  marshaling/un-marshaling these
> two operations are causing heap fragmentation. One factor I am looking at,
> to come to this conclusion is HeapIdle. In my tests, though HeapInUse is
> coming back to my expectation, HeapIdle is staying there forever. On another
> note, our protobuff operations consume any where between 700-900mb, given
> the size of the data we are trying to serialize.
> I am attaching a few graphs here.
>
> Please advise whether this is a sign for concern.

Heap fragmentation is not normally a concern with the current Go
runtime.  The heap is implemented to that large pages are divided up
into blocks that are all the same size.  This means that the usual
definition of fragmentation--a small block prevents coalescing of
large blocks--can not occur.

Your graphs seem to show your memory use is in a steady state, which
is a good thing.  When there are operations that temporarily use a lot
of memory, it's normal for HeapIdle to be high--the memory is there
ready and waiting for temporary memory use spikes.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.