Thanks @Ian for the clarification.
The BySize arrays gives the number of active objects in a size class but it
does not give the total memory allocated for a size class. E.g, for 1k size
class, if 100K was the total size allocated and there are only 10 objects
in it, it does not tell the memory consumption of these 10 objects.
I partially got what I am looking from the core dump. Taking core dump of
the process and analysing it with viewcore shows the follows:
(viewcore) breakdown
all 11055939584 100.00%
text 28172288 0.25%
readonly 7422455808 67.14%
data 3137536 0.03%
bss 1497206784 13.54% (grab bag, includes OS thread
stacks, ...)
heap 2037923840 18.43%
in use spans 1340497920 12.12%
alloc 1306139856 11.81%
live 269335312 2.44%
garbage 1036804544 9.38%
free 19920896 0.18%
round 14437168 0.13%
manual spans 424214528 3.84% (Go stacks)
alloc 420808704 3.81%
free 3405824 0.03%
free spans 273211392 2.47%
retained 256065536 2.32% (kept for reuse by Go)
released 17145856 0.16% (given back to the OS)
ptr bitmap 65011712 0.59%
span table 2031616 0.02%
(viewcore)
Cross referencing the output with this
comment: https://github.com/golang/go/issues/32284#issuecomment-497851388,
alloc
-> heap -> in use spans seems to be HeapInUse. In the alloc of 1306139856
bytes, there seems to be 1036804544 garbage.
I have a follow-up question on this:
For the memory profile taken around the same time as core dump, the total
memory seems to be 688M
(pprof) top 10
Showing nodes accounting for 592.62MB, 86.06% of 688.58MB total
Dropped 178 nodes (cum <= 3.44MB)
alloc -> heap -> in use spans -> live only accounts for 269M. Is the rest
~420M in the profile coming from StackInUse (alloc -> heap -> manual spans)?
Is it always the case that StackInUse is shown in heap profile ?
Thanks,
Varun
On Wednesday, May 26, 2021 at 1:24:32 AM UTC+5:30 Ian Lance Taylor wrote:
> On Tue, May 25, 2021 at 6:53 AM varun...@gmail.com <varun...@gmail.com>
> wrote:
> >
> > This statement is a bit confusing to me regarding HeapInuse: "which
> includes space allocated to hold objects that do not yet exist or that have
> been released by the garbage collector"
> >
> > According to memstats documentation, HeapInUse -> Includes all spans
> that have at least one object in them.
> >
> > So, why should it include space for objects that do not exist any more
> or that has been released by garbage collector. Should it not go to
> HeapIdle?
>
> A typical span can contain a large number of objects. Each one of
> those objects may be allocated or not. If a span contains at least
> one allocated object, the total size of the span will be counted as
> part of HeapInUse.
>
>
> > Is it the case of internal fragmentation. For example, if a span is
> shared by 2 objects and one object is freed - then, the entire span is
> accounted under HeapInUse. But since there is free space in span, it can be
> used by new objects and it can not be returned to HeapIdle as there is one
> object in the span. When pprof runs, it only uses the live objects and
> therefore, it does not account for the free space in spans while HeapInUse
> will account for it.
>
> Yes.
>
>
> > If this is the case, then how can we know the percentage of utilisation
> across a size class.
>
> That seems like a different question. But you can find the answer to
> that question by looking at the BySize array.
>
> Ian
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/b24ef3b5-339a-45eb-8a71-c431823d49c6n%40googlegroups.com.
