Re: [go-nuts] memory leak, heap size doesn't match process res size

[azhao155]

Have you figure the root cause?  We experience the same issue

On Thursday, February 15, 2018 at 7:21:12 AM UTC-8, Kane Kim wrote:
>
> I was surprised to see that overhead is so significant, is there any way 
> to peek into that and see what is taking up space in race detector? I've 
> tried --inuse_objects in a heap dump, but didn't see anything suspicious 
> there.
>
> On Thursday, February 15, 2018 at 7:16:42 AM UTC-8, Kane Kim wrote:
>>
>> I think I've got a clue. I've discovered that the process under question 
>> was built with -race flag. Apparently race detector structures (tsan) are 
>> not visible to runtime and are not reported.
>>
>> On Thursday, February 15, 2018 at 2:14:55 AM UTC-8, Peter Waller wrote:
>>>
>>> On 14 February 2018 at 16:15, Kane Kim  wrote:
>>>
 If we don't use CGO all memory should be reported somewhere?

>>>
>>> Well, assuming no part of your software calls mmap, or there isn't 
>>> something else funny going on.
>>>
>>> Can you capture when this large region is mapped with strace? At what 
>>> point in the process lifecycle does this happen? Perhaps you can use a 
>>> debugger (dlv) to catch the culprit in the act. Perhaps you could come up 
>>> with a mechanism to unmap the region and get a segfault and stacktrace when 
>>> the culprit tries to use it.
>>>
>>> If you find out what's going on please share if you can, it sounds like 
>>> an interesting issue.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1267ea72-6f03-46d4-b465-d8092ec233cc%40googlegroups.com.

Re: [go-nuts] memory leak, heap size doesn't match process res size

[Ian Lance Taylor]

On Thu, Feb 15, 2018 at 7:21 AM, Kane Kim  wrote:
>
> I was surprised to see that overhead is so significant, is there any way to
> peek into that and see what is taking up space in race detector? I've tried
> --inuse_objects in a heap dump, but didn't see anything suspicious there.

The race detector works by using shadow memory to track the status of
the real memory.  So in effect it multiplies the total amount of
possible memory usage by a constant multiplier (though I don't know
what that multiplier is offhand).  This shadow memory is not part of
the Go heap so the Go heap inspection tools will not help.

Ian


> On Thursday, February 15, 2018 at 7:16:42 AM UTC-8, Kane Kim wrote:
>>
>> I think I've got a clue. I've discovered that the process under question
>> was built with -race flag. Apparently race detector structures (tsan) are
>> not visible to runtime and are not reported.
>>
>> On Thursday, February 15, 2018 at 2:14:55 AM UTC-8, Peter Waller wrote:
>>>
>>> On 14 February 2018 at 16:15, Kane Kim  wrote:

 If we don't use CGO all memory should be reported somewhere?
>>>
>>>
>>> Well, assuming no part of your software calls mmap, or there isn't
>>> something else funny going on.
>>>
>>> Can you capture when this large region is mapped with strace? At what
>>> point in the process lifecycle does this happen? Perhaps you can use a
>>> debugger (dlv) to catch the culprit in the act. Perhaps you could come up
>>> with a mechanism to unmap the region and get a segfault and stacktrace when
>>> the culprit tries to use it.
>>>
>>> If you find out what's going on please share if you can, it sounds like
>>> an interesting issue.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] memory leak, heap size doesn't match process res size

[Kane Kim]

I was surprised to see that overhead is so significant, is there any way to 
peek into that and see what is taking up space in race detector? I've tried 
--inuse_objects in a heap dump, but didn't see anything suspicious there.

On Thursday, February 15, 2018 at 7:16:42 AM UTC-8, Kane Kim wrote:
>
> I think I've got a clue. I've discovered that the process under question 
> was built with -race flag. Apparently race detector structures (tsan) are 
> not visible to runtime and are not reported.
>
> On Thursday, February 15, 2018 at 2:14:55 AM UTC-8, Peter Waller wrote:
>>
>> On 14 February 2018 at 16:15, Kane Kim  wrote:
>>
>>> If we don't use CGO all memory should be reported somewhere?
>>>
>>
>> Well, assuming no part of your software calls mmap, or there isn't 
>> something else funny going on.
>>
>> Can you capture when this large region is mapped with strace? At what 
>> point in the process lifecycle does this happen? Perhaps you can use a 
>> debugger (dlv) to catch the culprit in the act. Perhaps you could come up 
>> with a mechanism to unmap the region and get a segfault and stacktrace when 
>> the culprit tries to use it.
>>
>> If you find out what's going on please share if you can, it sounds like 
>> an interesting issue.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] memory leak, heap size doesn't match process res size

[Kane Kim]

I think I've got a clue. I've discovered that the process under question 
was built with -race flag. Apparently race detector structures (tsan) are 
not visible to runtime and are not reported.

On Thursday, February 15, 2018 at 2:14:55 AM UTC-8, Peter Waller wrote:
>
> On 14 February 2018 at 16:15, Kane Kim > 
> wrote:
>
>> If we don't use CGO all memory should be reported somewhere?
>>
>
> Well, assuming no part of your software calls mmap, or there isn't 
> something else funny going on.
>
> Can you capture when this large region is mapped with strace? At what 
> point in the process lifecycle does this happen? Perhaps you can use a 
> debugger (dlv) to catch the culprit in the act. Perhaps you could come up 
> with a mechanism to unmap the region and get a segfault and stacktrace when 
> the culprit tries to use it.
>
> If you find out what's going on please share if you can, it sounds like an 
> interesting issue.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] memory leak, heap size doesn't match process res size

[Peter Waller]

On 14 February 2018 at 16:15, Kane Kim  wrote:

> If we don't use CGO all memory should be reported somewhere?
>

Well, assuming no part of your software calls mmap, or there isn't
something else funny going on.

Can you capture when this large region is mapped with strace? At what point
in the process lifecycle does this happen? Perhaps you can use a debugger
(dlv) to catch the culprit in the act. Perhaps you could come up with a
mechanism to unmap the region and get a segfault and stacktrace when the
culprit tries to use it.

If you find out what's going on please share if you can, it sounds like an
interesting issue.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] memory leak, heap size doesn't match process res size

[Kane Kim]

I've measured resident size multiple times, with top, /proc/pid/smem, pmap. 
There is a huge allocated region towards the end of virtual memory:
7fba77bb-7fbc3bf37000 rw-p  00:00 0
Size:7409180 kB
Rss: 7408840 kB
Pss: 7408840 kB
Shared_Clean:  0 kB
Shared_Dirty:  0 kB
Private_Clean: 0 kB
Private_Dirty:   7408840 kB
Referenced:  7179916 kB
Anonymous:   7408840 kB
AnonHugePages:243712 kB

I've tried to force GC through /pprof/debug/heap?gc=1, didn't have any 
effect (gc was called, but nothing freed).
This doesn't fit to my understanding as well. If we don't use CGO all 
memory should be reported somewhere?


On Wednesday, February 14, 2018 at 3:05:26 AM UTC-8, Peter Waller wrote:
>
> First, a sanity check: How are you measuring the resident set? Are you 
> certain the memstats data you presented are from the same run (and after) 
> you hit 10G resident?
>
> What you have described doesn't immediately fit in my mental model. Could 
> you be measuring the VM size rather than the resident size? The thing that 
> puzzles me a little is that your "Sys =" field is so much lower than 10G, 
> assuming the measurements were taken at the same time.
>
> Have you tried calling runtime.FreeOSMemory and playing with GOGC 
> (described in pkg/runtime  docs), do 
> those make any difference? You might also want to play with 
> GODEBUG=gctrace=1.
>
> On 13 February 2018 at 18:55, Kane Kim > 
> wrote:
>>
>> Both memory profile and heap debug show ~500MB memory is in use, idle 
>> heap is pretty low, given all that and lack of cgo, where rest 10.5G could 
>> go and what are tools to debug it?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] memory leak, heap size doesn't match process res size

[Peter Waller]

First, a sanity check: How are you measuring the resident set? Are you
certain the memstats data you presented are from the same run (and after)
you hit 10G resident?

What you have described doesn't immediately fit in my mental model. Could
you be measuring the VM size rather than the resident size? The thing that
puzzles me a little is that your "Sys =" field is so much lower than 10G,
assuming the measurements were taken at the same time.

Have you tried calling runtime.FreeOSMemory and playing with GOGC
(described in pkg/runtime  docs), do those
make any difference? You might also want to play with GODEBUG=gctrace=1.

On 13 February 2018 at 18:55, Kane Kim  wrote:
>
> Both memory profile and heap debug show ~500MB memory is in use, idle heap
> is pretty low, given all that and lack of cgo, where rest 10.5G could go
> and what are tools to debug it?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.