Re: [go-nuts] Re: godoc memory leak

[e]

wrt tracing out leaks, pprof is insanely powerful and relatively easy to add in.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e1388566-c451-468b-b5c9-799274f20de9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: godoc memory leak

[Tom Mitchell]

Thanks!

On Thu, Jun 20, 2019 at 11:23 AM Christopher Dang <
christopher.d...@wework.com> wrote:

> When I ran the godoc service in a docker image with less memory (6 GiB)
> the behavior was the same. For both a 8 GiB and 6 GiB machine, the godoc
> process continue to consume memory until it surpasses the memory on the
> machine and crashes. After the crash, the other processes on my machine
> continue to work and the memory is freed back entirely to the machine.
>
> On Thu, Jun 20, 2019 at 11:10 AM Tom Mitchell  wrote:
>
>>
>> On Mon, Jun 17, 2019 at 9:40 PM Agniva De Sarker <
>> agniva.quicksil...@gmail.com> wrote:
>>
>>> Couple of questions:
>>>
>>> 1. What version of godoc and Go are you using ? What is your go env ?
>>>
 One other question ...
>> If the OS supports limits does godoc fail nicely with smaller limits.
>> I am reminded of student code that would malloc(to-infinity-and-beyond)
>> because
>> student accounts had user limits and often began with a malloc() loop to
>> get
>> the largest amount possible to ease programming and speed things up.
>>
>>
>>
>> --
>>T o mM i t c h e l l
>>
>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "golang-nuts" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/golang-nuts/nU706aM7QpM/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/CAAMy4UT4EUDviAgcw0is2fsXzUfp1Xvjc5cPW-_UThkxivdEDg%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
   T o mM i t c h e l l

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAAMy4UR8%2B-1NJ9kyaHegGgN_%2Bg1JW8B-cswgae1aBB%2BpZwz-iQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: godoc memory leak

['Christopher Dang' via golang-nuts]

When I ran the godoc service in a docker image with less memory (6 GiB) the
behavior was the same. For both a 8 GiB and 6 GiB machine, the godoc
process continue to consume memory until it surpasses the memory on the
machine and crashes. After the crash, the other processes on my machine
continue to work and the memory is freed back entirely to the machine.

On Thu, Jun 20, 2019 at 11:10 AM Tom Mitchell  wrote:

>
> On Mon, Jun 17, 2019 at 9:40 PM Agniva De Sarker <
> agniva.quicksil...@gmail.com> wrote:
>
>> Couple of questions:
>>
>> 1. What version of godoc and Go are you using ? What is your go env ?
>>
>>> One other question ...
> If the OS supports limits does godoc fail nicely with smaller limits.
> I am reminded of student code that would malloc(to-infinity-and-beyond)
> because
> student accounts had user limits and often began with a malloc() loop to
> get
> the largest amount possible to ease programming and speed things up.
>
>
>
> --
>T o mM i t c h e l l
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/nU706aM7QpM/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAAMy4UT4EUDviAgcw0is2fsXzUfp1Xvjc5cPW-_UThkxivdEDg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAM8TFU7ROZ8sV2-9OcM0jM%3DYV6DawYN4A3ahj6NhxEErvxKfWg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: godoc memory leak

[Tom Mitchell]

On Mon, Jun 17, 2019 at 9:40 PM Agniva De Sarker <
agniva.quicksil...@gmail.com> wrote:

> Couple of questions:
>
> 1. What version of godoc and Go are you using ? What is your go env ?
>
>> One other question ...
If the OS supports limits does godoc fail nicely with smaller limits.
I am reminded of student code that would malloc(to-infinity-and-beyond)
because
student accounts had user limits and often began with a malloc() loop to
get
the largest amount possible to ease programming and speed things up.



-- 
   T o mM i t c h e l l

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAAMy4UT4EUDviAgcw0is2fsXzUfp1Xvjc5cPW-_UThkxivdEDg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: godoc memory leak

['Christopher Dang' via golang-nuts]

Thanks for the feedback. For anyone experiencing this issue you can follow
it here: https://github.com/golang/go/issues/32692

On Tue, Jun 18, 2019 at 9:34 PM Agniva De Sarker <
agniva.quicksil...@gmail.com> wrote:

> I can see that files are getting added. Which means index size is expected
> to increase. But it seems sometimes it is increasing even when there is no
> change.
>
> Could you please file an issue so that folks can investigate this ? Thanks.
>
> On Tue, Jun 18, 2019, 10:17 PM Christopher Dang <
> christopher.d...@wework.com> wrote:
>
>> Hello Agniva,
>>
>> Here is the information you've asked for:
>>
>> go version: 1.12.0
>> godoc version: latest
>>
>> I don't read from an index file because I need live updating from godoc
>> and boot time isn't a high priority. This is because the service I'm
>> creating can expect many files to be added, removed, or updated while godoc
>> is running. While it would be convenient to restart godoc between file
>> system changes my superiors would prefer I not use this option.
>>
>> I've attached a log dump from godoc -v below.
>>
>> I'm running the godoc process from within a docker so I use three methods
>> to measure the memory usage:
>> 1.) I docker exec into the container and run *top. *The *VSZ% *is the
>> stat I use to measure memory from inside the docker.
>> 2.) I run *docker stats* on the container from my host process
>> 3.) since I run godoc with the verbose flag I get logs that tell me how
>> many bytes are consumed by godoc. Example:
>> 2019/06/17 23:28:26 index updated (88.264700559s, 211342912 bytes of
>> source, 17421 files, 6533095 lines, 173980 unique words, 8997500 spots)
>> 2019/06/17 23:28:26 before GC: bytes = 2385978984 *footprint* =
>> 6429047608
>> 2019/06/17 23:28:26 after  GC: bytes = 614368968* footprint *= 6429047608
>>
>> The charts and excel sheet above get their data points from godoc's
>> footprint logs. I double check every now and then that footprint's output
>> is accurate by comparing it against *top* and *docker stats. *
>>
>> The docker image is golang:1.12.0-alpine3.9
>>
>> On Mon, Jun 17, 2019 at 9:40 PM Agniva De Sarker <
>> agniva.quicksil...@gmail.com> wrote:
>>
>>> Couple of questions:
>>>
>>> 1. What version of godoc and Go are you using ? What is your go env ?
>>>
>>> 2. You are using -index but not passing an index file. Any reason for
>>> that ? godoc will load faster if you write an index beforehand and pass
>>> that.
>>>
>>> 3. I am guessing since you have set index_interval, you expect files to
>>> be added in your GOPATH. How many new files are getting added while godoc
>>> is running ?
>>>
>>> 4. Please show us the output by adding -v flag.
>>>
>>>
>>>
>>> On Tuesday, 18 June 2019 05:28:39 UTC+5:30, christo...@wework.com wrote:

 Hi all,


 I've been playing around with an internal godoc server and noticed that
 over long periods of time the memory growth is unbounded. The command I use
 to invoke godoc is *godoc -index=true -index_interval=15m
 -index_throttle=.30 -maxresults=0*. The following images below track
 the memory usage and change in memory usage over a 24 hour period on a 8
 GiB machine. I've also attached a pdf with the data points I gathered
 during the experiment. Notice how the godoc process consumes 76% of memory
 by the end of the experiment. Is this indicative of a memory leak in the
 godoc source code or is this expected behavior?


 [image: Δ Mem Usage %.png][image: Mem Usage %.png]


 --
>>> You received this message because you are subscribed to a topic in the
>>> Google Groups "golang-nuts" group.
>>> To unsubscribe from this topic, visit
>>> https://groups.google.com/d/topic/golang-nuts/nU706aM7QpM/unsubscribe.
>>> To unsubscribe from this group and all its topics, send an email to
>>> golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/60b70a8c-3a3d-49c6-b9dc-b2bd1053c435%40googlegroups.com
>>> 
>>> .
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAM8TFU6s87RTws%2BxgMuLaoNmsNbbaCJT9GOC_fn1JDs7ahk%2Btw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: godoc memory leak

[Agniva De Sarker]

I can see that files are getting added. Which means index size is expected
to increase. But it seems sometimes it is increasing even when there is no
change.

Could you please file an issue so that folks can investigate this ? Thanks.

On Tue, Jun 18, 2019, 10:17 PM Christopher Dang 
wrote:

> Hello Agniva,
>
> Here is the information you've asked for:
>
> go version: 1.12.0
> godoc version: latest
>
> I don't read from an index file because I need live updating from godoc
> and boot time isn't a high priority. This is because the service I'm
> creating can expect many files to be added, removed, or updated while godoc
> is running. While it would be convenient to restart godoc between file
> system changes my superiors would prefer I not use this option.
>
> I've attached a log dump from godoc -v below.
>
> I'm running the godoc process from within a docker so I use three methods
> to measure the memory usage:
> 1.) I docker exec into the container and run *top. *The *VSZ% *is the
> stat I use to measure memory from inside the docker.
> 2.) I run *docker stats* on the container from my host process
> 3.) since I run godoc with the verbose flag I get logs that tell me how
> many bytes are consumed by godoc. Example:
> 2019/06/17 23:28:26 index updated (88.264700559s, 211342912 bytes of
> source, 17421 files, 6533095 lines, 173980 unique words, 8997500 spots)
> 2019/06/17 23:28:26 before GC: bytes = 2385978984 *footprint* = 6429047608
> 2019/06/17 23:28:26 after  GC: bytes = 614368968* footprint *= 6429047608
>
> The charts and excel sheet above get their data points from godoc's
> footprint logs. I double check every now and then that footprint's output
> is accurate by comparing it against *top* and *docker stats. *
>
> The docker image is golang:1.12.0-alpine3.9
>
> On Mon, Jun 17, 2019 at 9:40 PM Agniva De Sarker <
> agniva.quicksil...@gmail.com> wrote:
>
>> Couple of questions:
>>
>> 1. What version of godoc and Go are you using ? What is your go env ?
>>
>> 2. You are using -index but not passing an index file. Any reason for
>> that ? godoc will load faster if you write an index beforehand and pass
>> that.
>>
>> 3. I am guessing since you have set index_interval, you expect files to
>> be added in your GOPATH. How many new files are getting added while godoc
>> is running ?
>>
>> 4. Please show us the output by adding -v flag.
>>
>>
>>
>> On Tuesday, 18 June 2019 05:28:39 UTC+5:30, christo...@wework.com wrote:
>>>
>>> Hi all,
>>>
>>>
>>> I've been playing around with an internal godoc server and noticed that
>>> over long periods of time the memory growth is unbounded. The command I use
>>> to invoke godoc is *godoc -index=true -index_interval=15m
>>> -index_throttle=.30 -maxresults=0*. The following images below track
>>> the memory usage and change in memory usage over a 24 hour period on a 8
>>> GiB machine. I've also attached a pdf with the data points I gathered
>>> during the experiment. Notice how the godoc process consumes 76% of memory
>>> by the end of the experiment. Is this indicative of a memory leak in the
>>> godoc source code or is this expected behavior?
>>>
>>>
>>> [image: Δ Mem Usage %.png][image: Mem Usage %.png]
>>>
>>>
>>> --
>> You received this message because you are subscribed to a topic in the
>> Google Groups "golang-nuts" group.
>> To unsubscribe from this topic, visit
>> https://groups.google.com/d/topic/golang-nuts/nU706aM7QpM/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to
>> golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/60b70a8c-3a3d-49c6-b9dc-b2bd1053c435%40googlegroups.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOpCn4hCCSsnCq%3DG%3DAfAA4J7_TVjRo2rT3Z35YDG2VWXAsJVjg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: godoc memory leak

[Agniva De Sarker]

Couple of questions:

1. What version of godoc and Go are you using ? What is your go env ?

2. You are using -index but not passing an index file. Any reason for that 
? godoc will load faster if you write an index beforehand and pass that.

3. I am guessing since you have set index_interval, you expect files to be 
added in your GOPATH. How many new files are getting added while godoc is 
running ?

4. Please show us the output by adding -v flag.



On Tuesday, 18 June 2019 05:28:39 UTC+5:30, christo...@wework.com wrote:
>
> Hi all,
>
>
> I've been playing around with an internal godoc server and noticed that 
> over long periods of time the memory growth is unbounded. The command I use 
> to invoke godoc is *godoc -index=true -index_interval=15m 
> -index_throttle=.30 -maxresults=0*. The following images below track the 
> memory usage and change in memory usage over a 24 hour period on a 8 GiB 
> machine. I've also attached a pdf with the data points I gathered during 
> the experiment. Notice how the godoc process consumes 76% of memory by the 
> end of the experiment. Is this indicative of a memory leak in the godoc 
> source code or is this expected behavior?
>
>
> [image: Δ Mem Usage %.png][image: Mem Usage %.png]
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/60b70a8c-3a3d-49c6-b9dc-b2bd1053c435%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.