Re: [go-nuts] Memory issues when spawning 1 million go routines that communicate on a channel

[Michael Jones]

ok.

the short answer is that if you make a few million somethings --
goroutines, channels, whatever -- and they all wait for the first event in
order to start, then what is certain is that you'll attain the peak memory
usage needed to hold them all at once.

it means the OS will have surrendered that much memory before anything
interesting happens. then, once the addition chain is underway, the old
routines will be rendered moot one by one and your question becomes
four-fold: 1a), when will the Go runtime notice the stale memory, 1b) when
will it return that to the OS, and, 2a) when will the OS do something about
that released memory region, and, 2b) when will that action be visible in
memory statistics.

it seems that there can be some delay for 2b in any case. Difference in
peak memory use between my version and the original is a measure of 1a and
1b.

On Wed, Oct 4, 2017 at 10:15 AM, Unni Krishnan  wrote:

> That's an interesting solution, but what I'm trying to figure out is why
> the memory usage is this high and understand how go handles this.
>
> On Tue, Oct 3, 2017 at 9:16 PM Michael Jones 
> wrote:
>
>> Or you could do it this way...
>>
>> https://play.golang.org/p/rU2ONi51ec
>>
>> On Mon, Oct 2, 2017 at 7:26 PM, Unni Krishnan  wrote:
>>
>>> I tried FreeOSMemory, didn't make any difference.
>>>
>>
>>> On Mon, 2 Oct 2017, 5:26 a.m. Ian Lance Taylor,  wrote:
>>>
 On Sat, Sep 30, 2017 at 9:29 AM,   wrote:
 >
 > From what I understand reading and a few comments from the gopher
 slack,
 > this is because  go is not releasing memory back to OS but holds it
 for a
 > longer in case it needs this, if that makes sense? It would be really
 > helpful, if someone could help me understand this.

 Yes, that is what the Go runtime does.  It requests the memory it
 needs from the operating system.  Every five minutes or so it returns
 unused memory back to the operating system.  You can make this happen
 sooner by calling https://golang.org/pkg/runtime/debug/#FreeOSMemory .

 Note that on modern operating systems unused memory is quite cheap.

 Ian

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>> Michael T. Jones
>> michael.jo...@gmail.com
>>
>


-- 
Michael T. Jones
michael.jo...@gmail.com

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Memory issues when spawning 1 million go routines that communicate on a channel

[Unni Krishnan]

That's an interesting solution, but what I'm trying to figure out is why
the memory usage is this high and understand how go handles this.

On Tue, Oct 3, 2017 at 9:16 PM Michael Jones 
wrote:

> Or you could do it this way...
>
> https://play.golang.org/p/rU2ONi51ec
>
> On Mon, Oct 2, 2017 at 7:26 PM, Unni Krishnan  wrote:
>
>> I tried FreeOSMemory, didn't make any difference.
>>
>
>> On Mon, 2 Oct 2017, 5:26 a.m. Ian Lance Taylor,  wrote:
>>
>>> On Sat, Sep 30, 2017 at 9:29 AM,   wrote:
>>> >
>>> > From what I understand reading and a few comments from the gopher
>>> slack,
>>> > this is because  go is not releasing memory back to OS but holds it
>>> for a
>>> > longer in case it needs this, if that makes sense? It would be really
>>> > helpful, if someone could help me understand this.
>>>
>>> Yes, that is what the Go runtime does.  It requests the memory it
>>> needs from the operating system.  Every five minutes or so it returns
>>> unused memory back to the operating system.  You can make this happen
>>> sooner by calling https://golang.org/pkg/runtime/debug/#FreeOSMemory .
>>>
>>> Note that on modern operating systems unused memory is quite cheap.
>>>
>>> Ian
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Michael T. Jones
> michael.jo...@gmail.com
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Memory issues when spawning 1 million go routines that communicate on a channel

[Michael Jones]

Or you could do it this way...

https://play.golang.org/p/rU2ONi51ec

On Mon, Oct 2, 2017 at 7:26 PM, Unni Krishnan  wrote:

> I tried FreeOSMemory, didn't make any difference.
>
> On Mon, 2 Oct 2017, 5:26 a.m. Ian Lance Taylor,  wrote:
>
>> On Sat, Sep 30, 2017 at 9:29 AM,   wrote:
>> >
>> > From what I understand reading and a few comments from the gopher slack,
>> > this is because  go is not releasing memory back to OS but holds it for
>> a
>> > longer in case it needs this, if that makes sense? It would be really
>> > helpful, if someone could help me understand this.
>>
>> Yes, that is what the Go runtime does.  It requests the memory it
>> needs from the operating system.  Every five minutes or so it returns
>> unused memory back to the operating system.  You can make this happen
>> sooner by calling https://golang.org/pkg/runtime/debug/#FreeOSMemory .
>>
>> Note that on modern operating systems unused memory is quite cheap.
>>
>> Ian
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Michael T. Jones
michael.jo...@gmail.com

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Memory issues when spawning 1 million go routines that communicate on a channel

[Unni Krishnan]

I tried FreeOSMemory, didn't make any difference.

On Mon, 2 Oct 2017, 5:26 a.m. Ian Lance Taylor,  wrote:

> On Sat, Sep 30, 2017 at 9:29 AM,   wrote:
> >
> > From what I understand reading and a few comments from the gopher slack,
> > this is because  go is not releasing memory back to OS but holds it for a
> > longer in case it needs this, if that makes sense? It would be really
> > helpful, if someone could help me understand this.
>
> Yes, that is what the Go runtime does.  It requests the memory it
> needs from the operating system.  Every five minutes or so it returns
> unused memory back to the operating system.  You can make this happen
> sooner by calling https://golang.org/pkg/runtime/debug/#FreeOSMemory .
>
> Note that on modern operating systems unused memory is quite cheap.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Memory issues when spawning 1 million go routines that communicate on a channel

[Ian Lance Taylor]

On Sat, Sep 30, 2017 at 9:29 AM,   wrote:
>
> From what I understand reading and a few comments from the gopher slack,
> this is because  go is not releasing memory back to OS but holds it for a
> longer in case it needs this, if that makes sense? It would be really
> helpful, if someone could help me understand this.

Yes, that is what the Go runtime does.  It requests the memory it
needs from the operating system.  Every five minutes or so it returns
unused memory back to the operating system.  You can make this happen
sooner by calling https://golang.org/pkg/runtime/debug/#FreeOSMemory .

Note that on modern operating systems unused memory is quite cheap.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Memory issues when spawning 1 million go routines that communicate on a channel

[unni]

Hi,

I was trying to implement the following problem statement from the book 
*Programming 
Elixer* in go.

Let's write some code that creates n processes. The first will send a 
> number to the second. It will increment that number and pass it to the 
> third. This will continue until we get to the last process, which will pass 
> the number back to the top level


My implementation: https://play.golang.org/p/iI8GqQ08q6

While attempting to do this I realised once the process complete I see the 
application taking up a large chunk of memory. I was trying to understand 
why this was happening. Why is go still holding up memory after the 
computation is done?

memleak git:(master) ✗ GODEBUG=gctrace=1 ./memleak 100


Alloc = 60
TotalAlloc = 60
Sys = 1700
NumGC = 0


before: go routines 2
result: 100
after: go routines 2
Ctrl-C to exit
gc 1 @1.300s 4%: 40+96+0.10 ms clock, 322+77/89/123+0.81 ms cpu, 260->260->
170 MB, 261 MB goal, 8 P


Alloc = 184148
TotalAlloc = 260398
Sys = 1137084
NumGC = 1


GC forced
gc 2 @126.456s 0%: 0.012+101+0.049 ms clock, 0.10+0/96/118+0.39 ms cpu, 170
->170->170 MB, 340 MB goal, 8 P
scvg0: inuse: 180, idle: 882, sys: 1063, released: 0, consumed: 1063 (MB)
GC forced
gc 3 @246.577s 0%: 0.011+107+0.051 ms clock, 0.094+0/102/122+0.41 ms cpu, 
170->170->170 MB, 340 MB goal, 8 P
GC forced
scvg1: 882 MB released
scvg1: inuse: 180, idle: 882, sys: 1063, released: 882, consumed: 180 (MB)
gc 4 @366.708s 0%: 0.016+116+24 ms clock, 0.13+0/111/128+197 ms cpu, 170->
170->170 MB, 340 MB goal, 8 P
GC forced
gc 5 @486.868s 0%: 0.012+114+0.063 ms clock, 0.10+0/114/138+0.50 ms cpu, 170
->170->170 MB, 340 MB goal, 8 P
scvg2: 0 MB released
scvg2: inuse: 180, idle: 882, sys: 1063, released: 882, consumed: 180 (MB)
GC forced
gc 6 @607.003s 0%: 0.014+112+0.058 ms clock, 0.11+0/106/125+0.46 ms cpu, 170
->170->170 MB, 340 MB goal, 8 P
GC forced
scvg3: inuse: 180, idle: 882, sys: 1063, released: 882, consumed: 180 (MB)
gc 7 @727.144s 0%: 0.013+39+0.052 ms clock, 0.10+0/52/228+0.41 ms cpu, 170->
170->170 MB, 340 MB goal, 8 P
GC forced
gc 8 @847.203s 0%: 0.012+65+0.060 ms clock, 0.097+0/69/189+0.48 ms cpu, 170
->170->170 MB, 340 MB goal, 8 P
scvg4: inuse: 180, idle: 882, sys: 1063, released: 882, consumed: 180 (MB)


memleak git:(master) ✗ top -o mem

PIDCOMMAND  %CPU TIME #TH   #WQ  #PORTS MEMPURG   CMPRS 
 PGRP  PPID  STATEBOOSTS %CPU_ME %CPU_OTHRS UID  FAULTS 
COW  MSGSENT MSGRECV SYSBSD  SYSMACH
0  kernel_task  5.1  13:52:38 136/8 02  1501M- 0B 0B 0 
0 running   0[0]  0.0 0.00169958575 
  1476697193+ 1255953068+ 0   0
63272  memleak  0.0  00:05.91 11035 1163M  0B 0B 
63272 33103 sleeping *0[1]  0.0 0.0501  298605 
80   42  20  277052  320101



Also when I change up this implementation to do something different, while 
still spawning up 1 million goroutines and buffered channels this memory 
usage drops considerably. 
Changed code : https://play.golang.org/p/pJ2Y7ENXna

memleak git:(master) ✗ GODEBUG=gctrace=1 ./memleak 100


Alloc = 61
TotalAlloc = 61
Sys = 1700
NumGC = 0


before: go routines 2
gc 1 @0.000s 0%: 0.056+810+0.030 ms clock, 0.17+0.005/0.070/810+0.092 ms cpu
, 7->8->8 MB, 8 MB goal, 8 P
buffered: started all go goutines 2
buffered: final count 100
after: go routines 2


Alloc = 8198
TotalAlloc = 8205
Sys = 14434
NumGC = 1


Ctrl-C to exit

>From what I understand reading and a few comments from the gopher slack, 
this is because  go is not releasing memory back to OS but holds it for a 
longer in case it needs this, if that makes sense? It would be really 
helpful, if someone could help me understand this.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.