Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-05-07 Thread Urjit Singh Bhatia
Hi Naveen, Your expectations about the program immediately giving up memory on deleting an object are wrong. If there is a need for you to have very tight memory controls, you could look into turning GC off entirely and managing memory yourself - See

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-05-07 Thread Ian Lance Taylor
On Thu, May 7, 2020 at 11:57 AM Naveen Kak wrote: > > Ian, > Any thoughts on this? Appreciate a response. I'm sorry, I'm not sure what you want a response on. Using top is a fine way to see total memory usage of your application. It's a terrible way to examine the Go garbage collector. Using

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-05-07 Thread Naveen Kak
Ian, Any thoughts on this? Appreciate a response. Thanks Naveen On Tue, 5 May, 2020, 8:34 PM Naveen Kak, wrote: > Hi Ian, > I explored a few things, calling debug.FreeOSMemory periodically. This > does help, I see a definitely a change in the memory being returned to the > OS ( looking at top

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-05-05 Thread Naveen Kak
Hi Ian, I explored a few things, calling debug.FreeOSMemory periodically. This does help, I see a definitely a change in the memory being returned to the OS ( looking at top o/p). I also set the "GODEBUG=madvdonotneed=1", as per go documentation post 1.12 release, go release it uses "madvfree"

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-04-28 Thread Ian Lance Taylor
On Tue, Apr 28, 2020 at 1:00 PM Naveen Kak wrote: > Basically using the Top command at end of test. > The top command will show you the memory that the program has requested from the operating system and has not returned to the operating system. The Go memory allocator works by requesting

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-04-28 Thread Robert Engels
Also, it may just be that the runtime is better off allocating more and not doing a GC based on available memory and CPU usage. The “max heap” feature in development may help here. > On Apr 28, 2020, at 3:18 PM, 'Kevin Chowski' via golang-nuts > wrote: > >  > Guessing based on your latest

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-04-28 Thread 'Kevin Chowski' via golang-nuts
Guessing based on your latest description: are you aware that there is no partial slice collection in GC? That is: > bigSlice := make([]int, 1000*1000) subSlice := bigSlice[0:1:1] bigSlice = nil runtime.GC() // At this point, bigSlice is still allocated! It cannot be freed by the GC >

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-04-28 Thread Naveen Kak
Basically using the Top command at end of test. Let me give a quick summary of the scenario. Basically we have an application which keeps getting data on a TCP connection, we allocate a global slice to temporarily hold the data and then store in nested maps. For every TCP data transaction, we

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-04-27 Thread Ian Lance Taylor
On Sun, Apr 26, 2020 at 9:57 PM Naveen Kak wrote: > I have my system up and running for let's say 10 hours or so where these > operations on map ( continuous add/delete) keep happening. Memory keeps > growing and goes very high. > Eventually at end of test we clear all map entries, memory is

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-04-27 Thread Jake Montgomery
Perhaps a small program that reproduces the issue would help. I wrote a quick one: https://play.golang.org/p/hbnuZsEqUKV . However, after an hour of running I see no "leak". I am on "go1.14 windows/amd64". After 6,000,000 iterations and 4 TB total allocated The memory stats (HeapInuse and

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-04-26 Thread Naveen Kak
I have my system up and running for let's say 10 hours or so where these operations on map ( continuous add/delete) keep happening. Memory keeps growing and goes very high. Eventually at end of test we clear all map entries, memory is reclaimed and is good. It's basically kind of load test.

Re: [go-nuts] Deleting map entry from the top level of a nested map doesn't clean the underlying memory

2020-04-26 Thread Ian Lance Taylor
On Sun, Apr 26, 2020 at 2:48 PM wrote: > https://play.golang.org/p/e22ufH-T2M1 > > This is my sample data structure. > > package main > > import ( > "fmt" > ) > > type MicroChkpt struct { > comprtype uint32 > MicroChkptInfoMap map[uint32][]byte > } > > type CallChkpt struct { > FullChkptData