Are you certain the number of Go routines is not increasing - each Go routine requires stack. See https://tpaschalis.github.io/goroutines-size/
> On Mar 24, 2022, at 3:18 PM, Shlomi Amit <shlo...@gmail.com> wrote: > > Hi. > > I’m trying to find a memory leak in my application. > I’ve added some runtime memory stats to the logs (Heap & Stack stats.. go > routine and heapObject counts). > I can see that from time to time m.StackInuse is increasing without any > obvious reason from the application perspective (At least not one that I can > find yet). > Here some additional runtime mem stats (Taken at the same time): > StackInUse 56MB > HeapSys: 147MB. > HeapAlloc: 97MB > > While StackInUse and HeapSys seems to increase from time to time, HeapAlloc > stays about the same during 6 hours while the application is running. > There seems to be a bit of correlation between StackInUse and HeapSys, but > overtime StackInUse increase more than HeapSys. (Is HeapSize includes > StackInUse?) > > In addition to adding runtime memory logs, I’m also creating periodic heap > profile dumps. > My problem is that when analyzing the heap with pprof, it gives me no clue > why StackInUse is so high. > The pprof inuse_space shows: > “Showing nodes accounting for 55.31MB, 93.25% of 59.31MB total” > What this total MB represent? It doesn’t seem to match HeapAlloc, HeapSys or > StackInUse. > Does pprof heap profile even include StackInUse? > > I really need to understand where the leak is coming from, but after looking > in many places, the memory stats are still not clear to me, and neither what > memory stat pprof heap profile really represent. > Note that I’m also logging HeapObjects count and I don’t see any leak there… > It’s just StackInUse increasing from time to time (As it seems, it's always > double itself... ~8MB->16MB->32), and HeapSys. > > Note that I’m also using Cgo, but my understanding is Cgo memory allocations > will not be reflected by the runtime memory stats. Is this correct and I > assume if runtime memory stats are increasing this is defiantly because of go > code and not C code? > I hope I was clear, but added a screenshot for the different memory stats. > Marked in red the point in times that stackInUse increase. (My current > understanding which might be wrong, is that stackInUse is not included in > HeapSys, this is why they are stacked in the graph). > > I know my write is a bit messed up and you might not really be sure what's > being asked, so in if I'll try to summarize: > > > What stackInUse represents? Is it part of HeapSys? > What HeapSys represents? (Both it and StackInUse are way more high than > heapAlloc) > Why pprof inuse_space doesn't seem to have any notion of stuff which was > allocated by StackInUse or HeapSys? It seems to pretty much corelate with > HeapAlloc. > Since I'm also using Cgo, I do understand C memory allocation will not be > reflected in pprof, perhaps they are being reflected by StackInUse or > HeapInUse? > Considering StackInUse seems to double itself in size when it increase, it > seems like some kind of memory pre-allocation to have enough headroom than > needed, not something which I expect to happen by C code. > Most importantly, with the below memory stats being increased, what's the > approach to investigate? > Currently I'm using docker, and sometimes get OOM. What's best memory stats > which will the most close to describe container RSS memory in use? > (Specifically, my docker container was limited to 300MiB) > > I know these are many question, but I did try and go over the runtime memory > stats documentation, and I'm still very confused, especially not after the > actual memory stats I'm seeing. > BTW, I'm using go 1.17.5 > > <Untitled.jpg> > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com > <mailto:golang-nuts+unsubscr...@googlegroups.com>. > To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com > > <https://groups.google.com/d/msgid/golang-nuts/e7bedd37-76a2-48e9-82bb-6309c60787bbn%40googlegroups.com?utm_medium=email&utm_source=footer>. > <Untitled.jpg> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/954083DF-A505-4769-A5B4-953E58E7AD55%40ix.netcom.com.
