Thanks for your help on this! I really appreciate it. I've been pulling my hair out for weeks on this issue, so I truly appreciate any help you can offer.
The way this runs in my production app is similar to how it works in the example app I provided. A request is sent to a web server and then up to 1000+ http requests are made. Data is pulled, parsed, and ultimately added to a database. Each of those requests are done in goroutines and each request finishes. But, the memory is never cleaned up. That's why I created this the way that I did. I'm aware that if the goroutines never finish or if I create them forever, it'll grow infinitely, but that's not what I'm doing. Requests come in and are processed and completed, but they're never cleaned up. As I just mentioned in the Github issue you referenced, it looks like this example isn't actually breaking in Linux after all so I'll put together one that does break on Linux too. Blessings, Rob Archibald CTO, EndFirst LLC r...@robarchibald.com -----Original Message----- From: Ian Lance Taylor [mailto:i...@golang.org] Sent: Wednesday, October 25, 2017 11:03 AM To: Rob Archibald Cc: golang-nuts Subject: Re: [go-nuts] Memory leak when calling highly parallel goroutines On Wed, Oct 25, 2017 at 10:21 AM, Rob Archibald <r...@robarchibald.com> wrote: > > It is a memory leak because the memory continues to grow for as long > as the loop continues to run. If you change the code example I gave to > an infinite loop you'll see that it grows infinitely. I had it stop at > 1000 for demonstration purposes because each scan request that my > production app gets typically kicks off about 1000 goroutines. > > I posted to the issue you linked to as well. Thanks for letting me > know about that. > > I'm measuring memory from the system perspective. Windows and Linux > both report gigs of memory used on my production app before it crashes > due to out of memory. This is a show-stopper issue for me. If Go can't > release memory from a simple goroutine when it exits, I'll have to > rewrite using something else. Sorry, I didn't mean to suggest that you should add your program to that issue. I only meant to point to that issue as describing a known problem with programs that start a very large number of goroutines and then go back to just using a small number of goroutines. If your program continues to use an increasing amount of memory over time, then it is a different problem. When you say "measuring memory from the system perspective," what precisely do you mean? It really matters, as different system measurements report different things, and none of them really correspond to how Go manages its heap. Or, to put it another way, what is the real problem? I tried running your program. For me the virtual memory size as measured by the ps program goes up to 42176 (42M) and then stops. This is on GNU/Linux, though, not Windows. You say that you are describing a memory leak because memory continues to grow for as long as the loop continues to run. Since the loop is starting new goroutines more quickly than they can finish, that is not surprising. If you want to control memory usage with a loop like that, you need to limit the number of goroutines you start in parallel. Ian -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
