so you're creating a bunch of stat watchers, those require gc to be cleaned up, but they're still too young by the time you run gc outside the loop, if instead you move the gc into the loop or call it multiple times after the loop is done it can effectively clean them up.
On Fri, Nov 22, 2013 at 11:58 AM, Timothy J Fontaine <[email protected]>wrote: > For what it's worth, I just went through an arduous process of tracking > down a memory leak for walmart, you can read about it > http://www.joyent.com/blog/walmart-node-js-memory-leak you could use the > techniques described there to help narrow down your issues. There are other > pieces of information available at > http://www.joyent.com/developers/node/debug#mdb > > I will try and do a quick analysis for you, but if you have issues you can > hit me and others up on irc #node.js on freenode > > > On Fri, Nov 22, 2013 at 9:27 AM, Steve Freegard > <[email protected]>wrote: > >> Hi Ben, >> >> I managed to find the issue in Haraka - we were calling >> fs.unwatchFile/fs.watchFile instead of consulting our configuration >> cache which we should have been doing. >> >> However - this has pointed us to a rather large memory leak in the >> fs.unwatchFile and fs.watchFile functions. I've tested it on both 0.8 >> and 0.10. This is on 0.10: >> >> [root@mta41 ~]# node --expose_gc watchfile_leak.js >> { rss: 8478720, heapTotal: 6163968, heapUsed: 1918824 } >> { rss: 827813888, heapTotal: 13387520, heapUsed: 4990520 } >> >> Testcase is attached. >> >> Kind regards, >> Steve. >> >> >> On 22/11/13 16:25, Ben Noordhuis wrote: >> > On Fri, Nov 22, 2013 at 12:37 PM, Steve Freegard >> > <[email protected]> wrote: >> >> Hi, >> >> >> >> I'm trying to debug and understand an issue on several of my Haraka >> servers >> >> after a recent update. >> >> >> >> I'm seeing large amounts of memory being used and never returned back >> to the >> >> operating system causing the machine to swap and eventually the Linux >> OOM >> >> killer shooting the processes. >> >> >> >> I thought that there must be a memory leak somewhere - but I've used >> >> node-heapdump and taken multiple snapshots and compared them and >> whilst I'm >> >> seeing a small growth I'm not seeing anything major. However the >> heapdump >> >> reports the size at ~97Mb whereas the RSS of the process at the time >> of the >> >> dump was 1.1Gb which is what I really don't understand at all. >> >> >> >> Would such a disparity between heap and RSS indicate that an external >> C++ >> >> module that is being used is leaking memory?? If so, what is the best >> way >> >> of finding out which? >> >> >> >> I've tried the mtrace module - but it caused the process to completely >> hang >> >> after only a few minutes of running so I had to kill it. The output >> that I >> >> got from it is attached - I'm not really sure it tells me anything >> though. >> >> >> >> Any suggestions as to how to work out what is going on here and how to >> fix >> >> it would be gratefully received. >> >> >> >> Kind regards, >> >> Steve. >> > Tracking down memory leaks in garbage-collected environments is one of >> > the great unsolved problems of our generation. Okay, maybe not really >> > but there is no one-size-fits-all solution. Here are some things you >> > can try: >> > >> > * The mtrace log suggests you're using node.js v0.8. Consider >> upgrading. >> > >> > * Use Valgrind. Drawback: it's really, really slow and it uses a lot >> > of memory itself. >> > >> > * SystemTap. Write a small stap script that collects malloc/free/etc. >> > call graphs and aggregates them into histograms. Strong disparities >> > between calls to malloc() and free() generally show up quickly that >> > way. >> > >> > Apropos that heap snapshot, did you look at the self size or the >> > retained size? Buffer objects tend to have small self sizes (usually >> > on the order of a few hundred bytes) but can have massive retained >> > sizes. Note that Buffers in snapshots taken with node.js <= v0.8.11 >> > don't report the true retained size. >> > >> >> -- >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to [email protected] >> To unsubscribe from this group, send email to >> [email protected] >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> >> --- >> You received this message because you are subscribed to the Google Groups >> "nodejs" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
