I tried to reproduce the leak with a small test reconstructed from your test case, and I couldn't reproduce. Can you try to create a standalone test case that I can run locally?
It might also help to log the output of `node --trace-gc yourscript.js` to get an idea what the garbage collector is doing. > We create random ids for mysql row so that the request is non-blocking I'm not sure I understand what you mean, but nothing is magically becoming non-blocking here. If you're concerned about doing multiple concurrent INSERTs I'd suggest to use an AUTO_INCREMENT key. You can get the automatically created key with this<https://github.com/felixge/node-mysql#getting-the-id-of-an-inserted-row>. The current approach seems exceptionally brittle and non-scalable. There also are a few important issues with your code: - you're making synchronous callbacks which is entirely inappropriate here. I don't see why you'd want these functions to appear asynchronous, but you should at least use .nextTick() the callback. - recentRequestIds[table_name] is never initialized with an object. clearRecentRequestIds deletes the key instead of setting it to an empty object. - Bert On Friday, July 5, 2013 9:41:19 AM UTC+2, Sumit Agarwal wrote: > > We have 10 node.js http servers running on v0.10.12. The servers basically > act as a gateway where they pipe response of a http client request they > make, to the original http request. > > The code caches around 500k-600k 10 digit integer keys in a dictionary for > around 24 hours before releasing it. (notice the 2nd line of memoryUsage > where it frees that). However, after that point, the heapTotal starts > growing unboundedly even though headUsed does not grow, until the program > crashes. This has happened 5 days in a row but not on all servers. And its > pretty random about which server it happens to. > > Linux version 2.6.32-279.14.1.el6.x86_64 > ([email protected]<javascript:> > ) > Running the process with parameter --max_old_space_size=1700 or without > did not change anything. > > Here's the code snippet that caches the data. We create random ids for > mysql row so that the request is non-blocking, and we cache it to avoid > collisions. Also, we move to a new table everyday, so at midnight every day > the entire cache is deleted. > > //add to cache code > > var generateUniqueRandom = function() { > > var recentRequestIdsForTable = recentRequestIds[table_name]; // > recentRequestIds is a dictionary and recentRequestIdsForTable is also > dictionary keyed by table_name. The most filled recentRequestIdsForTable is > for the current day table and it goes up to 600k, expected to reach 1 > million per day soon. > > var attempts = 0; > > var random; > > while(attempts < 5) { > > random = Math.floor(Math.random() * (9999999999 - 1000000000) > + 1000000000); // 10 digit random id > > if (!(random in recentRequestIdsForTable)) { > > recentRequestIdsForTable[random] = true; > > return callback(null, table_name + ":" + random); > > } > > attempts++; > > } > > return callback("Could not generate requestID", null); > > }; > > //delete from cache code > > exports.clearRecentRequestIds = function(table_name) { > > if (table_name in recentRequestIds) { > > delete recentRequestIds[table_name]; // drop the entire cached > dictionary for the table_name. > > logger.info('cleared all recentRequestIds for table ' + > table_name); > > return; > > } > > logger.warn(table_name + ' not found in recentRequestIds'); > > } > > Here's the memory usage. It increased to the max in around 1.5 hours. > During that time, the server was functioning normally and had no CPU > spikes. It is weird that around 500k keys should mean up to around 5MB-10MB > of data if worst case keys are stored as string. However they take up space > over 100 MB. (Coz freeing up reduced memory from 217MB to 76MB). Maybe it > has something to do with that? > > {"rss":"368.0078125 MB","heapTotal":"291.7858581542969 > MB","heapUsed":"217.14698028564453 MB"} > {"rss":"357.16796875 MB","heapTotal":"287.84947204589844 > MB","heapUsed":"76.96298217773438 MB"} > {"rss":"475.1796875 MB","heapTotal":"409.88169860839844 > MB","heapUsed":"66.50176239013672 MB"} > {"rss":"577.9921875 MB","heapTotal":"514.1995697021484 > MB","heapUsed":"83.56008911132812 MB"} > > - and eventually to > > {"rss":"1463.27734375 MB","heapTotal":"1381.2594375610352 > MB","heapUsed":"95.49384307861328 MB"} > {"rss":"1500.05078125 MB","heapTotal":"1413.7357559204102 > MB","heapUsed":"93.21699523925781 MB"} > > Also, we did experience the issue in 0.10.10 which was busy loop in net > module that led to infinite memory increase and CPU spike a couple of times > which is why we upgraded to 0.10.12. Maybe this issue is related? Thought > this looks more like a v8 issue. > > Please let me know if anyone has any idea/hunch of why this could be > happening. Not sure how to come up with a scenario to reproduce this, but I > think the sudden release of large size of memory leaves the memory > allocator in a bad state or something. I could roll back to 0.8.21 and see > how that behaves. > > Thanks > > - Sumit > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
