Yeah its quite random but I can come up with a standalone script. Basically I feel the object needs to be in memory for long to be moved on old gen space or something. Anyway, will try to repro this outside of my application.
> I'm not sure I understand what you mean, but nothing is magically becoming non-blocking here. > If you're concerned about doing multiple concurrent INSERTs I'd suggest to use an AUTO_INCREMENT key. You can get the automatically created key with this<https://github.com/felixge/node-mysql#getting-the-id-of-an-inserted-row>. The current approach seems exceptionally brittle and non-scalable. - I meant non-blocking for the requestor. I can respond to him immediately with a random_id rather than wait for mysql's auto-incremented id. Esp useful during db slowdowns or failures. you're making synchronous callbacks which is entirely inappropriate here. I don't see why you'd want these functions to appear asynchronous, but you should at least use .nextTick() the callback. - I don't want the function to be async, it is completely sync in generating a random id for me, except the case when the key "table_name" does not exist in the dictionary, at which point is has to query the db to load all previously created keys. (can only happen on process restarts) The generateUniqueRandom is just an inner function in the actual function which takes care of loading if needed and all. > recentRequestIds[table_name] is never initialized with an object. clearRecentRequestIds deletes the key instead of setting it to an empty object. - It is initialized by the generateUniqueRandom's enclosing function if needed. I delete the key coz that key is for previous day. It will never be reused. The table name is different each day. Data is sharded per day and there is no limit to how many days in future the data can be. So, I create a new table for each day and have date appended to the name. Anyway let me try to come up with a standalone stript that mimics our application. Only thing that will be hard to incorporate is the high number of http requests to the app, if at all its relevant. On Friday, July 5, 2013 7:03:38 AM UTC-7, Bert Belder wrote: > > > I tried to reproduce the leak with a small test reconstructed from your > test case, and I couldn't reproduce. Can you try to create a standalone > test case that I can run locally? > > It might also help to log the output of `node --trace-gc yourscript.js` to > get an idea what the garbage collector is doing. > > > We create random ids for mysql row so that the request is non-blocking > > I'm not sure I understand what you mean, but nothing is magically becoming > non-blocking here. > If you're concerned about doing multiple concurrent INSERTs I'd suggest to > use an AUTO_INCREMENT key. You can get the automatically created key with > this<https://github.com/felixge/node-mysql#getting-the-id-of-an-inserted-row>. > > The current approach seems exceptionally brittle and non-scalable. > > There also are a few important issues with your code: > > - you're making synchronous callbacks which is entirely > inappropriate here. I don't see why you'd want these functions to appear > asynchronous, but you should at least use .nextTick() the callback. > - recentRequestIds[table_name] is never initialized with an object. > clearRecentRequestIds deletes the key instead of setting it to an empty > object. > > > - Bert > > On Friday, July 5, 2013 9:41:19 AM UTC+2, Sumit Agarwal wrote: >> >> We have 10 node.js http servers running on v0.10.12. The servers >> basically act as a gateway where they pipe response of a http client >> request they make, to the original http request. >> >> The code caches around 500k-600k 10 digit integer keys in a dictionary >> for around 24 hours before releasing it. (notice the 2nd line of >> memoryUsage where it frees that). However, after that point, the heapTotal >> starts growing unboundedly even though headUsed does not grow, until the >> program crashes. This has happened 5 days in a row but not on all servers. >> And its pretty random about which server it happens to. >> >> Linux version 2.6.32-279.14.1.el6.x86_64 ( >> [email protected]) >> Running the process with parameter --max_old_space_size=1700 or without >> did not change anything. >> >> Here's the code snippet that caches the data. We create random ids for >> mysql row so that the request is non-blocking, and we cache it to avoid >> collisions. Also, we move to a new table everyday, so at midnight every day >> the entire cache is deleted. >> >> //add to cache code >> >> var generateUniqueRandom = function() { >> >> var recentRequestIdsForTable = recentRequestIds[table_name]; // >> recentRequestIds is a dictionary and recentRequestIdsForTable is also >> dictionary keyed by table_name. The most filled recentRequestIdsForTable is >> for the current day table and it goes up to 600k, expected to reach 1 >> million per day soon. >> >> var attempts = 0; >> >> var random; >> >> while(attempts < 5) { >> >> random = Math.floor(Math.random() * (9999999999 - 1000000000) >> + 1000000000); // 10 digit random id >> >> if (!(random in recentRequestIdsForTable)) { >> >> recentRequestIdsForTable[random] = true; >> >> return callback(null, table_name + ":" + random); >> >> } >> >> attempts++; >> >> } >> >> return callback("Could not generate requestID", null); >> >> }; >> >> //delete from cache code >> >> exports.clearRecentRequestIds = function(table_name) { >> >> if (table_name in recentRequestIds) { >> >> delete recentRequestIds[table_name]; // drop the entire cached >> dictionary for the table_name. >> >> logger.info('cleared all recentRequestIds for table ' + >> table_name); >> >> return; >> >> } >> >> logger.warn(table_name + ' not found in recentRequestIds'); >> >> } >> >> Here's the memory usage. It increased to the max in around 1.5 hours. >> During that time, the server was functioning normally and had no CPU >> spikes. It is weird that around 500k keys should mean up to around 5MB-10MB >> of data if worst case keys are stored as string. However they take up space >> over 100 MB. (Coz freeing up reduced memory from 217MB to 76MB). Maybe it >> has something to do with that? >> >> {"rss":"368.0078125 MB","heapTotal":"291.7858581542969 >> MB","heapUsed":"217.14698028564453 MB"} >> {"rss":"357.16796875 MB","heapTotal":"287.84947204589844 >> MB","heapUsed":"76.96298217773438 MB"} >> {"rss":"475.1796875 MB","heapTotal":"409.88169860839844 >> MB","heapUsed":"66.50176239013672 MB"} >> {"rss":"577.9921875 MB","heapTotal":"514.1995697021484 >> MB","heapUsed":"83.56008911132812 MB"} >> >> - and eventually to >> >> {"rss":"1463.27734375 MB","heapTotal":"1381.2594375610352 >> MB","heapUsed":"95.49384307861328 MB"} >> {"rss":"1500.05078125 MB","heapTotal":"1413.7357559204102 >> MB","heapUsed":"93.21699523925781 MB"} >> >> Also, we did experience the issue in 0.10.10 which was busy loop in net >> module that led to infinite memory increase and CPU spike a couple of times >> which is why we upgraded to 0.10.12. Maybe this issue is related? Thought >> this looks more like a v8 issue. >> >> Please let me know if anyone has any idea/hunch of why this could be >> happening. Not sure how to come up with a scenario to reproduce this, but I >> think the sudden release of large size of memory leaves the memory >> allocator in a bad state or something. I could roll back to 0.8.21 and see >> how that behaves. >> >> Thanks >> >> - Sumit >> > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
