Correct. I don't know if this will reproduce it coz the live servers are doing much more than just id generation. I chose the values I did in order to mimic the servers as close as possible. More than load, I feel the issue is with time the data spends in memory. Regardless of the application/code, the fact that heapUsed stays almost constant and less than 100MB but heapTotal keeps increasing super quickly until program crashes (in around an hour) is alarming to me. Anyway, I will put this script on a few boxes with different parameters and see if I can repro in the few days. Will post if I do. Thanks
Sumit On Friday, July 5, 2013 11:12:38 AM UTC-7, Bert Belder wrote: > > > I meant non-blocking for the requestor. I can respond to him > immediately with a random_id rather than wait for mysql's auto-incremented > id. Esp useful during db slowdowns or failures. > > If you like your job, don't. > > > I don't want the function to be async, it is completely sync in > generating a random id for me, except the case when the key "table_name" > does not exist in the dictionary, at which point is has to query the db to > load all previously created keys. (can only happen on process restarts) The > generateUniqueRandom > is just an inner function in the actual function which takes care of > loading if needed and all. > > The general way node does it is it guarantees that callbacks are always > made from a fresh stack (except when .emit() is used). Your function will, > in that case, *sometimes* return from a fresh stack. > > > Attached a test that will add 20 random ids every second. The keys will > be deleted every 24 hours. You can configure the numbers to add more keys > or delete more frequently. I'll keep this running on a few boxes separately > and see if its reproducible. It definitely happens on our live server > everyday though but that is under way more stress than this script. > > I don't want to test something for 24 hours - unless you are certain that > it certainly does not happen within minutes. Why are you only generating 20 > keys per second? Also you don't even know if this reproduces it for > yourself since no 24 hours have passed since me asking for a reduced test > case. > > - Bert > > On Friday, July 5, 2013 6:11:52 PM UTC+2, Sumit Agarwal wrote: >> >> Attached a test that will add 20 random ids every second. The keys will >> be deleted every 24 hours. You can configure the numbers to add more keys >> or delete more frequently. I'll keep this running on a few boxes separately >> and see if its reproducible. It definitely happens on our live server >> everyday though but that is under way more stress than this script. >> >> On Friday, July 5, 2013 7:03:38 AM UTC-7, Bert Belder wrote: >>> >>> >>> I tried to reproduce the leak with a small test reconstructed from your >>> test case, and I couldn't reproduce. Can you try to create a standalone >>> test case that I can run locally? >>> >>> It might also help to log the output of `node --trace-gc yourscript.js` >>> to get an idea what the garbage collector is doing. >>> >>> > We create random ids for mysql row so that the request is >>> non-blocking >>> >>> I'm not sure I understand what you mean, but nothing is magically >>> becoming non-blocking here. >>> If you're concerned about doing multiple concurrent INSERTs I'd suggest >>> to use an AUTO_INCREMENT key. You can get the automatically created key >>> with >>> this<https://github.com/felixge/node-mysql#getting-the-id-of-an-inserted-row>. >>> >>> The current approach seems exceptionally brittle and non-scalable. >>> >>> There also are a few important issues with your code: >>> >>> - you're making synchronous callbacks which is entirely >>> inappropriate here. I don't see why you'd want these functions to appear >>> asynchronous, but you should at least use .nextTick() the callback. >>> - recentRequestIds[table_name] is never initialized with an >>> object. clearRecentRequestIds deletes the key instead of setting it to >>> an >>> empty object. >>> >>> >>> - Bert >>> >>> On Friday, July 5, 2013 9:41:19 AM UTC+2, Sumit Agarwal wrote: >>>> >>>> We have 10 node.js http servers running on v0.10.12. The servers >>>> basically act as a gateway where they pipe response of a http client >>>> request they make, to the original http request. >>>> >>>> The code caches around 500k-600k 10 digit integer keys in a dictionary >>>> for around 24 hours before releasing it. (notice the 2nd line of >>>> memoryUsage where it frees that). However, after that point, the heapTotal >>>> starts growing unboundedly even though headUsed does not grow, until the >>>> program crashes. This has happened 5 days in a row but not on all servers. >>>> And its pretty random about which server it happens to. >>>> >>>> Linux version 2.6.32-279.14.1.el6.x86_64 ( >>>> [email protected]) >>>> Running the process with parameter --max_old_space_size=1700 or without >>>> did not change anything. >>>> >>>> Here's the code snippet that caches the data. We create random ids for >>>> mysql row so that the request is non-blocking, and we cache it to avoid >>>> collisions. Also, we move to a new table everyday, so at midnight every >>>> day >>>> the entire cache is deleted. >>>> >>>> //add to cache code >>>> >>>> var generateUniqueRandom = function() { >>>> >>>> var recentRequestIdsForTable = recentRequestIds[table_name]; >>>> // recentRequestIds is a dictionary and recentRequestIdsForTable is also >>>> dictionary keyed by table_name. The most filled recentRequestIdsForTable >>>> is >>>> for the current day table and it goes up to 600k, expected to reach 1 >>>> million per day soon. >>>> >>>> var attempts = 0; >>>> >>>> var random; >>>> >>>> while(attempts < 5) { >>>> >>>> random = Math.floor(Math.random() * (9999999999 - >>>> 1000000000) + 1000000000); // 10 digit random id >>>> >>>> if (!(random in recentRequestIdsForTable)) { >>>> >>>> recentRequestIdsForTable[random] = true; >>>> >>>> return callback(null, table_name + ":" + random); >>>> >>>> } >>>> >>>> attempts++; >>>> >>>> } >>>> >>>> return callback("Could not generate requestID", null); >>>> >>>> }; >>>> >>>> //delete from cache code >>>> >>>> exports.clearRecentRequestIds = function(table_name) { >>>> >>>> if (table_name in recentRequestIds) { >>>> >>>> delete recentRequestIds[table_name]; // drop the entire cached >>>> dictionary for the table_name. >>>> >>>> logger.info('cleared all recentRequestIds for table ' + >>>> table_name); >>>> >>>> return; >>>> >>>> } >>>> >>>> logger.warn(table_name + ' not found in recentRequestIds'); >>>> >>>> } >>>> >>>> Here's the memory usage. It increased to the max in around 1.5 hours. >>>> During that time, the server was functioning normally and had no CPU >>>> spikes. It is weird that around 500k keys should mean up to around >>>> 5MB-10MB >>>> of data if worst case keys are stored as string. However they take up >>>> space >>>> over 100 MB. (Coz freeing up reduced memory from 217MB to 76MB). Maybe it >>>> has something to do with that? >>>> >>>> {"rss":"368.0078125 MB","heapTotal":"291.7858581542969 >>>> MB","heapUsed":"217.14698028564453 MB"} >>>> {"rss":"357.16796875 MB","heapTotal":"287.84947204589844 >>>> MB","heapUsed":"76.96298217773438 MB"} >>>> {"rss":"475.1796875 MB","heapTotal":"409.88169860839844 >>>> MB","heapUsed":"66.50176239013672 MB"} >>>> {"rss":"577.9921875 MB","heapTotal":"514.1995697021484 >>>> MB","heapUsed":"83.56008911132812 MB"} >>>> >>>> - and eventually to >>>> >>>> {"rss":"1463.27734375 MB","heapTotal":"1381.2594375610352 >>>> MB","heapUsed":"95.49384307861328 MB"} >>>> {"rss":"1500.05078125 MB","heapTotal":"1413.7357559204102 >>>> MB","heapUsed":"93.21699523925781 MB"} >>>> >>>> Also, we did experience the issue in 0.10.10 which was busy loop in net >>>> module that led to infinite memory increase and CPU spike a couple of >>>> times >>>> which is why we upgraded to 0.10.12. Maybe this issue is related? Thought >>>> this looks more like a v8 issue. >>>> >>>> Please let me know if anyone has any idea/hunch of why this could be >>>> happening. Not sure how to come up with a scenario to reproduce this, but >>>> I >>>> think the sudden release of large size of memory leaves the memory >>>> allocator in a bad state or something. I could roll back to 0.8.21 and see >>>> how that behaves. >>>> >>>> Thanks >>>> >>>> - Sumit >>>> >>> -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
