I got a chance to do some further testing on this. For our scenario, setting max_old_space_size certainly helped with both memory usage and performance under node 0.10.x, but over our full workload, under no value of max_old_space_size did performance match 0.8.x, and at some point sooner or later during the run RSS always exceeded available memory.
(Bruno, i also tried some experiments with different streamline runtimes. using callbacks definitely used less memory than fibers, but, at least with only that change and no max_old_space_size override, we still went into swap. I also tried running under node 0.11.x but a number of other package dependencies of our server would not run there, so no luck trying generators, at least without more work.) My new (or additional) suspect is worse non-heap memory fragmentation under node 0.10.x vs node 0.8.x. Running a small subset of our workload, and then once it finishes running system.gc() ten or more times to reclaim memory, head to head between the two node versions: under 0.10.x: we end with RSS ~400MB, heap total ~80MB, heap used ~35MB under 0.8.x: we end with RSS ~145MB, heap total ~55MB, heap used ~45MB So, even though node 0.10.x ended up after cleanup with slightly less heap used, RSS was much larger. This is also supported by a heapdump (via node-heapdump) from the end of a 0.10.x run with the subset workload test above, after all the system.gc() cleanup, where the sum of all "retained size" as reported by chrome was <150MB, whereas the RSS was close to 400MB. So, some questions to validate my assumptions above: 1) Ben, does "retained size" reported in the heapdump include non-heap allocations as hinted from node to V8, as you described above? Is the difference between the sum of retained size, vs. RSS total, due to fragmentation only, or could it be something else? 2) does system.gc() do full garbage collection? is running that ten times a fair way to reclaim most/all recoverable memory? 3) did the memory management of non-heap memory (say, for Buffers) change from node 0.8.x to 0.10.x? Btw, the subset workload does https external data fetches, buffer/zlib handling, database writes, and https data posts, over many items, running 25 in parallel. I will try to create a simplified test case, but does anything here jump out as suspicious in terms of increased memory usage in node 0.10.x vs. 0.8.x? Thanks, Seth -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
