On Thu, Sep 26, 2013 at 7:34 PM, spollack <[email protected]> wrote: > I got a chance to do some further testing on this. > > For our scenario, setting max_old_space_size certainly helped with both > memory usage and performance under node 0.10.x, but over our full workload, > under no value of max_old_space_size did performance match 0.8.x, and at > some point sooner or later during the run RSS always exceeded available > memory. > > (Bruno, i also tried some experiments with different streamline runtimes. > using callbacks definitely used less memory than fibers, but, at least with > only that change and no max_old_space_size override, we still went into > swap. I also tried running under node 0.11.x but a number of other package > dependencies of our server would not run there, so no luck trying > generators, at least without more work.) > > My new (or additional) suspect is worse non-heap memory fragmentation under > node 0.10.x vs node 0.8.x. > > Running a small subset of our workload, and then once it finishes running > system.gc() ten or more times to reclaim memory, head to head between the > two node versions: > under 0.10.x: we end with RSS ~400MB, heap total ~80MB, heap used ~35MB > under 0.8.x: we end with RSS ~145MB, heap total ~55MB, heap used ~45MB > > So, even though node 0.10.x ended up after cleanup with slightly less heap > used, RSS was much larger. > > This is also supported by a heapdump (via node-heapdump) from the end of a > 0.10.x run with the subset workload test above, after all the system.gc() > cleanup, where the sum of all "retained size" as reported by chrome was > <150MB, whereas the RSS was close to 400MB. > > So, some questions to validate my assumptions above: > 1) Ben, does "retained size" reported in the heapdump include non-heap > allocations as hinted from node to V8, as you described above?
For buffers, yes. > Is the difference between the sum of retained size, vs. RSS total, due to > fragmentation only, or could it be something else? Fragmentation or traditionally malloc'd memory. I'm not intimately familiar with node-fibers but I assume it uses a fair bit of memory. Same goes for any other native add-ons you may be using. > 2) does system.gc() do full garbage collection? is running that ten times a > fair way to reclaim most/all recoverable memory? It does a major garbage collection. Running it once or ten times in a row should make no observable difference. > 3) did the memory management of non-heap memory (say, for Buffers) change > from node 0.8.x to 0.10.x? Not really. The V8 GC however did change a great deal. > Btw, the subset workload does https external data fetches, buffer/zlib > handling, database writes, and https data posts, over many items, running 25 > in parallel. I will try to create a simplified test case, but does anything > here jump out as suspicious in terms of increased memory usage in node > 0.10.x vs. 0.8.x? > > Thanks, > Seth -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en --- You received this message because you are subscribed to the Google Groups "nodejs" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
