Yeah this does look like a fragmentation/lazyness issue. Heap has only 8MB of live data, but keeps 50mb reserved for old pointer space alone. This does not look bright.
First thing I would recommend is to try and roll V8 forward for your build of node from 3.11.10.12 to 3.11.10.13 and remeasure. I strongly suspect that memory usage will go down. -- Vyacheslav Egorov On Fri, Jul 6, 2012 at 12:32 PM, chusi <[email protected]> wrote: > Ok, i waited till ps showed around 50MB (still out-of-the box compiled node > 0.8.1 without arguments) > > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND > nodejs 12655 0.0 2.4 731204 50528 pts/0 Sl+ Jul05 0:13 node > /opt/turntable/bin/hook_luncher.js --hook stats > > at the same time i see debug output from nodes gc like this: > > 46965999 ms: Mark-sweep 8.7 (95.0) -> 8.3 (95.0) MB, 0 ms (+ 7 ms in 1 steps > since start of marking, biggest step 7.866211 ms) [idle notification: > finalize incremental] [GC in old space requested]. > Memory allocator, used: 99598336, available: 1435516928 > New space, used: 0, available: 1048576 > Old pointers, used: 50052944, available: 4272, waste: 0 > Old data space, used: 6388224, available: 0, waste: 0 > Code space, used: 2606944, available: 1472672, waste: 0 > Map space, used: 252056, available: 910952, waste: 0 > Cell space, used: 98240, available: 1032000, waste: 0 > Large object space, used: 0, available: 1434451712 > > 46966009 ms: Mark-sweep 8.7 (95.0) -> 8.3 (95.0) MB, 0 ms (+ 7 ms in 1 steps > since start of marking, biggest step 7.776855 ms) [idle notification: > finalize incremental] [GC in old space requested]. > Memory allocator, used: 99598336, available: 1435516928 > New space, used: 0, available: 1048576 > Old pointers, used: 50052944, available: 4272, waste: 0 > Old data space, used: 6388224, available: 0, waste: 0 > Code space, used: 2606944, available: 1472672, waste: 0 > Map space, used: 252056, available: 910952, waste: 0 > Cell space, used: 98240, available: 1032000, waste: 0 > Large object space, used: 0, available: 1434451712 > > how does this output reflect the 50MB RSS? Can you see something that is > wrong? > > i also posted the whole trace (from the beginning (-13h) till now) at > http://pastebin.com/K1RmPKUC > > > > On Thursday, July 5, 2012 12:37:09 PM UTC+2, Vyacheslav Egorov wrote: >> >> --trace-gc-verbose should be used together with --trace-gc. >> >> When enabled should produce something like: >> >> 1087 ms: Scavenge 2.6 (20.4) -> 1.6 (20.4) MB, 0 ms [allocation >> failure]. >> Memory allocator, used: 21409792, available: 746147840 >> New space, used: 0, available: 1048576 >> Old pointers, used: 1277824, available: 0, waste: 0 >> Old data space, used: 1173980, available: 38308, waste: 0 >> Code space, used: 1396000, available: 736, waste: 0 >> Map space, used: 131072, available: 0, waste: 0 >> Cell space, used: 98304, available: 0, waste: 0 >> Large object space, used: 0, available: 745066368 >> >> I need to see this kind of fine grained statistics to make a guess. >> >> You can also try to update V8 inside node from 3.11.10.12 to 3.11.10.13 so >> that >> CL improving memory usage >> (https://chromiumcodereview.appspot.com/10699051) is included. >> >> -- >> Vyacheslav Egorov >> >> >> On Thu, Jul 5, 2012 at 9:40 AM, chusi wrote: >> > >> > the --trace-gc-verbose parameter didn't print anything (at least not in >> > the >> > minute i waited) and until i reach the 50 MB (monitoring using >> > --trace-gc) >> > at least some hours have to pass by. I will try it later. >> > >> > here's a pic of running the same code once with node v0.6.17 (chris >> > lea's >> > packet) and node v.0.8.1 (my own compiled version). >> > >> > http://troll.ws/i/nJQJFr.png >> > >> > before I updated to the new node version i was fighting against some >> > memory >> > leaks in my code, but was able to fix them. when i switched to v.0.8.1 I >> > was >> > worried that they seemed to be back (when i saw the graph). besides the >> > "red >> > line" that goes crazy, i guess the rest can be explained by the new more >> > lazy garbage collection approach. >> > >> > btw, the app is almost idle. the different modules shown in the graph >> > just >> > communicate from time to time with each other and only exchange tiny >> > bits of >> > data. >> > >> > >> > >> > On Wednesday, July 4, 2012 5:33:34 PM UTC+2, Vyacheslav Egorov wrote: >> >> >> >> If you run you app with --trace-gc what does it print (I am interested >> >> in GC statistics for the time when app reaches 50MB)? >> >> >> >> I am also curious in --trace-gc-verbose output to see how memory is >> >> distributed between different spaces in V8? >> >> >> >> It might be that the fragmentation in the heap became higher due to >> >> lazyness of the new collector (it is tunes for interactive performance >> >> so it tries to do things in small increments). >> >> >> >> Also I am not sure what the current situation with idle notifications >> >> in node is but they are very important for the new collector to >> >> perform bigger steps when app is idle. >> >> >> >> Additionally there were some changes recently that increased >> >> aggressiveness with which new collector sweeps and compacts things out >> >> from almost empty pages. You might be being bitten by this issues. >> >> >> >> -- >> >> Vyacheslav Egorov >> >> >> >> >> >> On Wed, Jul 4, 2012 at 12:42 PM, Ben Noordhuis <[email protected]> >> >> wrote: >> >> > On Wed, Jul 4, 2012 at 11:39 AM, chusi wrote: >> >> >> On Tuesday, July 3, 2012 7:53:26 PM UTC+2, Ben Noordhuis wrote: >> >> >>> Sanity check - what do `file /path/to/old/node` and `file >> >> >>> /path/to/new/node` print? >> >> >> >> >> >> old 0.6.17: >> >> >> ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically >> >> >> linked >> >> >> (uses shared libs), for GNU/Linux 2.6.15, stripped >> >> >> >> >> >> new 0.8.1 >> >> >> ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically >> >> >> linked >> >> >> (uses shared libs), for GNU/Linux 2.6.15, stripped >> >> >> >> >> >> node is installed using chis-lea's package as described here: >> >> >> >> >> >> >> >> >> https://github.com/joyent/node/wiki/Installing-Node.js-via-package-manager >> >> > >> >> > Output from `file` looks good. If one binary was built for ia32 and >> >> > the other for x64, you'd expect to see different memory profiles, but >> >> > that's not the case here. >> >> > >> >> > I don't know if Chris lands patches on top in a way that changes >> >> > memory consumption. Do you see the same behavior with a (tarball, >> >> > git) >> >> > source build? >> >> > >> >> > One thing to note is that V8 has a new and improved (but also more >> >> > lazy) garbage collector. Run your app with --expose-gc and call gc() >> >> > from time to time if memory pressure is a real issue - but I would >> >> > generally advise against that. >> >> > >> >> > -- >> >> > Job Board: http://jobs.nodejs.org/ >> >> > Posting guidelines: >> >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> >> > You received this message because you are subscribed to the Google >> >> > Groups "nodejs" group. >> >> > To post to this group, send email to [email protected] >> >> > To unsubscribe from this group, send email to >> >> > [email protected] >> >> > For more options, visit this group at >> >> > http://groups.google.com/group/nodejs?hl=en?hl=en >> > >> > -- >> > Job Board: http://jobs.nodejs.org/ >> > Posting guidelines: >> > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> > You received this message because you are subscribed to the Google >> > Groups "nodejs" group. >> > To post to this group, send email to [email protected] >> > To unsubscribe from this group, send email to >> > [email protected] >> > For more options, visit this group at >> > http://groups.google.com/group/nodejs?hl=en?hl=en > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to [email protected] > To unsubscribe from this group, send email to > [email protected] > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
