On Thu, May 17, 2012 at 10:33 AM, Erik Dubbelboer <[email protected]> wrote: > It happened again. > > But judging from the stack trace it seems more like a problem in V8. > > There are 3 threads running. > > This one seems to be from libeio waiting for work (which it will never get > in our app): > #0 0x00007f3269366d84 in pthread_cond_wait@@GLIBC_2.3.2 () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x00000000007cba48 in etp_proc (thr_arg=0x246e680) at > src/unix/eio/eio.c:2068 > #2 0x00007f3269362e9a in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #3 0x00007f32690904bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > #4 0x0000000000000000 in ?? () > > This one seems to be a V8 maintenance thread that's just waiting. > #0 0x00007f3269368fd0 in sem_wait () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #1 0x0000000000683ae3 in v8::internal::LinuxSemaphore::Wait() () > #2 0x000000000069c7a0 in > v8::internal::RuntimeProfilerRateLimiter::SuspendIfNecessary() () > #3 0x0000000000683ede in v8::internal::SignalSender::Run() () > #4 0x0000000000683bce in v8::internal::ThreadEntry(void*) () > #5 0x00007f3269362e9a in start_thread () from > /lib/x86_64-linux-gnu/libpthread.so.0 > #6 0x00007f32690904bd in clone () from /lib/x86_64-linux-gnu/libc.so.6 > #7 0x0000000000000000 in ?? () > > And this is the main thread: > #0 0x0000000000641af6 in > v8::internal::MarkCompactCollector::EncodeForwardingAddresses() () > #1 0x00000000006432a6 in > v8::internal::MarkCompactCollector::CollectGarbage() () > #2 0x00000000005e2693 in > v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, > v8::internal::GCTracer*) [clone .constprop.435] () > #3 0x00000000005e2ab9 in > v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, > v8::internal::GarbageCollector) () > #4 0x00000000005e2be2 in v8::internal::Heap::CollectAllAvailableGarbage() > () > #5 0x00000000005b8e3f in > v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, > v8::internal::Code::Flags, v8::internal::Handle<v8::internal::Object>, bool) > () > #6 0x0000000000774689 in > v8::internal::RegExpMacroAssemblerX64::GetCode(v8::internal::Handle<v8::internal::String>) > () > #7 0x000000000061edc4 in > v8::internal::RegExpEngine::Compile(v8::internal::RegExpCompileData*, bool, > bool, v8::internal::Handle<v8::internal::String>, bool) () > #8 0x000000000061f031 in > v8::internal::RegExpImpl::CompileIrregexp(v8::internal::Handle<v8::internal::JSRegExp>, > bool) () > #9 0x000000000061f3d7 in > v8::internal::RegExpImpl::IrregexpPrepare(v8::internal::Handle<v8::internal::JSRegExp>, > v8::internal::Handle<v8::internal::String>) () > #10 0x000000000061f7bf in > v8::internal::RegExpImpl::IrregexpExec(v8::internal::Handle<v8::internal::JSRegExp>, > v8::internal::Handle<v8::internal::String>, int, > v8::internal::Handle<v8::internal::JSArray>) () > #11 0x000000000061fab5 in > v8::internal::RegExpImpl::Exec(v8::internal::Handle<v8::internal::JSRegExp>, > v8::internal::Handle<v8::internal::String>, int, > v8::internal::Handle<v8::internal::JSArray>) > () > #12 0x00000000006afe99 in > v8::internal::Runtime_RegExpExec(v8::internal::Arguments, > v8::internal::Isolate*) () > #13 0x000015e6ad92a14e in ?? () > > From what I can see it starts garbage collecting when executing a RegExp. > I'm thinking maybe V8 had problems allocating more memory for some reason > and kept waiting for memory to be freed which never happened. The process is > indeed using a lot of memory but our server still had over 4GB of free > memory left and there are no memory limits in place. I'm keeping an eye on > our app for the next couple of days cause it shouldn't be using that much > memory so my guess is the high memory usage is also caused by the bug. > > PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command > 9553 root 20 0 2090m 1.4g 12m R 100 17.8 403:25.15 node > rtb-adserver.js > > I have also created a core file but when I start gdb from that and print a > stack trace I get a different stack: > #0 0x0000000000636e03 in > v8::internal::UpdatingVisitor::VisitPointers(v8::internal::Object**, > v8::internal::Object**) () > #1 0x000000000064d825 in > v8::internal::HeapObject::IterateBody(v8::internal::InstanceType, int, > v8::internal::ObjectVisitor*) () > #2 0x0000000000637e06 in > v8::internal::MarkCompactCollector::UpdatePointersInOldObject(v8::internal::HeapObject*) > () > #3 0x000000000063f11b in > v8::internal::MarkCompactCollector::IterateLiveObjects(v8::internal::PagedSpace*, > int (v8::internal::MarkCompactCollector::*)(v8::internal::HeapObject*)) () > #4 0x000000000063f2a7 in > v8::internal::MarkCompactCollector::UpdatePointers() () > #5 0x00000000006432bd in > v8::internal::MarkCompactCollector::CollectGarbage() () > #6 0x00000000005e2693 in > v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, > v8::internal::GCTracer*) [clone .constprop.435] () > #7 0x00000000005e2ab9 in > v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, > v8::internal::GarbageCollector) () > #8 0x00000000005e2bb6 in v8::internal::Heap::CollectAllAvailableGarbage() > () > #9 0x00000000005b8e3f in > v8::internal::Factory::NewCode(v8::internal::CodeDesc const&, > v8::internal::Code::Flags, v8::internal::Handle<v8::internal::Object>, bool) > () > #10 0x0000000000774689 in > v8::internal::RegExpMacroAssemblerX64::GetCode(v8::internal::Handle<v8::internal::String>) > () > #11 0x000000000061edc4 in > v8::internal::RegExpEngine::Compile(v8::internal::RegExpCompileData*, bool, > bool, v8::internal::Handle<v8::internal::String>, bool) () > #12 0x000000000061f031 in > v8::internal::RegExpImpl::CompileIrregexp(v8::internal::Handle<v8::internal::JSRegExp>, > bool) () > #13 0x000000000061f3d7 in > v8::internal::RegExpImpl::IrregexpPrepare(v8::internal::Handle<v8::internal::JSRegExp>, > v8::internal::Handle<v8::internal::String>) () > #14 0x000000000061f7bf in > v8::internal::RegExpImpl::IrregexpExec(v8::internal::Handle<v8::internal::JSRegExp>, > v8::internal::Handle<v8::internal::String>, int, > v8::internal::Handle<v8::internal::JSArray>) () > #15 0x000000000061fab5 in > v8::internal::RegExpImpl::Exec(v8::internal::Handle<v8::internal::JSRegExp>, > v8::internal::Handle<v8::internal::String>, int, > v8::internal::Handle<v8::internal::JSArray>) > () > #16 0x00000000006afe99 in > v8::internal::Runtime_RegExpExec(v8::internal::Arguments, > v8::internal::Isolate*) () > #17 0x000015e6ad92a14e in ?? () > > What should I do next? > > Ben, do you want me to send you the core so you can have a look yourself?
I don't think there's much I can do. A lockup or a hang is one thing but those backtraces indicate that V8 simply spends (a lot?) time collecting garbage. Is it always a regular expression that kicks of the GC? Does the GC run finish or does it enter some kind of infinite loop? gdb's tracing facilities can help you with that (or use the poor man's tracer: repeatedly interrupt and continue the process, inspect the call stack in between). -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
