It happened again.
But judging from the stack trace it seems more like a problem in V8.
There are 3 threads running.
This one seems to be from libeio waiting for work (which it will never get
in our app):
#0 0x00007f3269366d84 in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x00000000007cba48 in etp_proc (thr_arg=0x246e680) at
src/unix/eio/eio.c:2068
#2 0x00007f3269362e9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#3 0x00007f32690904bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#4 0x0000000000000000 in ?? ()
This one seems to be a V8 maintenance thread that's just waiting.
#0 0x00007f3269368fd0 in sem_wait () from
/lib/x86_64-linux-gnu/libpthread.so.0
#1 0x0000000000683ae3 in v8::internal::LinuxSemaphore::Wait() ()
#2 0x000000000069c7a0 in
v8::internal::RuntimeProfilerRateLimiter::SuspendIfNecessary() ()
#3 0x0000000000683ede in v8::internal::SignalSender::Run() ()
#4 0x0000000000683bce in v8::internal::ThreadEntry(void*) ()
#5 0x00007f3269362e9a in start_thread () from
/lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007f32690904bd in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()
And this is the main thread:
#0 0x0000000000641af6 in
v8::internal::MarkCompactCollector::EncodeForwardingAddresses() ()
#1 0x00000000006432a6 in
v8::internal::MarkCompactCollector::CollectGarbage() ()
#2 0x00000000005e2693 in
v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector,
v8::internal::GCTracer*) [clone .constprop.435] ()
#3 0x00000000005e2ab9 in
v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace,
v8::internal::GarbageCollector) ()
#4 0x00000000005e2be2 in v8::internal::Heap::CollectAllAvailableGarbage()
()
#5 0x00000000005b8e3f in
v8::internal::Factory::NewCode(v8::internal::CodeDesc const&,
v8::internal::Code::Flags, v8::internal::Handle<v8::internal::Object>,
bool) ()
#6 0x0000000000774689 in
v8::internal::RegExpMacroAssemblerX64::GetCode(v8::internal::Handle<v8::internal::String>)
()
#7 0x000000000061edc4 in
v8::internal::RegExpEngine::Compile(v8::internal::RegExpCompileData*, bool,
bool, v8::internal::Handle<v8::internal::String>, bool) ()
#8 0x000000000061f031 in
v8::internal::RegExpImpl::CompileIrregexp(v8::internal::Handle<v8::internal::JSRegExp>,
bool) ()
#9 0x000000000061f3d7 in
v8::internal::RegExpImpl::IrregexpPrepare(v8::internal::Handle<v8::internal::JSRegExp>,
v8::internal::Handle<v8::internal::String>) ()
#10 0x000000000061f7bf in
v8::internal::RegExpImpl::IrregexpExec(v8::internal::Handle<v8::internal::JSRegExp>,
v8::internal::Handle<v8::internal::String>, int,
v8::internal::Handle<v8::internal::JSArray>) ()
#11 0x000000000061fab5 in
v8::internal::RegExpImpl::Exec(v8::internal::Handle<v8::internal::JSRegExp>,
v8::internal::Handle<v8::internal::String>, int,
v8::internal::Handle<v8::internal::JSArray>)
()
#12 0x00000000006afe99 in
v8::internal::Runtime_RegExpExec(v8::internal::Arguments,
v8::internal::Isolate*) ()
#13 0x000015e6ad92a14e in ?? ()
>From what I can see it starts garbage collecting when executing a RegExp.
I'm thinking maybe V8 had problems allocating more memory for some reason
and kept waiting for memory to be freed which never happened. The process
is indeed using a lot of memory but our server still had over 4GB of free
memory left and there are no memory limits in place. I'm keeping an eye on
our app for the next couple of days cause it shouldn't be using that much
memory so my guess is the high memory usage is also caused by the bug.
PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command
9553 root 20 0 2090m 1.4g 12m R 100 17.8 403:25.15 node
rtb-adserver.js
I have also created a core file but when I start gdb from that and print a
stack trace I get a different stack:
#0 0x0000000000636e03 in
v8::internal::UpdatingVisitor::VisitPointers(v8::internal::Object**,
v8::internal::Object**) ()
#1 0x000000000064d825 in
v8::internal::HeapObject::IterateBody(v8::internal::InstanceType, int,
v8::internal::ObjectVisitor*) ()
#2 0x0000000000637e06 in
v8::internal::MarkCompactCollector::UpdatePointersInOldObject(v8::internal::HeapObject*)
()
#3 0x000000000063f11b in
v8::internal::MarkCompactCollector::IterateLiveObjects(v8::internal::PagedSpace*,
int (v8::internal::MarkCompactCollector::*)(v8::internal::HeapObject*)) ()
#4 0x000000000063f2a7 in
v8::internal::MarkCompactCollector::UpdatePointers() ()
#5 0x00000000006432bd in
v8::internal::MarkCompactCollector::CollectGarbage() ()
#6 0x00000000005e2693 in
v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector,
v8::internal::GCTracer*) [clone .constprop.435] ()
#7 0x00000000005e2ab9 in
v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace,
v8::internal::GarbageCollector) ()
#8 0x00000000005e2bb6 in v8::internal::Heap::CollectAllAvailableGarbage()
()
#9 0x00000000005b8e3f in
v8::internal::Factory::NewCode(v8::internal::CodeDesc const&,
v8::internal::Code::Flags, v8::internal::Handle<v8::internal::Object>,
bool) ()
#10 0x0000000000774689 in
v8::internal::RegExpMacroAssemblerX64::GetCode(v8::internal::Handle<v8::internal::String>)
()
#11 0x000000000061edc4 in
v8::internal::RegExpEngine::Compile(v8::internal::RegExpCompileData*, bool,
bool, v8::internal::Handle<v8::internal::String>, bool) ()
#12 0x000000000061f031 in
v8::internal::RegExpImpl::CompileIrregexp(v8::internal::Handle<v8::internal::JSRegExp>,
bool) ()
#13 0x000000000061f3d7 in
v8::internal::RegExpImpl::IrregexpPrepare(v8::internal::Handle<v8::internal::JSRegExp>,
v8::internal::Handle<v8::internal::String>) ()
#14 0x000000000061f7bf in
v8::internal::RegExpImpl::IrregexpExec(v8::internal::Handle<v8::internal::JSRegExp>,
v8::internal::Handle<v8::internal::String>, int,
v8::internal::Handle<v8::internal::JSArray>) ()
#15 0x000000000061fab5 in
v8::internal::RegExpImpl::Exec(v8::internal::Handle<v8::internal::JSRegExp>,
v8::internal::Handle<v8::internal::String>, int,
v8::internal::Handle<v8::internal::JSArray>)
()
#16 0x00000000006afe99 in
v8::internal::Runtime_RegExpExec(v8::internal::Arguments,
v8::internal::Isolate*) ()
#17 0x000015e6ad92a14e in ?? ()
What should I do next?
Ben, do you want me to send you the core so you can have a look yourself?
Thanks in advance,
Erik
On Thursday, May 10, 2012 2:19:45 PM UTC+2, Ben Noordhuis wrote:
>
> On Thu, May 10, 2012 at 12:48 PM, Erik Dubbelboer <[email protected]>
> wrote:
> > We have an application that handles a couple of http and udp requests
> per
> > second. It uses no C modules, only modules written in javascript.
> >
> > From time to time the application will hang. Attaching strace at this
> point
> > will only output lines of:
> >
> > futex(0x10e36c8, FUTEX_WAKE_PRIVATE, 1) = 1
> >
> > As you can see the call does wake up a process but for some reason it's
> > still in a deadlock. Since we only use javascript code it is most likely
> a
> > bug in nodejs.
> >
> > The problem is that I have no idea which conditions cause the deadlock.
> It
> > can takes days (sometimes over a week) for this to happen. I haven't
> been
> > able to reproduce the deadlock in a development environment.
> >
> > What would be the best way to hunt down this bug?
>
> Attach with gdb and inspect the call stack. Alternatively, make a core
> dump with gcore and send it my way, I can probably figure out what's
> going on.
>
> What do `uname -a`, `node -v` and `ldd node` say? Is your application
> using TLS or crypto?
>
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
