Thank's for the reply! 😄 On Wednesday, September 27, 2017 at 3:44:32 PM UTC+8, Alex Bagehot wrote: > > Nitsan reported and fixed a 1/10 occurrence hang here: > https://github.com/jvm-profiling-tools/honest-profiler/issues/75 > Assuming You're using the lightweight java profiler which doesn't have > that fix. > It is allocating memory in a signal handler see #3 and #9 below. > > 1 Point in time snapshot of thread state doesn't help much understand cpu > usage - you need to take several of them or preferably get some perf output > (which gives you a large number of unbiased cpu samples to work with). > > However, the high/100% cpu of 1 thread associated with the hang happens > because the thread initiating the safepoint (here LWP 25184) will > spinpause/yield until all the threads have come to safepoint (which can't > happen because of the profiler bug) or a safepoint timeout occurs. > > thanks, > Alex > > pstack.txt : > > Thread 2 (Thread 0x7fe359282700 (LWP 26488)): > #0 0x00000037b4af805e in __lll_lock_wait_private () from /lib64/libc.so.6 > #1 0x00000037b4a7cc82 in _L_lock_3495 () from /lib64/libc.so.6 > #2 0x00000037b4a77903 in arena_get2 () from /lib64/libc.so.6 > #3 0x00000037b4a7a794 in malloc () from /lib64/libc.so.6 > #4 0x00000037b4611190 in tls_get_addr_tail () from > /lib64/ld-linux-x86-64.so.2 > #5 0x00000037b4611660 in __tls_get_addr () from > /lib64/ld-linux-x86-64.so.2 > #6 0x00007fe4adabd762 in Profiler::Handle(int, siginfo*, void*) () from > /usr/local/lightweight-java-profiler-master/build-64/liblagent.so > #7 <signal handler called> > #8 0x00000037b4a77916 in arena_get2 () from /lib64/libc.so.6 > #9 0x00000037b4a7a794 in malloc () from /lib64/libc.so.6 > #10 0x00000037b5208b3f in pthread_getattr_np () from /lib64/libpthread.so.0 > #11 0x00007fe4af00f724 in current_stack_region(unsigned char**, unsigned > long*) () from /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so > #12 0x00007fe4af00f815 in os::current_stack_base() () from > /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so > #13 0x00007fe4af13e614 in Thread::record_stack_base_and_size() () from > /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so > #14 0x00007fe4af144dd4 in JavaThread::run() () from > /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so > #15 0x00007fe4af00b988 in java_start(Thread*) () from > /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so > #16 0x00000037b52079d1 in start_thread () from /lib64/libpthread.so.0 > #17 0x00000037b4ae88fd in clone () from /lib64/libc.so.6 > > > > > > On Wed, Sep 27, 2017 at 7:51 AM, yang liu <[email protected] > <javascript:>> wrote: > >> One of tomcat program hung and failed to response http request. After >> inspect the output of "jstack -F" and "pstack", I found the jvm stuck in >> SafepointSynchronize::begin() method. Why could this happen? >> >> More details following. >> *1. top -H -p output**, thread 25184 cpu 100%* >> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND >> 25184 root 20 0 14.8g 5.0g 62m R 100.3 32.8 17:50.67 java >> <-------------- >> 25170 root 20 0 14.8g 5.0g 62m S 0.0 32.8 0:00.01 java >> 25172 root 20 0 14.8g 5.0g 62m S 0.0 32.8 0:01.09 java >> 25173 root 20 0 14.8g 5.0g 62m S 0.0 32.8 0:02.61 java >> 25174 root 20 0 14.8g 5.0g 62m S 0.0 32.8 0:02.61 java >> 25175 root 20 0 14.8g 5.0g 62m S 0.0 32.8 0:02.59 java >> 25176 root 20 0 14.8g 5.0g 62m S 0.0 32.8 0:02.66 java >> 25177 root 20 0 14.8g 5.0g 62m S 0.0 32.8 0:02.62 java >> 25178 root 20 0 14.8g 5.0g 62m S 0.0 32.8 0:02.63 java >> >> *2. pstack **output**, thread 25184 cpu 100%* >> Thread 797 (Thread 0x7fe47d506700 (LWP 25184)): >> #0 0x00007fe4af098d2b in SafepointSynchronize::begin() () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so <-------------- >> #1 0x00007fe4af191fef in VMThread::loop() () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #2 0x00007fe4af192470 in VMThread::run() () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #3 0x00007fe4af00b988 in java_start(Thread*) () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #4 0x00000037b52079d1 in start_thread () from /lib64/libpthread.so.0 >> #5 0x00000037b4ae88fd in clone () from /lib64/libc.so.6 >> >> *3. jstack -F result* >> ...... >> Thread 26488: (state = NEW) >> Thread 2 (Thread 0x7fe359282700 (LWP 26488)): >> #0 0x00000037b4af805e in __lll_lock_wait_private () from /lib64/libc.so.6 >> #1 0x00000037b4a7cc82 in _L_lock_3495 () from /lib64/libc.so.6 >> #2 0x00000037b4a77903 in arena_get2 () from /lib64/libc.so.6 >> #3 0x00000037b4a7a794 in malloc () from /lib64/libc.so.6 >> #4 0x00000037b4611190 in tls_get_addr_tail () from >> /lib64/ld-linux-x86-64.so.2 >> #5 0x00000037b4611660 in __tls_get_addr () from >> /lib64/ld-linux-x86-64.so.2 >> #6 0x00007fe4adabd762 in Profiler::Handle(int, siginfo*, void*) () from >> /usr/local/lightweight-java-profiler-master/build-64/liblagent.so >> ....... >> Thread 26362: (state = IN_VM) >> Thread 63 (Thread 0x7fe366e5d700 (LWP 26362)): >> #0 0x00000037b520b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from >> /lib64/libpthread.so.0 >> #1 0x00007fe4af005243 in os::PlatformEvent::park() () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #2 0x00007fe4aefcc328 in Monitor::ILock(Thread*) () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #3 0x00007fe4aefcc55f in Monitor::lock_without_safepoint_check() () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #4 0x00007fe4af098127 in SafepointSynchronize::block(JavaThread*) () >> from /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #5 0x00007fe4aedead0b in JavaCallWrapper::JavaCallWrapper(methodHandle, >> Handle, JavaValue*, Thread*) () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #6 0x00007fe4aedeb94a in JavaCalls::call_helper(JavaValue*, >> methodHandle*, JavaCallArguments*, Thread*) () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #7 0x00007fe4aedea5c8 in JavaCalls::call(JavaValue*, methodHandle, >> JavaCallArguments*, Thread*) () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #8 0x00007fe4aedea897 in JavaCalls::call_virtual(JavaValue*, >> KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*) () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #9 0x00007fe4aedea966 in JavaCalls::call_virtual(JavaValue*, Handle, >> KlassHandle, Symbol*, Symbol*, Handle, Thread*) () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> #10 0x00007fe4af10e249 in SystemDictionary::load_instance_class(Symbol*, >> Handle, Thread*) () from >> /usr/java/jdk1.7.0_65/jre/lib/amd64/server/libjvm.so >> >> -- >> You received this message because you are subscribed to the Google Groups >> "mechanical-sympathy" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > >
-- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
