I am trying to profile an application with callgrind. The application uses
hadoop's libhdfs api which in turn makes JNI calls to a JVM. This causes a
crash in the JVM. When running normally the application seems to work fine.
I am running on Centos 6.7 with Valgrind 3.11.0. Java is 1.8 Oracle JDK (I
have also tried open jdk without much luck).
Here is my commandline:
valgrind -v --tool=callgrind --dump-instr=yes --trace-jump=yes
--trace-children=yes --smc-check=all --collect-jumps=yes
--simulate-cache=yes ./example
Below are some of the output from valgrind:
[CodeBlob (0x00000000064d8f50)]
Framesize: 84
SafepointBlob
Could not load hsdis-amd64.so; library not loadable; PrintAssembly is
disabled
#
# A fatal error has been detected by the Java Runtime Environment:
#
# Internal Error (sharedRuntime.cpp:834), pid=37623, tid=0x0000000005c138e0
# fatal error: exception happened outside interpreter, nmethods and vtable
stubs at pc 0x00000000064d9007
#
# JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build
1.8.0_92-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.92-b14 mixed mode
linux-amd64 compressed oops)
# Core dump written. Default location: /home/myuser/libhdfs/core or
core.37623
#
# An error report file with more information is saved as:
# /home/myuser/libhdfs/hs_err_pid37623.log
==37623== brk segment overflow in thread #1: can't grow to 0x4900000
==37623== brk segment overflow in thread #1: can't grow to 0x49a7000
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
==37623==
==37623== Process terminating with default action of signal 6 (SIGABRT):
dumping core
==37623== at 0x3127032625: raise (in /lib64/libc-2.12.so)
==37623== by 0x3127033E04: abort (in /lib64/libc-2.12.so)
==37623== by 0x554A604: os::abort(bool) (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623== by 0x56E9A62: VMError::report_and_die() (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623== by 0x5127098: report_fatal(char const*, int, char const*) (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623== by 0x55E7479:
SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned
char*, SharedRuntime::ImplicitExceptionKind) (in /usr/java/jdk1.8.0_92
/jre/lib/amd64/server/libjvm.so)
==37623== by 0x5550169: JVM_handle_linux_signal (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623== by 0x55465C2: signalHandler(int, siginfo*, void*) (in
/usr/java/jdk1.8.0_92/jre/lib/amd64/server/libjvm.so)
==37623== by 0x312703269F: ??? (in /lib64/libc-2.12.so)
==37623== by 0x64D9006: ???
==37623== by 0x44: ???
==37623== by 0x649A03F: ???
----------
Here is the stack trace from jvm error log:
Current thread (0x0000000004022800): JavaThread "main" [_thread_in_Java,
id=37623, stack(0x0000000ffef01000,0x0000000fff001000)]
Stack: [0x0000000ffef01000,0x0000000fff001000], sp=0x0000000ffeff1ae0,
free space=962k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
code)
V [libjvm.so+0xabd65a] VMError::report_and_die()+0x2ba
V [libjvm.so+0x4fb099] report_fatal(char const*, int, char const*)+0x59
V [libjvm.so+0x9bb47a]
SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned
char*, SharedRuntime::ImplicitExceptionKind)+0x33a
V [libjvm.so+0x92416a] JVM_handle_linux_signal+0x48a
V [libjvm.so+0x91a5c3] signalHandler(int, siginfo*, void*)+0x43
C [libc.so.6+0x326a0]
C 0x0000000000000045
j java.net.URLClassLoader$1.run()Ljava/lang/Object;+1
v ~StubRoutines::call_stub
V [libjvm.so+0x68e746] JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x7275bc] JVM_DoPrivileged+0x27c
j
java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0
j java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13
J 327 C1
java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122
bytes) @ 0x000000000663a8e4 [0x0000000006639f80+0x964]
j
sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+81
j java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
v ~StubRoutines::call_stub
V [libjvm.so+0x68e746] JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x68ec51] JavaCalls::call_virtual(JavaValue*, KlassHandle,
Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V [libjvm.so+0x68f0a6] JavaCalls::call_virtual(JavaValue*, Handle,
KlassHandle, Symbol*, Symbol*, Handle, Thread*)+0x56
V [libjvm.so+0xa349a0] SystemDictionary::load_instance_class(Symbol*,
Handle, Thread*)+0x3f0
V [libjvm.so+0xa3388c]
SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle,
Thread*)+0x7cc
V [libjvm.so+0xa34dc3] SystemDictionary::resolve_or_fail(Symbol*, Handle,
Handle, bool, Thread*)+0x33
V [libjvm.so+0x70cd6e] find_class_from_class_loader(JNIEnv_*, Symbol*,
unsigned char, Handle, Handle, unsigned char, Thread*)+0x3e
V [libjvm.so+0x7142b1] JVM_FindClassFromCaller+0x2e1
C [libjava.so+0xe2d0] Java_java_lang_Class_forName0+0x130
j
java.lang.Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;Ljava/lang/Class;)Ljava/lang/Class;+0
j
java.lang.Class.forName(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class;+49
j
com.sun.beans.finder.ClassFinder.findClass(Ljava/lang/String;Ljava/lang/ClassLoader;)Ljava/lang/Class;+11
j
com.sun.beans.finder.InstanceFinder.instantiate(Ljava/lang/Class;Ljava/lang/String;)Ljava/lang/Object;+13
j
com.sun.beans.finder.InstanceFinder.find(Ljava/lang/Class;)Ljava/lang/Object;+34
j
java.beans.Introspector.findExplicitBeanInfo(Ljava/lang/Class;)Ljava/beans/BeanInfo;+7
j java.beans.Introspector.<init>(Ljava/lang/Class;Ljava/lang/Class;I)V+121
j
java.beans.Introspector.getBeanInfo(Ljava/lang/Class;)Ljava/beans/BeanInfo;+60
j
java.beans.Introspector.getBeanInfo(Ljava/lang/Class;Ljava/lang/Class;I)Ljava/beans/BeanInfo;+10
j java.beans.Introspector.<init>(Ljava/lang/Class;Ljava/lang/Class;I)V+157
j
java.beans.Introspector.getBeanInfo(Ljava/lang/Class;)Ljava/beans/BeanInfo;+60
j
java.beans.Introspector.getBeanInfo(Ljava/lang/Class;Ljava/lang/Class;I)Ljava/beans/BeanInfo;+10
j java.beans.Introspector.<init>(Ljava/lang/Class;Ljava/lang/Class;I)V+157
j
java.beans.Introspector.getBeanInfo(Ljava/lang/Class;)Ljava/beans/BeanInfo;+60
j org.apache.log4j.config.PropertySetter.introspect()V+7
j
org.apache.log4j.config.PropertySetter.getPropertyDescriptor(Ljava/lang/String;)Ljava/beans/PropertyDescriptor;+8
j
org.apache.log4j.config.PropertySetter.setProperties(Ljava/util/Properties;Ljava/lang/String;)V+113
j
org.apache.log4j.config.PropertySetter.setProperties(Ljava/lang/Object;Ljava/util/Properties;Ljava/lang/String;)V+10
j
org.apache.log4j.PropertyConfigurator.parseAppender(Ljava/util/Properties;Ljava/lang/String;)Lorg/apache/log4j/Appender;+694
j
org.apache.log4j.PropertyConfigurator.parseCategory(Ljava/util/Properties;Lorg/apache/log4j/Logger;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V+280
j
org.apache.log4j.PropertyConfigurator.configureRootCategory(Ljava/util/Properties;Lorg/apache/log4j/spi/LoggerRepository;)V+63
j
org.apache.log4j.PropertyConfigurator.doConfigure(Ljava/util/Properties;Lorg/apache/log4j/spi/LoggerRepository;)V+134
j
org.apache.log4j.PropertyConfigurator.doConfigure(Ljava/net/URL;Lorg/apache/log4j/spi/LoggerRepository;)V+246
j
org.apache.log4j.helpers.OptionConverter.selectAndConfigure(Ljava/net/URL;Ljava/lang/String;Lorg/apache/log4j/spi/LoggerRepository;)V+129
j org.apache.log4j.LogManager.<clinit>()V+156
v ~StubRoutines::call_stub
V [libjvm.so+0x68e746] JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x640807]
InstanceKlass::call_class_initializer_impl(instanceKlassHandle,
Thread*)+0xd7
V [libjvm.so+0x642cfc]
InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x1ac
V [libjvm.so+0x6430c1] InstanceKlass::initialize(Thread*)+0x41
V [libjvm.so+0x7f7df6] LinkResolver::resolve_static_call(CallInfo&,
KlassHandle&, Symbol*, Symbol*, KlassHandle, bool, bool, Thread*)+0x246
V [libjvm.so+0x7f807f] LinkResolver::resolve_invokestatic(CallInfo&,
constantPoolHandle, int, Thread*)+0x23f
V [libjvm.so+0x7f9131] LinkResolver::resolve_invoke(CallInfo&, Handle,
constantPoolHandle, int, Bytecodes::Code, Thread*)+0x4f1
V [libjvm.so+0x687cc2] InterpreterRuntime::resolve_invoke(JavaThread*,
Bytecodes::Code)+0x1b2
j
org.apache.log4j.Logger.getLogger(Ljava/lang/String;)Lorg/apache/log4j/Logger;+1
j
org.apache.commons.logging.impl.Log4JLogger.getLogger()Lorg/apache/log4j/Logger;+27
j
org.apache.commons.logging.impl.Log4JLogger.<init>(Ljava/lang/String;)V+16
v ~StubRoutines::call_stub
V [libjvm.so+0x68e746] JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x995797] Reflection::invoke(instanceKlassHandle,
methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle,
bool, Thread*)+0x5d7
V [libjvm.so+0x996413] Reflection::invoke_constructor(oopDesc*,
objArrayHandle, Thread*)+0x313
V [libjvm.so+0x71cf5a] JVM_NewInstanceFromConstructor+0x10a
j
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Ljava/lang/reflect/Constructor;[Ljava/lang/Object;)Ljava/lang/Object;+0
j
sun.reflect.NativeConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;+85
j
sun.reflect.DelegatingConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;+5
j
java.lang.reflect.Constructor.newInstance([Ljava/lang/Object;)Ljava/lang/Object;+79
j
org.apache.commons.logging.impl.LogFactoryImpl.createLogFromClass(Ljava/lang/String;Ljava/lang/String;Z)Lorg/apache/commons/logging/Log;+397
j
org.apache.commons.logging.impl.LogFactoryImpl.discoverLogImplementation(Ljava/lang/String;)Lorg/apache/commons/logging/Log;+187
j
org.apache.commons.logging.impl.LogFactoryImpl.newInstance(Ljava/lang/String;)Lorg/apache/commons/logging/Log;+9
j
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(Ljava/lang/String;)Lorg/apache/commons/logging/Log;+18
j
org.apache.commons.logging.impl.LogFactoryImpl.getInstance(Ljava/lang/Class;)Lorg/apache/commons/logging/Log;+5
j
org.apache.commons.logging.LogFactory.getLog(Ljava/lang/Class;)Lorg/apache/commons/logging/Log;+4
j org.apache.hadoop.fs.FileSystem.<clinit>()V+3
v ~StubRoutines::call_stub
V [libjvm.so+0x68e746] JavaCalls::call_helper(JavaValue*, methodHandle*,
JavaCallArguments*, Thread*)+0x1056
V [libjvm.so+0x640807]
InstanceKlass::call_class_initializer_impl(instanceKlassHandle,
Thread*)+0xd7
V [libjvm.so+0x642cfc]
InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x1ac
V [libjvm.so+0x6430c1] InstanceKlass::initialize(Thread*)+0x41
V [libjvm.so+0x70cdab] find_class_from_class_loader(JNIEnv_*, Symbol*,
unsigned char, Handle, Handle, unsigned char, Thread*)+0x7b
V [libjvm.so+0x6e4874] jni_FindClass+0x424
C [libhdfs.so.0.0.0+0x364c] globalClassReference+0xac
----
Any pointers on how I can make this work?
------------------------------------------------------------------------------
Attend Shape: An AT&T Tech Expo July 15-16. Meet us at AT&T Park in San
Francisco, CA to explore cutting-edge tech and listen to tech luminaries
present their vision of the future. This family event has something for
everyone, including kids. Get more information and register today.
http://sdm.link/attshape
_______________________________________________
Valgrind-users mailing list
Valgrind-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/valgrind-users
