Hi all, Recently, our Spark application's (2.3.1) driver has been crashing before exiting with the following error.
* Could not load hsdis-amd64.so; library not loadable; PrintAssembly is disabled* * #* * # A fatal error has been detected by the Java Runtime Environment:* * #* * # Internal Error (sharedRuntime.cpp:834), pid=40111, tid=0x00002ac46180a700* * # fatal error: exception happened outside interpreter, nmethods and vtable stubs at pc 0x00002ac1a832edb1* * #* * # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build 1.8.0_131-b11)* * # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode linux-amd64 compressed oops)* Similar errors were reported in older JVM and Linux kernel versions https://bugs.openjdk.java.net/browse/JDK-8203612 but we are running on Java 1.8.0_131 and kernel 3.10.0-693.21.1.el7.x86_64. Here is the stack in error report. *Current thread (0x00002ac438005000): JavaThread "block-manager-slave-async-thread-pool-2" daemon [_thread_in_Java, id=40362, stack(0x00002ac46170a000,0x00002ac46180b000)]* *Stack: [0x00002ac46170a000,0x00002ac46180b000], sp=0x00002ac461808cd0, free space=1019k* *Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)* *V [libjvm.so+0xac826a] VMError::report_and_die()+0x2ba* *V [libjvm.so+0x4fd089] report_fatal(char const*, int, char const*)+0x59* *V [libjvm.so+0x9c391a] SharedRuntime::continuation_for_implicit_exception(JavaThread*, unsigned char*, SharedRuntime::ImplicitExceptionKind)+0x33a* *V [libjvm.so+0x92bbfa] JVM_handle_linux_signal+0x48a* *V [libjvm.so+0x921e13] signalHandler(int, siginfo*, void*)+0x43* *C [libpthread.so.0+0xf5d0]* *j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5* *j java.lang.Thread.run()V+11* *v ~StubRoutines::call_stub* *V [libjvm.so+0x691d16] JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056* *V [libjvm.so+0x692221] JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321* *V [libjvm.so+0x6926c7] JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47* *V [libjvm.so+0x72da50] thread_entry(JavaThread*, Thread*)+0xa0* *V [libjvm.so+0xa76833] JavaThread::thread_main_inner()+0x103* *V [libjvm.so+0xa7697c] JavaThread::run()+0x11c* *V [libjvm.so+0x927568] java_start(Thread*)+0x108* *C [libpthread.so.0+0x7dd5]* Has anyone seen this kind of error before ? I could provide more information if needed. Thanks, Manu Zhang