[
https://issues.apache.org/jira/browse/HBASE-26024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17371308#comment-17371308
]
Anoop Sam John commented on HBASE-26024:
----------------------------------------
Your version in use is 2.1.9? And its read workloads? What type of read
cache in use?
As per the stack trace where you see issue which causes SIGSEGV, its while
creating CellBlock stream for writing cells response. This is common for both
RPC server impl.
> Region server JVM Crash - A fatal error has been detected by the Java Runtime
> Environment
> -----------------------------------------------------------------------------------------
>
> Key: HBASE-26024
> URL: https://issues.apache.org/jira/browse/HBASE-26024
> Project: HBase
> Issue Type: Bug
> Affects Versions: 2.1.9
> Reporter: Mohamed Mohideen Meeran
> Priority: Major
> Attachments: error.log
>
>
> Our production Region servers JVM crashed with the following error logs.
>
> Register to memory mapping:
> RAX=0x000000002eea8f42 is an unknown value
> RBX=0x00007f1f8c7900d6 is an unknown value
> RCX=0x0000000000000021 is an unknown value
> RDX=0x0000000000000000 is an unknown value
> RSP=0x00007f1fe3092200 is pointing into the stack for thread:
> 0x00007f29775fb000
> RBP=0x00007f1fe3092200 is pointing into the stack for thread:
> 0x00007f29775fb000
> RSI=0x00007f1f8c7900cc is an unknown value
> RDI=0x000000002eea8f38 is an unknown value
> R8 =0x00007f28e14a3a38 is an oop
> java.nio.DirectByteBuffer
> - klass: 'java/nio/DirectByteBuffer'
> R9 =0x00007f1f8c790094 is an unknown value
> R10=0x00007f2965053400 is at begin+0 in a stub
> StubRoutines::unsafe_arraycopy [0x00007f2965053400, 0x00007f296505343b[ (59
> bytes)
> R11=0x00007f28e14a3a38 is an oop
> java.nio.DirectByteBuffer
> - klass: 'java/nio/DirectByteBuffer'
> R12=
> [error occurred during error reporting (printing register info), id 0xb]
>
> Stack: [0x00007f1fe2f93000,0x00007f1fe3094000], sp=0x00007f1fe3092200, free
> space=1020k
> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native
> code)
> v ~StubRoutines::jshort_disjoint_arraycopy
> J 18388 C2
> org.apache.hadoop.hbase.io.ByteBufferListOutputStream.write(Ljava/nio/ByteBuffer;II)V
> (53 bytes) @ 0x00007f2967fa0ea2 [0x00007f2967fa0d40+0x162]
> J 11722 C2
> org.apache.hadoop.hbase.util.ByteBufferUtils.copyBufferToStream(Ljava/io/OutputStream;Ljava/nio/ByteBuffer;II)V
> (75 bytes) @ 0x00007f29670aa0fc [0x00007f29670a9fa0+0x15c]
> J 13251 C2
> org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I
> (21 bytes) @ 0x00007f2965cbe87c [0x00007f2965cbe820+0x5c]
> J 8703 C2
> org.apache.hadoop.hbase.KeyValueUtil.oswrite(Lorg/apache/hadoop/hbase/Cell;Ljava/io/OutputStream;Z)I
> (259 bytes) @ 0x00007f296684a2d4 [0x00007f296684a140+0x194]
> J 15474 C2
> org.apache.hadoop.hbase.ipc.CellBlockBuilder.buildCellBlockStream(Lorg/apache/hadoop/hbase/codec/Codec;Lorg/apache/hadoop/io/compress/CompressionCodec;Lorg/apache/hadoop/hbase/CellScanner;Lorg/apache/hadoop/hbase/io/ByteBufferPool;)Lorg/apache/hadoop/hbase/io/ByteBufferListOutputStream;
> (75 bytes) @ 0x00007f29675f9dc8 [0x00007f29675f7c80+0x2148]
> J 14260 C2
> org.apache.hadoop.hbase.ipc.ServerCall.setResponse(Lorg/apache/hbase/thirdparty/com/google/protobuf/Message;Lorg/apache/hadoop/hbase/CellScanner;Ljava/lang/Throwable;Ljava/lang/String;)V
> (408 bytes) @ 0x00007f29678ad11c [0x00007f29678acec0+0x25c]
> J 14732 C2 org.apache.hadoop.hbase.ipc.CallRunner.run()V (1376 bytes) @
> 0x00007f296797f690 [0x00007f296797e6a0+0xff0]
> J 14293 C2
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(Lorg/apache/hadoop/hbase/ipc/CallRunner;)V
> (268 bytes) @ 0x00007f29667b7464 [0x00007f29667b72e0+0x184]
> J 17796% C1 org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run()V (72 bytes)
> @ 0x00007f2967c9cbe4 [0x00007f2967c9caa0+0x144]
> v ~StubRoutines::call_stub
> V [libjvm.so+0x65ebbb] JavaCalls::call_helper(JavaValue*, methodHandle*,
> JavaCallArguments*, Thread*)+0x108b
> V [libjvm.so+0x65ffd7] JavaCalls::call_virtual(JavaValue*, KlassHandle,
> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x2f7
> V [libjvm.so+0x660497] JavaCalls::call_virtual(JavaValue*, Handle,
> KlassHandle, Symbol*, Symbol*, Thread*)+0x47
> V [libjvm.so+0x6ada71] thread_entry(JavaThread*, Thread*)+0x91
> V [libjvm.so+0x9f24f1] JavaThread::thread_main_inner()+0xf1
> V [libjvm.so+0x9f26d8] JavaThread::run()+0x1b8
> V [libjvm.so+0x8af502] java_start(Thread*)+0x122
> C [libpthread.so.0+0x7dc5] start_thread+0xc5
>
> we used the workaround by switching *hbase.rpc.server.impl* back to
> SimpleRpcServer as mentioned in the following JIRA
> https://issues.apache.org/jira/browse/HBASE-22539?focusedCommentId=16855688&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16855688
> Also, attached the error logs during JVM crash. Any help?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
