[ https://issues.apache.org/jira/browse/HBASE-25628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17299509#comment-17299509 ]
Cheng.Xiong commented on HBASE-25628: ------------------------------------- https://issues.apache.org/jira/browse/HBASE-22539 may be related to this issue. > Regionserver crash occasionally due to SIGSEGV > ---------------------------------------------- > > Key: HBASE-25628 > URL: https://issues.apache.org/jira/browse/HBASE-25628 > Project: HBase > Issue Type: Bug > Components: regionserver > Affects Versions: 2.1.0 > Reporter: Cheng.Xiong > Priority: Major > Attachments: hs_err_pid15967.log, hs_err_pid3375.log > > > Regionserver had a problem which is JVM crash after running for a period of > time. From the error dump logs, the main reason is due to SIGSEGV singal. > BTW, from the start of the regionserver to the crash, there is no error > record in the hbase log > After many crash, there are some common points. > # Thread is AsyncFSWAL-0. > # Problematic frame is v ~StubRoutines::jshort_disjoint_arraycopy > # Error method is > org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write. > jdk: Oracle jdk-1.8_181 > os: CentOS Linux release 7.4.1708 (Core) > hbase: 2.1.0-cdh6.3.0 > > {code:java} > # JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build > 1.8.0_181-b13) > # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode > linux-amd64 ) > # Problematic frame:# v ~StubRoutines::jshort_disjoint_arraycopy > {code} > > {code:java} > --------------- T H R E A D --------------- > Current thread (0x00007f4758820800): JavaThread "AsyncFSWAL-0" daemon > [_thread_in_Java, id=17396, stack(0x00007f36f0aa3000,0x00007f36f0ae4000)] > siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: > 0x00007f36bc3dd93a > Stack: [0x00007f36f0aa3000,0x00007f36f0ae4000], sp=0x00007f36f0ae2310, free > space=252kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, > C=native code) > v ~StubRoutines::jshort_disjoint_arraycopy > J 13350 C2 > org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write(Ljava/nio/ByteBuffer;II)V > (34 bytes) @ 0x00007f475f113fd1 [0x00007f475f113cc0+0x311] > J 16839 C2 > org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I > (21 bytes) @ 0x00007f475f4329dc [0x00007f475f432960+0x7c] > J 12594 C2 > org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$EnsureKvEncoder.write(Lorg/apache/hadoop/hbase/Cell;)V > (27 bytes) @ 0x00007f475ef80414 [0x00007f475ef7fc20+0x7f4] > VM Arguments: > jvm_args: > -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p > -Djava.net.preferIPv4Stack=true -Xms33285996544 -Xmx33285996544 -Xmx64g > -Xms32g -Xmn6g -Xss256k > -XX:MaxPermSize=384m -XX:SurvivorRatio=6 -XX:+UseParNewGC > -XX:ParallelGCThreads=10 -XX:+UseConcMarkSweepGC > -XX:ParallelCMSThreads=16 -XX:+CMSParallelRemarkEnabled > -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly > -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSMaxAbortablePrecleanTime=500 > -XX:CMSFullGCsBeforeCompaction=5 -XX:+CMSClassUnloadingEnabled > -XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError > -XX:HeapDumpPath=/log/tmp/hbase/hbase_hbase-REGIONSERVER-58f025f30c5505722d8dee21c92b9469_pid15967.hprof > -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh > -Dhbase.log.dir=/log/hbase > -Dhbase.log.file=hbase-cmf-hbase-REGIONSERVER-spbsjzy05.log.out > -Dhbase.home.dir=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase > -Dhbase.id.str= -Dhbase.root.logger=INFO,RFA > -Djava.library.path=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase/lib/native/Linux-amd64-64 > -Dhbase.security.logger=INFO,RFAS > java_command: org.apache.hadoop.hbase.regionserver.HRegionServer start > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)