[
https://issues.apache.org/jira/browse/HBASE-25628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
SweetCandy updated HBASE-25628:
-------------------------------
Description:
Regionserver had a problem which is JVM crash after running for a period of
time. From the error dump logs, the main reason is due to SIGSEGV singal.
BTW, from the start of the regionserver to the crash, there is no error record
in the hbase log
After many crash, there are some common points.
# Thread is AsyncFSWAL-0.
# Problematic frame is v ~StubRoutines::jshort_disjoint_arraycopy
# Error method is
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write.
jdk: Oracle jdk-1.8_181
os: CentOS Linux release 7.4.1708 (Core)
hbase: 2.1.0-cdh6.3.0
{code:java}
# JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build
1.8.0_181-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode linux-amd64
)
# Problematic frame:# v ~StubRoutines::jshort_disjoint_arraycopy
{code}
{code:java}
--------------- T H R E A D ---------------
Current thread (0x00007f4758820800): JavaThread "AsyncFSWAL-0" daemon
[_thread_in_Java, id=17396, stack(0x00007f36f0aa3000,0x00007f36f0ae4000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr:
0x00007f36bc3dd93a
Stack: [0x00007f36f0aa3000,0x00007f36f0ae4000], sp=0x00007f36f0ae2310, free
space=252kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
v ~StubRoutines::jshort_disjoint_arraycopy
J 13350 C2
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write(Ljava/nio/ByteBuffer;II)V
(34 bytes) @ 0x00007f475f113fd1 [0x00007f475f113cc0+0x311]
J 16839 C2
org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I (21
bytes) @ 0x00007f475f4329dc [0x00007f475f432960+0x7c]
J 12594 C2
org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$EnsureKvEncoder.write(Lorg/apache/hadoop/hbase/Cell;)V
(27 bytes) @ 0x00007f475ef80414 [0x00007f475ef7fc20+0x7f4]
VM Arguments:
jvm_args:
-Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p
-Djava.net.preferIPv4Stack=true -Xms33285996544 -Xmx33285996544 -Xmx64g -Xms32g
-Xmn6g -Xss256k
-XX:MaxPermSize=384m -XX:SurvivorRatio=6 -XX:+UseParNewGC
-XX:ParallelGCThreads=10 -XX:+UseConcMarkSweepGC
-XX:ParallelCMSThreads=16 -XX:+CMSParallelRemarkEnabled
-XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70 -XX:CMSMaxAbortablePrecleanTime=500
-XX:CMSFullGCsBeforeCompaction=5 -XX:+CMSClassUnloadingEnabled
-XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/log/tmp/hbase/hbase_hbase-REGIONSERVER-58f025f30c5505722d8dee21c92b9469_pid15967.hprof
-XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh
-Dhbase.log.dir=/log/hbase
-Dhbase.log.file=hbase-cmf-hbase-REGIONSERVER-spbsjzy05.log.out
-Dhbase.home.dir=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase
-Dhbase.id.str= -Dhbase.root.logger=INFO,RFA
-Djava.library.path=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase/lib/native/Linux-amd64-64
-Dhbase.security.logger=INFO,RFAS
java_command: org.apache.hadoop.hbase.regionserver.HRegionServer start
{code}
was:
Regionserver had a problem which is JVM crash after running for a period of
time. From the error dump logs, the main reason is due to SIGSEGV singal. how
to deal with it.
BTW, from the start of the regionserver to the crash, there is no error record
in the hbase log
After many crash, there are some common points.
# Thread is AsyncFSWAL-0.
# Problematic frame is v ~StubRoutines::jshort_disjoint_arraycopy
# Error method is
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write.
jdk: Oracle jdk-1.8_181
os: CentOS Linux release 7.4.1708 (Core)
hbase: 2.1.0-cdh6.3.0
{code:java}
# JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build
1.8.0_181-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode linux-amd64
)
# Problematic frame:# v ~StubRoutines::jshort_disjoint_arraycopy
{code}
{code:java}
--------------- T H R E A D ---------------
Current thread (0x00007f4758820800): JavaThread "AsyncFSWAL-0" daemon
[_thread_in_Java, id=17396, stack(0x00007f36f0aa3000,0x00007f36f0ae4000)]
siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr:
0x00007f36bc3dd93a
Stack: [0x00007f36f0aa3000,0x00007f36f0ae4000], sp=0x00007f36f0ae2310, free
space=252kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code,
C=native code)
v ~StubRoutines::jshort_disjoint_arraycopy
J 13350 C2
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write(Ljava/nio/ByteBuffer;II)V
(34 bytes) @ 0x00007f475f113fd1 [0x00007f475f113cc0+0x311]
J 16839 C2
org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I (21
bytes) @ 0x00007f475f4329dc [0x00007f475f432960+0x7c]
J 12594 C2
org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$EnsureKvEncoder.write(Lorg/apache/hadoop/hbase/Cell;)V
(27 bytes) @ 0x00007f475ef80414 [0x00007f475ef7fc20+0x7f4]
VM Arguments:
jvm_args:
-Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p
-Djava.net.preferIPv4Stack=true -Xms33285996544 -Xmx33285996544 -Xmx64g -Xms32g
-Xmn6g -Xss256k
-XX:MaxPermSize=384m -XX:SurvivorRatio=6 -XX:+UseParNewGC
-XX:ParallelGCThreads=10 -XX:+UseConcMarkSweepGC
-XX:ParallelCMSThreads=16 -XX:+CMSParallelRemarkEnabled
-XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSInitiatingOccupancyFraction=70 -XX:CMSMaxAbortablePrecleanTime=500
-XX:CMSFullGCsBeforeCompaction=5 -XX:+CMSClassUnloadingEnabled
-XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/log/tmp/hbase/hbase_hbase-REGIONSERVER-58f025f30c5505722d8dee21c92b9469_pid15967.hprof
-XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh
-Dhbase.log.dir=/log/hbase
-Dhbase.log.file=hbase-cmf-hbase-REGIONSERVER-spbsjzy05.log.out
-Dhbase.home.dir=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase
-Dhbase.id.str= -Dhbase.root.logger=INFO,RFA
-Djava.library.path=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase/lib/native/Linux-amd64-64
-Dhbase.security.logger=INFO,RFAS
java_command: org.apache.hadoop.hbase.regionserver.HRegionServer start
{code}
> Regionserver crash occasionally due to SIGSEGV
> ----------------------------------------------
>
> Key: HBASE-25628
> URL: https://issues.apache.org/jira/browse/HBASE-25628
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 2.1.0
> Reporter: SweetCandy
> Priority: Major
> Attachments: hs_err_pid15967.log, hs_err_pid3375.log
>
>
> Regionserver had a problem which is JVM crash after running for a period of
> time. From the error dump logs, the main reason is due to SIGSEGV singal.
> BTW, from the start of the regionserver to the crash, there is no error
> record in the hbase log
> After many crash, there are some common points.
> # Thread is AsyncFSWAL-0.
> # Problematic frame is v ~StubRoutines::jshort_disjoint_arraycopy
> # Error method is
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write.
> jdk: Oracle jdk-1.8_181
> os: CentOS Linux release 7.4.1708 (Core)
> hbase: 2.1.0-cdh6.3.0
>
> {code:java}
> # JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build
> 1.8.0_181-b13)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode
> linux-amd64 )
> # Problematic frame:# v ~StubRoutines::jshort_disjoint_arraycopy
> {code}
>
> {code:java}
> --------------- T H R E A D ---------------
> Current thread (0x00007f4758820800): JavaThread "AsyncFSWAL-0" daemon
> [_thread_in_Java, id=17396, stack(0x00007f36f0aa3000,0x00007f36f0ae4000)]
> siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr:
> 0x00007f36bc3dd93a
> Stack: [0x00007f36f0aa3000,0x00007f36f0ae4000], sp=0x00007f36f0ae2310, free
> space=252kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code,
> C=native code)
> v ~StubRoutines::jshort_disjoint_arraycopy
> J 13350 C2
> org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter$OutputStreamWrapper.write(Ljava/nio/ByteBuffer;II)V
> (34 bytes) @ 0x00007f475f113fd1 [0x00007f475f113cc0+0x311]
> J 16839 C2
> org.apache.hadoop.hbase.ByteBufferKeyValue.write(Ljava/io/OutputStream;Z)I
> (21 bytes) @ 0x00007f475f4329dc [0x00007f475f432960+0x7c]
> J 12594 C2
> org.apache.hadoop.hbase.regionserver.wal.WALCellCodec$EnsureKvEncoder.write(Lorg/apache/hadoop/hbase/Cell;)V
> (27 bytes) @ 0x00007f475ef80414 [0x00007f475ef7fc20+0x7f4]
> VM Arguments:
> jvm_args:
> -Dproc_regionserver -XX:OnOutOfMemoryError=kill -9 %p
> -Djava.net.preferIPv4Stack=true -Xms33285996544 -Xmx33285996544 -Xmx64g
> -Xms32g -Xmn6g -Xss256k
> -XX:MaxPermSize=384m -XX:SurvivorRatio=6 -XX:+UseParNewGC
> -XX:ParallelGCThreads=10 -XX:+UseConcMarkSweepGC
> -XX:ParallelCMSThreads=16 -XX:+CMSParallelRemarkEnabled
> -XX:+UseCMSCompactAtFullCollection -XX:+UseCMSInitiatingOccupancyOnly
> -XX:CMSInitiatingOccupancyFraction=70 -XX:CMSMaxAbortablePrecleanTime=500
> -XX:CMSFullGCsBeforeCompaction=5 -XX:+CMSClassUnloadingEnabled
> -XX:+HeapDumpOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=/log/tmp/hbase/hbase_hbase-REGIONSERVER-58f025f30c5505722d8dee21c92b9469_pid15967.hprof
> -XX:OnOutOfMemoryError=/opt/cloudera/cm-agent/service/common/killparent.sh
> -Dhbase.log.dir=/log/hbase
> -Dhbase.log.file=hbase-cmf-hbase-REGIONSERVER-spbsjzy05.log.out
> -Dhbase.home.dir=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase
> -Dhbase.id.str= -Dhbase.root.logger=INFO,RFA
> -Djava.library.path=/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-6.3.0-1.cdh6.3.0.p0.1279813/lib/hbase/lib/native/Linux-amd64-64
> -Dhbase.security.logger=INFO,RFAS
> java_command: org.apache.hadoop.hbase.regionserver.HRegionServer start
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
