[
https://issues.apache.org/jira/browse/HBASE-28494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17874424#comment-17874424
]
chaijunjie edited comment on HBASE-28494 at 8/17/24 2:37 AM:
-------------------------------------------------------------
When we use HBase 2.4.14, it occour similer....,we have some slow disk, and
before stuck we find some "Slow sync" log, it cost 200s...then all handler
stuck like this...., then we stop the slow DataNode and restart RegionServer,
it recove.
I think HBase should not write the slow datanode....
was (Author: JIRAUSER286971):
When we use HBase 2.4.14, it occour similer....,we have some slow disk, and
before stuck we find some "Slow sync" log, it cost 200s...then all handler
stuck like this....
> "WAL system stuck?" due to threads hang at
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.complete
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: HBASE-28494
> URL: https://issues.apache.org/jira/browse/HBASE-28494
> Project: HBase
> Issue Type: Bug
> Components: regionserver, wal
> Affects Versions: 2.5.5
> Environment: hbase-2.5.5
> hadoop-3.3.6
> kerberos authentication enabled.
> OS: debian 11
> Reporter: Athish Babu
> Priority: Major
> Attachments: RS_thread_dump.txt
>
>
> Currently we come across a issue in write handler threads of a regionserver
> during AsyncFSWAL append operation. We could see regionserver's write handler
> threads is going to WAITING State while acquiring lock for WAL append
> operation at MultiVersionConcurrencyControl.begin
>
> {code:java}
> "RpcServer.default.FPRWQ.Fifo.write.handler=7,queue=3,port=16020" #133 daemon
> prio=5 os_prio=0 tid=0x00007f9301fe7800 nid=0x329a02 runnable
> [0x00007f8a6489a000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:136)
> at
> com.lmax.disruptor.MultiProducerSequencer.next(MultiProducerSequencer.java:105)
> at com.lmax.disruptor.RingBuffer.next(RingBuffer.java:263)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$stampSequenceIdAndPublishToRingBuffer$10(AbstractFSWAL.java:1202)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL$$Lambda$631/875615795.run(Unknown
> Source)
> at
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.begin(MultiVersionConcurrencyControl.java:144)
> - locked <0x00007f8afa4d1a80> (a java.util.LinkedList)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.stampSequenceIdAndPublishToRingBuffer(AbstractFSWAL.java:1201)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.append(AsyncFSWAL.java:647)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$appendData$14(AbstractFSWAL.java:1255)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL$$Lambda$699/1762709833.call(Unknown
> Source)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:216)
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.appendData(AbstractFSWAL.java:1255)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doWALAppend(HRegion.java:7800)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4522)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4446)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4368)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1033)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:951)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:916)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45008)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82) {code}
>
> Other write handler threads are getting BLOCKED state while waiting for above
> lock to get released.
>
> {code:java}
> "RpcServer.default.FPRWQ.Fifo.write.handler=38,queue=2,port=16020" #164
> daemon prio=5 os_prio=0 tid=0x00007f9303147800 nid=0x329a21 waiting for
> monitor entry [0x00007f8a61586000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.complete(MultiVersionConcurrencyControl.java:179)
> - waiting to lock <0x00007f8afa4d1a80> (a java.util.LinkedList)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doWALAppend(HRegion.java:7808)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion.java:4522)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4446)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4368)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1033)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:951)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:916)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2892)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:45008)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:415)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:102)
> at org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82){code}
> Due to above occurence , WAL system getting stucked and facing
> callqueueTooBig Exception
> {code:java}
> ERROR [MemStoreFlusher.0] o.a.h.h.r.MemStoreFlusher - Cache flush failed for
> region
> analytics:RawData,DIO|DISK_USAGE_STATS,1710357005367.55ec1db85903bbf9d2588b6d60b7b66f.
> org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync
> result after 300000 ms for txid=5472571, WAL system stuck?
> at
> org.apache.hadoop.hbase.regionserver.wal.SyncFuture.get(SyncFuture.java:171)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.blockOnSync(AbstractFSWAL.java:869)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.doSync(AsyncFSWAL.java:668)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$sync$1(AbstractFSWAL.java:590)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
> ~[hbase-common-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:590)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.sync(AbstractFSWAL.java:580)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doSyncOfUnflushedWALChanges(HRegion.java:2779)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2721)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2580)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2554)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2424)
> ~[hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:603)
> [hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:572)
> [hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:65)
> [hbase-server-2.5.5.jar:2.5.5]
> at
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:344)
> [hbase-server-2.5.5.jar:2.5.5] {code}
> Issue get fixed in above mentioned regionserver after restarting the process.
> But after sometime again same issue occurs in other regionserver.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
