Master status for masterserver.gmd9.intern,16000,1713515965162 as of Fri
Apr 19 10:55:22 CEST 2024
Version Info:
===========================================================
HBase 2.5.7
Source code repository
git://buildbox.localdomain/home/apurtell/tmp/RM/hbase
revision=6788f98356dd70b4a7ff766ea7a8298e022e7b95
Compiled by apurtell on Thu Dec 14 15:59:16 PST 2023
>From source with checksum
1501d7fdf72398791ee335a229d099fc972cea7c2a952da7622eb087ddf975361f107cbbbee5d0ad6f603466e9afa1f4fd242ffccbd4371eb0b56059bb3b5402
Hadoop 2.10.2
Source code repository Unknown
revision=965fd380006fa78b2315668fbc7eb432e1d8200f
Compiled by ubuntu on 2022-05-25T00:12Z
Tasks:
===========================================================
Task: Master startup
Status: RUNNING:Starting assignment manager
Running for 954s
Task: Flushing master:store,,1.1595e783b53d99cd5eef43b6debb2682.
Status: COMPLETE:Flush successful flush result:CANNOT_FLUSH_MEMSTORE_EMPTY,
failureReason:Nothing to flush,flush seq id14
Completed 49s ago
Ran for 0s
Task: RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000
Status: WAITING:Waiting for a call
Running for 951s
Task: RpcServer.priority.RWQ.Fifo.write.handler=1,queue=0,port=16000
Status: WAITING:Waiting for a call
Running for 951s
Servers:
===========================================================
servername1ct.gmd9.intern,16020,1713514863737: requestsPerSecond=0.0,
numberOfOnlineRegions=0, usedHeapMB=37.0MB, maxHeapMB=2966.0MB,
numberOfStores=0, numberOfStorefiles=0, storeRefCount=0,
maxCompactedStoreFileRefCount=0, storefileUncompressedSizeMB=0,
storefileSizeMB=0, memstoreSizeMB=0, readRequestsCount=0,
filteredReadRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=0,
totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
servername2ct.gmd9.intern,16020,1713514925960: requestsPerSecond=0.0,
numberOfOnlineRegions=0, usedHeapMB=20.0MB, maxHeapMB=2966.0MB,
numberOfStores=0, numberOfStorefiles=0, storeRefCount=0,
maxCompactedStoreFileRefCount=0, storefileUncompressedSizeMB=0,
storefileSizeMB=0, memstoreSizeMB=0, readRequestsCount=0,
filteredReadRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=0,
totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
servername3ct.gmd9.intern,16020,1713514937151: requestsPerSecond=0.0,
numberOfOnlineRegions=0, usedHeapMB=67.0MB, maxHeapMB=2966.0MB,
numberOfStores=0, numberOfStorefiles=0, storeRefCount=0,
maxCompactedStoreFileRefCount=0, storefileUncompressedSizeMB=0,
storefileSizeMB=0, memstoreSizeMB=0, readRequestsCount=0,
filteredReadRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=0,
totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
servername4ct.gmd9.intern,16020,1713514968019: requestsPerSecond=0.0,
numberOfOnlineRegions=0, usedHeapMB=24.0MB, maxHeapMB=2966.0MB,
numberOfStores=0, numberOfStorefiles=0, storeRefCount=0,
maxCompactedStoreFileRefCount=0, storefileUncompressedSizeMB=0,
storefileSizeMB=0, memstoreSizeMB=0, readRequestsCount=0,
filteredReadRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=0,
totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
servername5ct.gmd9.intern,16020,1713514979294: requestsPerSecond=0.0,
numberOfOnlineRegions=0, usedHeapMB=58.0MB, maxHeapMB=2966.0MB,
numberOfStores=0, numberOfStorefiles=0, storeRefCount=0,
maxCompactedStoreFileRefCount=0, storefileUncompressedSizeMB=0,
storefileSizeMB=0, memstoreSizeMB=0, readRequestsCount=0,
filteredReadRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=0,
totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
servername6ct.gmd9.intern,16020,1713514994770: requestsPerSecond=0.0,
numberOfOnlineRegions=0, usedHeapMB=31.0MB, maxHeapMB=2966.0MB,
numberOfStores=0, numberOfStorefiles=0, storeRefCount=0,
maxCompactedStoreFileRefCount=0, storefileUncompressedSizeMB=0,
storefileSizeMB=0, memstoreSizeMB=0, readRequestsCount=0,
filteredReadRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=0,
totalStaticIndexSizeKB=0, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
currentCompactedKVs=0, compactionProgressPct=NaN, coprocessors=[]
Regions-in-transition:
===========================================================
Executors:
===========================================================
Status for executor:
Executor-4-MASTER_META_SERVER_OPERATIONS-master/masterserver:16000
=======================================
0 events queued, 0 running
Status for executor:
Executor-6-MASTER_SNAPSHOT_OPERATIONS-master/masterserver:16000
=======================================
0 events queued, 0 running
Status for executor:
Executor-3-MASTER_SERVER_OPERATIONS-master/masterserver:16000
=======================================
0 events queued, 0 running
Status for executor: Executor-5-M_LOG_REPLAY_OPS-master/masterserver:16000
=======================================
0 events queued, 0 running
Status for executor:
Executor-2-MASTER_CLOSE_REGION-master/masterserver:16000
=======================================
0 events queued, 0 running
Status for executor:
Executor-7-MASTER_MERGE_OPERATIONS-master/masterserver:16000
=======================================
0 events queued, 0 running
Status for executor:
Executor-8-MASTER_TABLE_OPERATIONS-master/masterserver:16000
=======================================
0 events queued, 0 running
Status for executor:
Executor-1-MASTER_OPEN_REGION-master/masterserver:16000
=======================================
0 events queued, 0 running
Stacks:
===========================================================
Process Thread Dump:
131 active threads
Thread 186 (WAL-Archive-0):
State: WAITING
Blocked count: 5
Waited count: 11
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@42f44d41
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 185 (Close-WAL-Writer-0):
State: TIMED_WAITING
Blocked count: 2
Waited count: 6
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 152 (Session-Scheduler-3bc4ef12-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 151
(master/masterserver:16000:becomeActiveMaster-HFileCleaner.small.0-1713515973400):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58626ec5
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549)
org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:285)
org.apache.hadoop.hbase.master.cleaner.HFileCleaner$2.run(HFileCleaner.java:269)
Thread 150
(master/masterserver:16000:becomeActiveMaster-HFileCleaner.large.0-1713515973400):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18916420
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
org.apache.hadoop.hbase.util.StealJobQueue.take(StealJobQueue.java:101)
org.apache.hadoop.hbase.master.cleaner.HFileCleaner.consumerLoop(HFileCleaner.java:285)
org.apache.hadoop.hbase.master.cleaner.HFileCleaner$1.run(HFileCleaner.java:254)
Thread 149 (snapshot-hfile-cleaner-cache-refresher):
State: TIMED_WAITING
Blocked count: 4
Waited count: 11
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 148 (master/masterserver:16000.Chore.1):
State: TIMED_WAITING
Blocked count: 2
Waited count: 10
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 147 (OldWALsCleaner-1):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a6a3b7e
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:172)
org.apache.hadoop.hbase.master.cleaner.LogCleaner.lambda$createOldWalsCleaner$1(LogCleaner.java:152)
org.apache.hadoop.hbase.master.cleaner.LogCleaner$$Lambda$494/556458560.run(Unknown
Source)
java.lang.Thread.run(Thread.java:750)
Thread 146 (OldWALsCleaner-0):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7a6a3b7e
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.master.cleaner.LogCleaner.deleteFile(LogCleaner.java:172)
org.apache.hadoop.hbase.master.cleaner.LogCleaner.lambda$createOldWalsCleaner$1(LogCleaner.java:152)
org.apache.hadoop.hbase.master.cleaner.LogCleaner$$Lambda$494/556458560.run(Unknown
Source)
java.lang.Thread.run(Thread.java:750)
Thread 139 (PEWorker-16):
State: TIMED_WAITING
Blocked count: 0
Waited count: 16
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 138 (PEWorker-15):
State: TIMED_WAITING
Blocked count: 0
Waited count: 16
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 137 (PEWorker-14):
State: TIMED_WAITING
Blocked count: 0
Waited count: 16
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 136 (PEWorker-13):
State: TIMED_WAITING
Blocked count: 0
Waited count: 16
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 135 (PEWorker-12):
State: TIMED_WAITING
Blocked count: 0
Waited count: 16
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 134 (PEWorker-11):
State: TIMED_WAITING
Blocked count: 0
Waited count: 17
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 133 (PEWorker-10):
State: TIMED_WAITING
Blocked count: 0
Waited count: 16
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 132 (PEWorker-9):
State: TIMED_WAITING
Blocked count: 0
Waited count: 17
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 131 (PEWorker-8):
State: TIMED_WAITING
Blocked count: 0
Waited count: 18
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 130 (PEWorker-7):
State: TIMED_WAITING
Blocked count: 0
Waited count: 16
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 129 (PEWorker-6):
State: TIMED_WAITING
Blocked count: 0
Waited count: 18
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 128 (PEWorker-5):
State: TIMED_WAITING
Blocked count: 0
Waited count: 18
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 127 (PEWorker-4):
State: TIMED_WAITING
Blocked count: 0
Waited count: 17
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 126 (PEWorker-3):
State: TIMED_WAITING
Blocked count: 0
Waited count: 18
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 125 (PEWorker-2):
State: TIMED_WAITING
Blocked count: 0
Waited count: 21
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 124 (PEWorker-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 16
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:165)
org.apache.hadoop.hbase.procedure2.AbstractProcedureScheduler.poll(AbstractProcedureScheduler.java:147)
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:2113)
Thread 123 (WorkerMonitor):
State: TIMED_WAITING
Blocked count: 0
Waited count: 191
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.DelayQueue.poll(DelayQueue.java:273)
org.apache.hadoop.hbase.procedure2.util.DelayedUtil.takeWithoutInterrupt(DelayedUtil.java:81)
org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:56)
Thread 122 (ProcExecTimeout):
State: TIMED_WAITING
Blocked count: 0
Waited count: 64
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.DelayQueue.poll(DelayQueue.java:268)
org.apache.hadoop.hbase.procedure2.util.DelayedUtil.takeWithoutInterrupt(DelayedUtil.java:81)
org.apache.hadoop.hbase.procedure2.TimeoutExecutorThread.run(TimeoutExecutorThread.java:56)
Thread 145 (ActiveMasterInitializationMonitor-1713515973319):
State: TIMED_WAITING
Blocked count: 0
Waited count: 2
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hbase.master.MasterInitializationMonitor.run(MasterInitializationMonitor.java:63)
java.lang.Thread.run(Thread.java:750)
Thread 143 (SnapshotHandlerChoreCleaner):
State: TIMED_WAITING
Blocked count: 0
Waited count: 95
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 142 (normalizer-worker-0):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1ac17b12
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerWorkQueue.take(RegionNormalizerWorkQueue.java:146)
org.apache.hadoop.hbase.master.normalizer.RegionNormalizerWorker.run(RegionNormalizerWorker.java:191)
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.util.concurrent.FutureTask.run(FutureTask.java:266)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 141 (masterserver:16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@20c47452
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
org.apache.hadoop.hbase.master.assignment.AssignmentManager.waitOnAssignQueue(AssignmentManager.java:2195)
org.apache.hadoop.hbase.master.assignment.AssignmentManager.processAssignQueue(AssignmentManager.java:2217)
org.apache.hadoop.hbase.master.assignment.AssignmentManager.access$600(AssignmentManager.java:109)
org.apache.hadoop.hbase.master.assignment.AssignmentManager$1.run(AssignmentManager.java:2157)
Thread 140 (ProcedureDispatcherTimeoutThread):
State: TIMED_WAITING
Blocked count: 0
Waited count: 48
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
org.apache.hadoop.hbase.procedure2.util.DelayedUtil.takeWithoutInterrupt(DelayedUtil.java:81)
org.apache.hadoop.hbase.procedure2.RemoteProcedureDispatcher$TimeoutExecutorThread.run(RemoteProcedureDispatcher.java:320)
Thread 121 (Idle-Rpc-Conn-Sweeper-pool-0):
State: TIMED_WAITING
Blocked count: 0
Waited count: 8
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 119 (master:store-Flusher):
State: TIMED_WAITING
Blocked count: 0
Waited count: 3
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
org.apache.hadoop.hbase.master.region.MasterRegionFlusherAndCompactor.flushLoop(MasterRegionFlusherAndCompactor.java:193)
org.apache.hadoop.hbase.master.region.MasterRegionFlusherAndCompactor$$Lambda$433/1001275970.run(Unknown
Source)
java.lang.Thread.run(Thread.java:750)
Thread 114 (AsyncFSWAL-0-hdfs://masterserver:9000/hbase/MasterData):
State: WAITING
Blocked count: 0
Waited count: 3
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a093bfa
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 113 (Connector-Scheduler-5ec5ea63-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 36
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 112 (prometheus-http-1-1):
State: WAITING
Blocked count: 0
Waited count: 64
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f7dcfc6
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 111 (RS-EventLoopGroup-1-7):
State: RUNNABLE
Blocked count: 1
Waited count: 26
Stack:
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native
Method)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:312)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:376)
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
Thread 110 (RS-EventLoopGroup-1-6):
State: RUNNABLE
Blocked count: 0
Waited count: 29
Stack:
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native
Method)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:316)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:373)
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
Thread 109 (RS-EventLoopGroup-1-5):
State: RUNNABLE
Blocked count: 0
Waited count: 54
Stack:
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait0(Native
Method)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:182)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWait(EpollEventLoop.java:312)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:376)
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
Thread 108 (RS-EventLoopGroup-1-4):
State: RUNNABLE
Blocked count: 98
Waited count: 38
Stack:
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native
Method)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:316)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:373)
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
Thread 107 (RS-EventLoopGroup-1-3):
State: RUNNABLE
Blocked count: 130
Waited count: 39
Stack:
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native
Method)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:316)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:373)
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
Thread 106 (RS-EventLoopGroup-1-2):
State: RUNNABLE
Blocked count: 51
Waited count: 7
Stack:
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native
Method)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:316)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:373)
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
Thread 105 (master:store-WAL-Roller):
State: TIMED_WAITING
Blocked count: 3
Waited count: 104
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.wal.AbstractWALRoller.run(AbstractWALRoller.java:179)
Thread 103 (LeaseRenewer:seritrack@masterserver:9000):
State: TIMED_WAITING
Blocked count: 31
Waited count: 1016
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.client.impl.LeaseRenewer.run(LeaseRenewer.java:412)
org.apache.hadoop.hdfs.client.impl.LeaseRenewer.access$600(LeaseRenewer.java:76)
org.apache.hadoop.hdfs.client.impl.LeaseRenewer$1.run(LeaseRenewer.java:308)
java.lang.Thread.run(Thread.java:750)
Thread 100 (org.apache.hadoop.hdfs.PeerCache@2af0ac32):
State: TIMED_WAITING
Blocked count: 0
Waited count: 319
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253)
org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46)
org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124)
java.lang.Thread.run(Thread.java:750)
Thread 99 (IPC Parameter Sending Thread #0):
State: TIMED_WAITING
Blocked count: 1
Waited count: 204
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 97 (master/masterserver:16000:becomeActiveMaster-MemStoreChunkPool
Statistics):
State: TIMED_WAITING
Blocked count: 0
Waited count: 4
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 95 (master/masterserver:16000:becomeActiveMaster-MemStoreChunkPool
Statistics):
State: TIMED_WAITING
Blocked count: 0
Waited count: 4
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 93 (Monitor thread for TaskMonitor):
State: TIMED_WAITING
Blocked count: 0
Waited count: 96
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hbase.monitoring.TaskMonitor$MonitorRunnable.run(TaskMonitor.java:325)
java.lang.Thread.run(Thread.java:750)
Thread 92 (master/masterserver:16000:becomeActiveMaster):
State: TIMED_WAITING
Blocked count: 165
Waited count: 404
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:125)
org.apache.hadoop.hbase.master.HMaster.isRegionOnline(HMaster.java:1358)
org.apache.hadoop.hbase.master.HMaster.waitForMetaOnline(HMaster.java:1328)
org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:1069)
org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2405)
org.apache.hadoop.hbase.master.HMaster.lambda$null$0(HMaster.java:565)
org.apache.hadoop.hbase.master.HMaster$$Lambda$265/1598878738.run(Unknown
Source)
org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:177)
org.apache.hadoop.hbase.master.HMaster.lambda$run$1(HMaster.java:562)
org.apache.hadoop.hbase.master.HMaster$$Lambda$264/1129144214.run(Unknown
Source)
java.lang.Thread.run(Thread.java:750)
Thread 19 (master/masterserver:16000):
State: TIMED_WAITING
Blocked count: 5
Waited count: 322
Stack:
java.lang.Object.wait(Native Method)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:81)
org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:64)
org.apache.hadoop.hbase.master.HMaster.waitForMasterActive(HMaster.java:677)
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:999)
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:942)
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1048)
org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:578)
Thread 91 (Session-HouseKeeper-513b52af-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 2
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 90 (qtp85435056-90):
State: RUNNABLE
Blocked count: 0
Waited count: 24
Stack:
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:185)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:144)
org.apache.hadoop.hbase.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:181)
org.apache.hadoop.hbase.util.Threads.printThreadInfo(Threads.java:186)
org.apache.hadoop.hbase.master.http.MasterDumpServlet.doGet(MasterDumpServlet.java:86)
javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:117)
org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
org.apache.hadoop.hbase.http.SecurityHeadersFilter.doFilter(SecurityHeadersFilter.java:65)
org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
org.apache.hadoop.hbase.http.ClickjackingPreventionFilter.doFilter(ClickjackingPreventionFilter.java:49)
org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
org.apache.hbase.thirdparty.org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626)
org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1521)
Thread 89 (qtp85435056-89):
State: RUNNABLE
Blocked count: 2
Waited count: 26
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:183)
org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:190)
org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:606)
org.apache.hbase.thirdparty.org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:543)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:362)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:186)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
java.lang.Thread.run(Thread.java:750)
Thread 88 (qtp85435056-88):
State: TIMED_WAITING
Blocked count: 2
Waited count: 24
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018)
java.lang.Thread.run(Thread.java:750)
Thread 87 (qtp85435056-87):
State: TIMED_WAITING
Blocked count: 8
Waited count: 24
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:324)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:399)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
java.lang.Thread.run(Thread.java:750)
Thread 86 (qtp85435056-86):
State: TIMED_WAITING
Blocked count: 3
Waited count: 27
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018)
java.lang.Thread.run(Thread.java:750)
Thread 85 (qtp85435056-85):
State: TIMED_WAITING
Blocked count: 3
Waited count: 26
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018)
java.lang.Thread.run(Thread.java:750)
Thread 84 (qtp85435056-84-acceptor-0@acc4c8d-ServerConnector@5ec5ea63{HTTP/1.1,
(http/1.1)}{0.0.0.0:16010}):
State: RUNNABLE
Blocked count: 0
Waited count: 1
Stack:
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:421)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:249)
org.apache.hbase.thirdparty.org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:388)
org.apache.hbase.thirdparty.org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:704)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
java.lang.Thread.run(Thread.java:750)
Thread 83 (qtp85435056-83):
State: TIMED_WAITING
Blocked count: 6
Waited count: 21
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:382)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:974)
org.apache.hbase.thirdparty.org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1018)
java.lang.Thread.run(Thread.java:750)
Thread 81 (RpcServer.metaPriority.FPBQ.Fifo.handler=0,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@52fbca80
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 80 (RpcServer.replication.FPBQ.Fifo.handler=2,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@2a42337c
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 79 (RpcServer.replication.FPBQ.Fifo.handler=1,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@550383c4
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 78 (RpcServer.replication.FPBQ.Fifo.handler=0,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@cdb4b1f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 77 (RpcServer.priority.RWQ.Fifo.read.handler=19,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 76 (RpcServer.priority.RWQ.Fifo.read.handler=18,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 75 (RpcServer.priority.RWQ.Fifo.read.handler=17,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 74 (RpcServer.priority.RWQ.Fifo.read.handler=16,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 73 (RpcServer.priority.RWQ.Fifo.read.handler=15,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 72 (RpcServer.priority.RWQ.Fifo.read.handler=14,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 71 (RpcServer.priority.RWQ.Fifo.read.handler=13,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 70 (RpcServer.priority.RWQ.Fifo.read.handler=12,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 69 (RpcServer.priority.RWQ.Fifo.read.handler=11,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 68 (RpcServer.priority.RWQ.Fifo.read.handler=10,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 67 (RpcServer.priority.RWQ.Fifo.read.handler=9,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 66 (RpcServer.priority.RWQ.Fifo.read.handler=8,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 65 (RpcServer.priority.RWQ.Fifo.read.handler=7,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 64 (RpcServer.priority.RWQ.Fifo.read.handler=6,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 63 (RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 62 (RpcServer.priority.RWQ.Fifo.read.handler=4,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 61 (RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 60 (RpcServer.priority.RWQ.Fifo.read.handler=2,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@783fce
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 59 (RpcServer.priority.RWQ.Fifo.write.handler=1,queue=0,port=16000):
State: WAITING
Blocked count: 35
Waited count: 837
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b9a256e
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 58 (RpcServer.priority.RWQ.Fifo.write.handler=0,queue=0,port=16000):
State: WAITING
Blocked count: 25
Waited count: 819
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3b9a256e
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.hadoop.hbase.ipc.RpcHandler.getCallRunner(RpcHandler.java:68)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 57 (RpcServer.default.FPBQ.Fifo.handler=29,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@724e8c7a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 56 (RpcServer.default.FPBQ.Fifo.handler=28,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@249bb29f
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 55 (RpcServer.default.FPBQ.Fifo.handler=27,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@655c7f8d
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 54 (RpcServer.default.FPBQ.Fifo.handler=26,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@f87971d
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 53 (RpcServer.default.FPBQ.Fifo.handler=25,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@9ace7cb
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 52 (RpcServer.default.FPBQ.Fifo.handler=24,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@705aa32b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 51 (RpcServer.default.FPBQ.Fifo.handler=23,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@767c0aba
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 50 (RpcServer.default.FPBQ.Fifo.handler=22,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@45704417
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 49 (RpcServer.default.FPBQ.Fifo.handler=21,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@2b8c61cd
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 48 (RpcServer.default.FPBQ.Fifo.handler=20,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@eeadc6c
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 47 (RpcServer.default.FPBQ.Fifo.handler=19,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@18552ed1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 46 (RpcServer.default.FPBQ.Fifo.handler=18,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@5fc29130
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 45 (RpcServer.default.FPBQ.Fifo.handler=17,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@1708110c
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 44 (RpcServer.default.FPBQ.Fifo.handler=16,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@4f59a63d
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 43 (RpcServer.default.FPBQ.Fifo.handler=15,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@7e4b2aa1
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 42 (RpcServer.default.FPBQ.Fifo.handler=14,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@354b7f49
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 41 (RpcServer.default.FPBQ.Fifo.handler=13,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@629c889d
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 40 (RpcServer.default.FPBQ.Fifo.handler=12,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@6076a2bd
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 39 (RpcServer.default.FPBQ.Fifo.handler=11,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@474f8230
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 38 (RpcServer.default.FPBQ.Fifo.handler=10,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@29901ca7
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 37 (RpcServer.default.FPBQ.Fifo.handler=9,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@2fceb168
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 36 (RpcServer.default.FPBQ.Fifo.handler=8,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@7af89d66
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 35 (RpcServer.default.FPBQ.Fifo.handler=7,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@411dce0b
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 34 (RpcServer.default.FPBQ.Fifo.handler=6,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@684187ab
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 33 (RpcServer.default.FPBQ.Fifo.handler=5,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@325e2e3a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 32 (RpcServer.default.FPBQ.Fifo.handler=4,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@cf386ba
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 31 (RpcServer.default.FPBQ.Fifo.handler=3,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@23b17cb9
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 30 (RpcServer.default.FPBQ.Fifo.handler=2,queue=2,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@4729dbbb
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 29 (RpcServer.default.FPBQ.Fifo.handler=1,queue=1,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@445129a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 28 (RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=16000):
State: WAITING
Blocked count: 0
Waited count: 1
Waiting on java.util.concurrent.Semaphore$NonfairSync@141392fe
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
java.util.concurrent.Semaphore.acquire(Semaphore.java:312)
org.apache.hadoop.hbase.ipc.FastPathRpcHandler.getCallRunner(FastPathRpcHandler.java:55)
org.apache.hadoop.hbase.ipc.RpcHandler.run(RpcHandler.java:82)
Thread 27 (zk-event-processor-pool-0):
State: WAITING
Blocked count: 10
Waited count: 16
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3cef7071
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 26 (main-EventThread):
State: WAITING
Blocked count: 0
Waited count: 8
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@35929054
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
Thread 25 (main-SendThread(masterserver:2181)):
State: RUNNABLE
Blocked count: 3
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
Thread 23
(org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner):
State: WAITING
Blocked count: 3
Waited count: 4
Waiting on java.lang.ref.ReferenceQueue$Lock@19ef85be
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3712)
java.lang.Thread.run(Thread.java:750)
Thread 22 (RS-EventLoopGroup-1-1):
State: RUNNABLE
Blocked count: 21
Waited count: 0
Stack:
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native
Method)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209)
org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:316)
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:373)
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
java.lang.Thread.run(Thread.java:750)
Thread 21 (HBase-Metrics2-1):
State: TIMED_WAITING
Blocked count: 0
Waited count: 96
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:750)
Thread 20 (Timer for 'HBase' metrics system):
State: TIMED_WAITING
Blocked count: 0
Waited count: 96
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 16 (RMI TCP Accept-0):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
java.net.ServerSocket.implAccept(ServerSocket.java:560)
java.net.ServerSocket.accept(ServerSocket.java:528)
sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
java.lang.Thread.run(Thread.java:750)
Thread 15 (RMI TCP Accept-10101):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
java.net.ServerSocket.implAccept(ServerSocket.java:560)
java.net.ServerSocket.accept(ServerSocket.java:528)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
java.lang.Thread.run(Thread.java:750)
Thread 10 (Thread-3):
State: RUNNABLE
Blocked count: 2
Waited count: 0
Stack:
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:453)
java.lang.Thread.run(Thread.java:750)
Thread 8 (req-rsp-timeout-task):
State: TIMED_WAITING
Blocked count: 1
Waited count: 960
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 7 (idle-timeout-task):
State: TIMED_WAITING
Blocked count: 0
Waited count: 96
Stack:
java.lang.Object.wait(Native Method)
java.util.TimerThread.mainLoop(Timer.java:552)
java.util.TimerThread.run(Timer.java:505)
Thread 5 (Signal Dispatcher):
State: RUNNABLE
Blocked count: 0
Waited count: 0
Stack:
Thread 3 (Finalizer):
State: WAITING
Blocked count: 92
Waited count: 24
Waiting on java.lang.ref.ReferenceQueue$Lock@27c20538
Stack:
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:188)
Thread 2 (Reference Handler):
State: WAITING
Blocked count: 52
Waited count: 23
Waiting on java.lang.ref.Reference$Lock@72d818d1
Stack:
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.ref.Reference.tryHandlePending(Reference.java:191)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Thread 1 (main):
State: WAITING
Blocked count: 20
Waited count: 19
Waiting on org.apache.hadoop.hbase.master.HMaster@4772c3a0
Stack:
java.lang.Object.wait(Native Method)
java.lang.Thread.join(Thread.java:1257)
java.lang.Thread.join(Thread.java:1331)
org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:254)
org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:147)
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:140)
org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3311)
Am Sa., 20. Apr. 2024 um 15:11 Uhr schrieb 张铎(Duo Zhang) <
palomino...@gmail.com>:
> Just post it somewhere so we can check it.
>
> Udo Offermann <udo.offerm...@zfabrik.de> 于2024年4月20日周六 20:25写道：
> >
> > I do have the dump File from the web ui. I can sende it all or you Tell
> me
> > threads you are interessted in. Fortunately they all have meaningfull
> named.
> >
> > 张铎(Duo Zhang) <palomino...@gmail.com> schrieb am Sa., 20. Apr. 2024,
> 14:13:
> >
> > > What is the jstack result for HMaster while hanging? Wait on the
> > > namespace table online or meta table online?
> > >
> > > Udo Offermann <udo.offerm...@zfabrik.de> 于2024年4月20日周六 19:43写道：
> > > >
> > > > Hello everyone,
> > > >
> > > > We are upgrading our Hadoop/HBase cluster from Hadoop 2.8.5 & HBase
> 2.2.5
> > > > to Hadoop 3.3.6 & HBase 2.5.7
> > > >
> > > > The Hadoop upgrade worked well, but unfortunately we have problems
> with
> > > the
> > > > Hbase upgrade, because the master hangs on startup inside the
> „Starting
> > > > assignment manger“ task.
> > > >
> > > > After 15 minutes the following message appears in the log file:
> > > >
> > > > Master failed to complete initialization after 900000ms. Please
> > > > consider submitting a bug report including a thread dump of this
> > > > process.
> > > >
> > > >
> > > > We face the same problem as Adam a couple of weeks ago: "Rolling
> upgrade
> > > > from HBase 2.2.2 to 2.5.8 [typo corrected]: There are 2336 corrupted
> > > > procedures“ and we fixed it in the same way by deleting the
> > > > MasterProcWALs-folder
> > > > in HDFS.
> > > >
> > > > I can provide HMaster dump and a dump of one data nodes!
> > > >
> > > > How can we proceed with the upgrade?
> > > >
> > > > Thanks and best regards
> > > > Udo
> > >
>
--
Udo Offermann
udo.offerm...@zfabrik.de
ZFabrik <http://www.zfabrik.de/>
Blog <http://www.z2-environment.net/blog>
Z2-Environment <http://www.z2-environment.eu/>
Z2 Wiki <http://redmine.z2-environment.net/>
T: +49 6227 3984255
F: +49 6227 3984254
M: +49 1781891820
*ZFabrik Software GmbH & Co. KG*
Lammstrasse 2, 69190 Walldorf
Handelsregister: Amtsgericht Mannheim HRA 702598
Persönlich haftende Gesellschafterin: ZFabrik Verwaltungs GmbH, Sitz
Walldorf
Geschäftsführer: Dr. H. Blohm u. Udo Offermann
Handelsregister: Amtsgericht Mannheim HRB 723699
