cshannon commented on issue #4209:
URL: https://github.com/apache/accumulo/issues/4209#issuecomment-1924920177
I looked into this a little bit with @keith-turner today as I saw the same
thing. A thread dump shows stuck threads because the Manager is trying to scan
the Tservers but they were shut down.
```
"manager" #42 prio=5 os_prio=0 cpu=168.68ms elapsed=176.53s
tid=0x00007f938860de60 nid=0x3305b in Object.wait() [0x00007f93cef47000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait([email protected]/Native Method)
- waiting on <no object reference available>
at java.lang.Thread.join([email protected]/Thread.java:1313)
- locked <0x00000000f14f5d60> (a
org.apache.accumulo.core.util.threads.Threads$AccumuloDaemonThread)
at java.lang.Thread.join([email protected]/Thread.java:1381)
at org.apache.accumulo.core.fate.Fate.shutdown(Fate.java:472)
at org.apache.accumulo.manager.Manager.lambda$run$7(Manager.java:1148)
at
org.apache.accumulo.manager.Manager$$Lambda$743/0x00007f93ec480248.accept(Unknown
Source)
at java.lang.Iterable.forEach([email protected]/Iterable.java:75)
at org.apache.accumulo.manager.Manager.run(Manager.java:1148)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run([email protected]/Thread.java:840)
Locked ownable synchronizers:
- None
```
```
"Fate work finder" #67 daemon prio=5 os_prio=0 cpu=45.49ms elapsed=176.24s
tid=0x00007f93540dfc20 nid=0x3309e waiting on condition [0x00007f93cd82f000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00000000f14e2a48> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at
java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared([email protected]/AbstractQueuedSynchronizer.java:1027)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock([email protected]/ReentrantReadWriteLock.java:738)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl._findTablet(ClientTabletCacheImpl.java:860)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablet(ClientTabletCacheImpl.java:552)
at
org.apache.accumulo.core.clientImpl.ClientTabletCache.findTablet(ClientTabletCache.java:120)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTablet(ClientTabletCacheImpl.java:712)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTabletLocationAndCheckLock(ClientTabletCacheImpl.java:891)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl._findTablet(ClientTabletCacheImpl.java:876)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablet(ClientTabletCacheImpl.java:552)
at
org.apache.accumulo.core.clientImpl.ThriftScanner.getNextScanAddress(ThriftScanner.java:559)
at
org.apache.accumulo.core.clientImpl.ThriftScanner.scan(ThriftScanner.java:662)
at
org.apache.accumulo.core.clientImpl.ScannerIterator.readBatch(ScannerIterator.java:159)
- locked <0x00000000f43e0938> (a
org.apache.accumulo.core.clientImpl.ThriftScanner$ScanState)
at
org.apache.accumulo.core.clientImpl.ScannerIterator.getNextBatch(ScannerIterator.java:177)
at
org.apache.accumulo.core.clientImpl.ScannerIterator.hasNext(ScannerIterator.java:111)
at
java.util.Iterator.forEachRemaining([email protected]/Iterator.java:132)
at
java.util.Spliterators$IteratorSpliterator.forEachRemaining([email protected]/Spliterators.java:1845)
at
java.util.stream.AbstractPipeline.copyInto([email protected]/AbstractPipeline.java:509)
at
java.util.stream.AbstractPipeline.wrapAndCopyInto([email protected]/AbstractPipeline.java:499)
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential([email protected]/ForEachOps.java:150)
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential([email protected]/ForEachOps.java:173)
at
java.util.stream.AbstractPipeline.evaluate([email protected]/AbstractPipeline.java:234)
at
java.util.stream.ReferencePipeline.forEach([email protected]/ReferencePipeline.java:596)
at
org.apache.accumulo.core.fate.AbstractFateStore.runnable(AbstractFateStore.java:163)
at
org.apache.accumulo.core.logging.FateLogger$1.runnable(FateLogger.java:124)
at org.apache.accumulo.core.fate.Fate$WorkFinder.run(Fate.java:93)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run([email protected]/Thread.java:840)
Locked ownable synchronizers:
- <0x00000000f1534078> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
```
```
"batch scanner 27--Worker-1" #194 daemon prio=5 os_prio=0 cpu=178.97ms
elapsed=158.34s tid=0x00007f931c00ed70 nid=0x33334 sleeping [0x00007f9297dfb000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep([email protected]/Native Method)
at java.lang.Thread.sleep([email protected]/Thread.java:344)
at
java.util.concurrent.TimeUnit.sleep([email protected]/TimeUnit.java:446)
at
com.google.common.util.concurrent.Uninterruptibles.sleepUninterruptibly(Uninterruptibles.java:406)
at
org.apache.accumulo.core.clientImpl.RootClientTabletCache.findTablet(RootClientTabletCache.java:149)
at
org.apache.accumulo.core.clientImpl.ClientTabletCache.findTablet(ClientTabletCache.java:120)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTablet(ClientTabletCacheImpl.java:712)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.lookupTabletLocationAndCheckLock(ClientTabletCacheImpl.java:891)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl._findTablet(ClientTabletCacheImpl.java:881)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablets(ClientTabletCacheImpl.java:340)
at
org.apache.accumulo.core.clientImpl.ClientTabletCacheImpl.findTablets(ClientTabletCacheImpl.java:450)
at
org.apache.accumulo.core.clientImpl.ClientTabletCache.binRanges(ClientTabletCache.java:174)
at
org.apache.accumulo.core.clientImpl.SyncingClientTabletCache.binRanges(SyncingClientTabletCache.java:97)
at
org.apache.accumulo.core.clientImpl.TimeoutClientTabletCache.binRanges(TimeoutClientTabletCache.java:110)
at
org.apache.accumulo.core.clientImpl.TabletServerBatchReaderIterator.binRanges(TabletServerBatchReaderIterator.java:275)
at
org.apache.accumulo.core.clientImpl.TabletServerBatchReaderIterator.processFailures(TabletServerBatchReaderIterator.java:375)
at
org.apache.accumulo.core.clientImpl.TabletServerBatchReaderIterator$QueryTask.run(TabletServerBatchReaderIterator.java:485)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at
java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
at
org.apache.accumulo.core.trace.TraceWrappedRunnable.run(TraceWrappedRunnable.java:52)
at java.lang.Thread.run([email protected]/Thread.java:840)
Locked ownable synchronizers:
- <0x00000000f14e2a48> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
- <0x00000000f3dad138> (a
java.util.concurrent.ThreadPoolExecutor$Worker)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
