I'm trying to debug a node that has a backup in MiscStage. Starting a
bit under 24 hours ago the number of Pending tasks jumped to a bit under
400 and hovered around there. It looks like repair requests from other
nodes (tpstats on this node shows AntiEntropySessions: 0, 0, 0, which I
think indicates it did not originate the repair). After each MiscStage
task completes a series of Streams are kicked off.
I am confused why MiscStage is backing up:
(A) This node has only been down a few hours over the past week so it
should not be wildly out of sync
(B) no other node in this cluster has had a comparable backup of
pending Misc stages.
Repairs are run on all nodes once a week. Physical resources on this
node are not particularity saturated compared to the rest of the
cluster; reads are slower but I can't tell cause from effect in that case.
Graph of MiscStage pending tasks: http://imgur.com/sHqHTvt
This is with a 1.2.11-ish dual-DC vnode cluster.
"MiscStage:1" daemon prio=10 tid=0x00007f84e8598800 nid=0x43b2 waiting on
condition [0x00007f83c3734000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000069d23c700> (a
java.util.concurrent.FutureTask$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
at java.util.concurrent.FutureTask.get(FutureTask.java:83)
at
org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:375)
at
org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:368)
at
org.apache.cassandra.streaming.StreamOut.flushSSTables(StreamOut.java:108)
at
org.apache.cassandra.streaming.StreamOut.transferRanges(StreamOut.java:136)
at
org.apache.cassandra.streaming.StreamOut.transferRanges(StreamOut.java:116)
at
org.apache.cassandra.streaming.StreamRequestVerbHandler.doVerb(StreamRequestVerbHandler.java:44)
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
