[
https://issues.apache.org/jira/browse/CASSANDRA-13308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15902609#comment-15902609
]
Jeff Jirsa commented on CASSANDRA-13308:
----------------------------------------
{code}
Thread 28548: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may
be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14,
line=175 (Compiled frame)
- java.util.concurrent.FutureTask.awaitDone(boolean, long) @bci=165, line=429
(Compiled frame)
- java.util.concurrent.FutureTask.get() @bci=13, line=191 (Compiled frame)
-
org.apache.cassandra.hints.HintsDispatchExecutor.completeDispatchBlockingly(org.apache.cassandra.hints.HintsStore)
@bci=22, line=112 (Interpreted frame)
- org.apache.cassandra.hints.HintsService.excise(java.util.UUID) @bci=75,
line=323 (Interpreted frame)
- org.apache.cassandra.service.StorageService.excise(java.util.Collection,
java.net.InetAddress) @bci=35, line=2229 (Interpreted frame)
- org.apache.cassandra.service.StorageService.excise(java.util.Collection,
java.net.InetAddress, long) @bci=9, line=2242 (Interpreted frame)
-
org.apache.cassandra.service.StorageService.handleStateLeft(java.net.InetAddress,
java.lang.String[]) @bci=58, line=2146 (Interpreted frame)
- java.util.concurrent.ConcurrentHashMap.get(java.lang.Object) @bci=1,
line=936 (Compiled frame)
-
org.apache.cassandra.gms.Gossiper.getEndpointStateForEndpoint(java.net.InetAddress)
@bci=5, line=817 (Compiled frame)
- org.apache.cassandra.service.StorageService.onChange(java.net.InetAddress,
org.apache.cassandra.gms.ApplicationState,
org.apache.cassandra.gms.VersionedValue) @bci=418, line=1685 (Compiled frame)
-
org.apache.cassandra.gms.Gossiper.doOnChangeNotifications(java.net.InetAddress,
org.apache.cassandra.gms.ApplicationState,
org.apache.cassandra.gms.VersionedValue) @bci=38, line=1200 (Compiled frame)
- org.apache.cassandra.gms.Gossiper.applyNewStates(java.net.InetAddress,
org.apache.cassandra.gms.EndpointState, org.apache.cassandra.gms.EndpointState)
@bci=164, line=1183 (Compiled frame)
- org.apache.cassandra.gms.Gossiper.applyStateLocally(java.util.Map) @bci=366,
line=1146 (Compiled frame)
-
org.apache.cassandra.gms.GossipDigestAckVerbHandler.doVerb(org.apache.cassandra.net.MessageIn,
int) @bci=143, line=58 (Compiled frame)
- org.apache.cassandra.net.MessageDeliveryTask.run() @bci=82, line=67
(Compiled frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=511
(Compiled frame)
- java.util.concurrent.FutureTask.run() @bci=42, line=266 (Compiled frame)
-
java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker)
@bci=95, line=1142 (Compiled frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=617
(Interpreted frame)
- java.lang.Thread.run() @bci=11, line=745 (Compiled frame)
{code}
{{excise}}
[here|https://github.com/apache/cassandra/blob/cassandra-3.0/src/java/org/apache/cassandra/hints/HintsService.java#L287-L327]
attempts to complete the running dispatch if it exists (for example, if the
host was just down, but came up, and hint delivery is in progress), even though
that endpoint is going away (was just decom'd).
[~iamaleksey] - I'm not very familiar with this code - are we really gaining
much from this? Do we need to block trying to deliver hints we know aren't
going to be deliverable, risking getting into this situation where we're
blocking waiting for {{isHostAlive()}} to finally fail (which won't happen if
Gossip is blocked and thus FD won't kick in), when the very next thing we do is
{{exciseStore()}}?
> Hint files not being deleted on nodetool decommission
> -----------------------------------------------------
>
> Key: CASSANDRA-13308
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13308
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
> Environment: Using Cassandra version 3.0.9
> Reporter: Arijit
> Attachments: 28207.stack, logs, logs_decommissioned_node
>
>
> How to reproduce the issue I'm seeing:
> Shut down Cassandra on one node of the cluster and wait until we accumulate a
> ton of hints. Start Cassandra on the node and immediately run "nodetool
> decommission" on it.
> The node streams its replicas and marks itself as DECOMMISSIONED, but other
> nodes do not seem to see this message. "nodetool status" shows the
> decommissioned node in state "UL" on all other nodes (it is also present in
> system.peers), and Cassandra logs show that gossip tasks on nodes are not
> proceeding (number of pending tasks keeps increasing). Jstack suggests that a
> gossip task is blocked on hints dispatch (I can provide traces if this is not
> obvious). Because the cluster is large and there are a lot of hints, this is
> taking a while.
> On inspecting "/var/lib/cassandra/hints" on the nodes, I see a bunch of hint
> files for the decommissioned node. Documentation seems to suggest that these
> hints should be deleted during "nodetool decommission", but it does not seem
> to be the case here. This is the bug being reported.
> To recover from this scenario, if I manually delete hint files on the nodes,
> the hints dispatcher threads throw a bunch of exceptions and the
> decommissioned node is now in state "DL" (perhaps it missed some gossip
> messages?). The node is still in my "system.peers" table
> Restarting Cassandra on all nodes after this step does not fix the issue (the
> node remains in the peers table). In fact, after this point the
> decommissioned node is in state "DN"
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
