1. A hibernating node is participating in gossip but intentionally hasn't yet joined the ring. The two cases where a node would set a hibernating status are when the node was started with "-Dcassandra.join_ring=False" and has tokens or when the node was started to replace another node (using "-Dcassandra.replace_address" or "-Dcassandra.replace_address_first_boot").
2. A rolling restart is probably your best bet. You may have more luck with an assassinate in the case that you connect to a node that is not continuously removing/adding the state. I suspect that this node will have an alive status for this endpoint state. As usual, you should wield assassinate with lots of caution. This issue sounds most similar to CASSANDRA-10371. If you provide debugging information similar to that requested on the above ticket as well as what operation you were performing on the node (was it a failed attempt at replacing? etc) on a JIRA ticket, someone might have a chance to look into this further. On Wed, Oct 12, 2016 at 9:48 AM, Kasper Petersen <kas...@sybogames.com> wrote: > Hi, > > I've recently upgraded our Cassandra cluster from 2.1 to 3.9. By > default(?) 3.9 creates a debug.log file containing a ton of lines (a new > one every second) with: > > DEBUG [GossipTasks:1] 2016-10-12 14:43:38,761 Gossiper.java:337 - >> Convicting /172.31.137.65 with status hibernate - alive false > > > That node has not been around for a very long time now. > > It does not show up in nodetool status and nodetool gossipinfo returns > the following output about that node: > > /172.31.137.65 >> generation:1433571405 >> heartbeat:232 >> STATUS:3:hibernate,true >> LOAD:225:96445.0 >> SCHEMA:53:e2d1a288-581c-3f35-b492-1b9d5a803631 >> DC:9:us-east >> RACK:11:1b >> RELEASE_VERSION:7:2.1.5 >> RPC_ADDRESS:6:172.31.137.65 >> SEVERITY:231:0.2512562870979309 >> NET_VERSION:4:8 >> HOST_ID:5:7988d3c9-dec8-4b71-b5a9-0b962aad0680 >> TOKENS:2:<hidden> > > > nodetool removenode 7988d3c9-dec8-4b71-b5a9-0b962aad0680 resulted in: > > error: Host ID not found. >> > > Now my questions are: > > 1. What does it mean for a node to be "hibernating"? How does it end > up in that state? > 2. How do I get rid of it? Its not coming back. > > > > -- > Best regards, > Kasper Middelboe Petersen > > *Lead Backend Developer* > > *SYBO Games ApS* > Jorcks Passage 1A, 4th. > 1162 Copenhagen K >
