Well, I tried : rolling restart did not work its magic.
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 192.168.1.9 26.32 GiB 256 42.8%
76223d4c-9d9f-417f-be27-cebb791cddcc rack1
UN 192.168.1.12 31.35 GiB 256 38.9%
719601e2-54a6-440e-a379-c9cf2dc20564 rack1
UN 192.168.1.17 25.2 GiB 256 41.4%
fa238b21-1db1-47dc-bfb7-beedc6c9967a rack1
DN 192.168.1.18 ? 256 39.8% null
rack1
UN 192.168.1.22 27.7 GiB 256 37.2%
09d24557-4e98-44c3-8c9d-53c4c31066e1 rack1
Alex
Le 04.04.2019 18:26, Alex a écrit :
> Hi,
>
> @ Alain and Kenneth :
>
> I use C* for a time series database (KairosDB) ; replication and consistency
> are set by KairosDB and I would rather not mingle with it.
>
> @ Nick and Alain :
>
> I have tried to stop / start every node but not with this process. I will
> try.
>
> @ Jeff : I removed (replaced) this node 13 days ago.
>
> @ Alain : In system.peers I see both the dead node and its replacement with
> the same ID :
>
> peer | host_id
> --------------+--------------------------------------
> 192.168.1.18 | 09d24557-4e98-44c3-8c9d-53c4c31066e1
> 192.168.1.22 | 09d24557-4e98-44c3-8c9d-53c4c31066e1
>
> Is it expected ?
>
> If I cannot fix this, I think I will add new nodes and remove, one by one,
> the nodes that show the dead node in nodetool status.
>
> Thanks all for your help.
>
> Alex
>
> Le 04.04.2019 17:45, Alain RODRIGUEZ a écrit :
>
> Hi Alex,
>
> About previous advices:
>
> You might have inconsistent data in your system tables. Try setting the
> consistency level to ALL, then do read query of system tables to force
> repair.
>
> System tables use the 'LocalStrategy', thus I don't think any repair would
> happen for the system.* tables. Regardless the consistency you use. It should
> not harm, but I really think it won't help.
> This will sound a little silly but, have you tried rolling the cluster?
>
> The other way around, the rolling restart does not sound that silly to me. I
> would try it before touching any other 'deeper' systems. It has indeed
> sometimes proven to do some magic for me as well. It's hard to guess on this
> kind of ghost node issues without being working on the machine (and sometimes
> even when accessing the machine I had some trouble =)). Also a rolling
> restart is an operation that should be easy to perform and with low risk (if
> everything is well configured).
>
> Other idea to explore:
>
> You can actually select the 'system.peers' table to see if all (other) nodes
> are referenced for each node. There should not be any dead nodes in there. By
> the way you will see that different nodes have slightly different data in
> system.peers, and are not in sync, thus no way to 'repair' that really.
> 'Select' is safe. If you delete non-existing 'peers' if any, If the node is
> dead anyway, this shouldn't hurt, but make sure you are doing the right thing
> you can easily break your cluster from there. I did not see an issue (a bug)
> of those for a while though. Normally you should not have to go that deep
> touching system tables.
>
> Then also nodes removed should be immediately removed from peers but persist
> for some time (7 days maybe?) in the gossip information (normally as 'LEFT').
> This should not create the issue in 'nodetool describecluster' though.
>
> C*heers,
>
> -----------------------
> Alain Rodriguez - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> Le jeu. 4 avr. 2019 à 16:09, Nick Hatfield <nick.hatfi...@metricly.com> a
> écrit :
>
> This will sound a little silly but, have you tried rolling the cluster?
>
> $> nodetool flush; nodetool drain; service cassandra stop
> $> ps aux | grep 'cassandra'
>
> # make sure the process actually dies. If not you may need to kill -9 <pid>.
> Check first to see if nodetool can connect first, nodetool gossipinfo. If the
> connection is live and listening on the port, then just try re-running
> service cassandra stop again. Kill -9 as a last resort
>
> $> service cassandra start
> $> nodetool netstats | grep 'NORMAL' # wait for this to return before moving
> on to the next node.
>
> Restart them all using this method, then run nodetool status again and see if
> it is listed.
>
> Once other thing, I recall you said something about having to terminate a
> node and then replace it. Make sure that whichever node you did the -Dreplace
> flag on, does not still have it set when you start cassandra on it again!
>
> FROM: Alex [mailto:m...@aca-o.com]
> SENT: Thursday, April 04, 2019 4:58 AM
> TO: user@cassandra.apache.org
> SUBJECT: Re: Assassinate fails
>
> Hi Anthony,
>
> Thanks for your help.
>
> I tried to run multiple times in quick succession but it fails with :
>
> -- StackTrace --
> java.lang.RuntimeException: Endpoint still alive: /192.168.1.18 [1]
> generation changed while trying to assassinate it
> at org.apache.cassandra.gms.Gossiper.assassinateEndpoint(Gossiper.java:592)
>
> I can see that the generation number for this node increases by 1 every time
> I call nodetool assassinate ; and the command itself waits for 30 seconds
> before assassinating node. When ran multiple times in quick succession, the
> command fails because the generation number has been changed by the previous
> instance.
>
> In 'nodetool gossipinfo', the node is marked as "LEFT" on every node.
>
> However, in 'nodetool describecluster', this node is marked as "unreacheable"
> on 3 nodes out of 5.
>
> Alex
>
> Le 04.04.2019 00:56, Anthony Grasso a écrit :
>
> Hi Alex,
>
> We wrote a blog post on this topic late last year:
> http://thelastpickle.com/blog/2018/09/18/assassinate.html.
>
> In short, you will need to run the assassinate command on each node
> simultaneously a number of times in quick succession. This will generate a
> number of messages requesting all nodes completely forget there used to be an
> entry within the gossip state for the given IP address.
>
> Regards,
>
> Anthony
>
> On Thu, 4 Apr 2019 at 03:32, Alex <m...@aca-o.com> wrote:
>
> Same result it seems:
> Welcome to JMX terminal. Type "help" for available commands.
> $>open localhost:7199
> #Connection to localhost:7199 is opened
> $>bean org.apache.cassandra.net:type=Gossiper
> #bean is set to org.apache.cassandra.net:type=Gossiper
> $>run unsafeAssassinateEndpoint 192.168.1.18
> #calling operation unsafeAssassinateEndpoint of mbean
> org.apache.cassandra.net:type=Gossiper
> #RuntimeMBeanException: java.lang.NullPointerException
>
> There not much more to see in log files :
> WARN [RMI TCP Connection(10)-127.0.0.1] 2019-04-03 16:25:13,626
> Gossiper.java:575 - Assassinating /192.168.1.18 [1] via gossip
> INFO [RMI TCP Connection(10)-127.0.0.1] 2019-04-03 16:25:13,627
> Gossiper.java:585 - Sleeping for 30000ms to ensure /192.168.1.18 [1] does
> not change
> INFO [RMI TCP Connection(10)-127.0.0.1] 2019-04-03 16:25:43,628
> Gossiper.java:1029 - InetAddress /192.168.1.18 [1] is now DOWN
> INFO [RMI TCP Connection(10)-127.0.0.1] 2019-04-03 16:25:43,631
> StorageService.java:2324 - Removing tokens [..] for /192.168.1.18 [1]
>
> Le 03.04.2019 17:10, Nick Hatfield a écrit :
>> Run assassinate the old way. I works very well...
>>
>> wget -q -O jmxterm.jar
>> http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar
>> [2]
>>
>> java -jar ./jmxterm.jar
>>
>> $>open localhost:7199
>>
>> $>bean org.apache.cassandra.net:type=Gossiper
>>
>> $>run unsafeAssassinateEndpoint 192.168.1.18
>>
>> $>quit
>>
>>
>> Happy deleting
>>
>> -----Original Message-----
>> From: Alex [mailto:m...@aca-o.com]
>> Sent: Wednesday, April 03, 2019 10:42 AM
>> To: user@cassandra.apache.org
>> Subject: Assassinate fails
>>
>> Hello,
>>
>> Short story:
>> - I had to replace a dead node in my cluster
>> - 1 week after, dead node is still seen as DN by 3 out of 5 nodes
>> - dead node has null host_id
>> - assassinate on dead node fails with error
>>
>> How can I get rid of this dead node ?
>>
>>
>> Long story:
>> I had a 3 nodes cluster (Cassandra 3.9) ; one node went dead. I built
>> a new node from scratch and "replaced" the dead node using the
>> information from this page
>> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceNode.html
>> [3].
>> It looked like the replacement went ok.
>>
>> I added two more nodes to strengthen the cluster.
>>
>> A few days have passed and the dead node is still visible and marked
>> as "down" on 3 of 5 nodes in nodetool status:
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> UN 192.168.1.9 16 GiB 256 35.0%
>> 76223d4c-9d9f-417f-be27-cebb791cddcc rack1
>> UN 192.168.1.12 16.09 GiB 256 34.0%
>> 719601e2-54a6-440e-a379-c9cf2dc20564 rack1
>> UN 192.168.1.14 14.16 GiB 256 32.6%
>> d8017a03-7e4e-47b7-89b9-cd9ec472d74f rack1
>> UN 192.168.1.17 15.4 GiB 256 34.1%
>> fa238b21-1db1-47dc-bfb7-beedc6c9967a rack1
>> DN 192.168.1.18 24.3 GiB 256 33.7% null
>> rack1
>> UN 192.168.1.22 19.06 GiB 256 30.7%
>> 09d24557-4e98-44c3-8c9d-53c4c31066e1 rack1
>>
>> Its host ID is null, so I cannot use nodetool removenode. Moreover
>> nodetool assassinate 192.168.1.18 fails with :
>>
>> error: null
>> -- StackTrace --
>> java.lang.NullPointerException
>>
>> And in system.log:
>>
>> INFO [RMI TCP Connection(16)-127.0.0.1] 2019-03-27 17:39:38,595
>> Gossiper.java:585 - Sleeping for 30000ms to ensure /192.168.1.18 [1] does
>> not change INFO [CompactionExecutor:547] 2019-03-27 17:39:38,669
>> AutoSavingCache.java:393 - Saved KeyCache (27316 items) in 163 ms INFO
>> [IndexSummaryManager:1] 2019-03-27 17:40:03,620
>> IndexSummaryRedistribution.java:75 - Redistributing index summaries
>> INFO [RMI TCP Connection(16)-127.0.0.1] 2019-03-27 17:40:08,597
>> Gossiper.java:1029 - InetAddress /192.168.1.18 [1] is now DOWN INFO [RMI
>> TCP Connection(16)-127.0.0.1] 2019-03-27 17:40:08,599
>> StorageService.java:2324 - Removing tokens [-1061369577393671924,...]
>> ERROR [GossipStage:1] 2019-03-27 17:40:08,600 CassandraDaemon.java:226
>> - Exception in thread Thread[GossipStage:1,5,main]
>> java.lang.NullPointerException: null
>>
>>
>> In system.peers, the dead node shows and has the same ID as the
>> replacing node :
>>
>> cqlsh> select peer, host_id from system.peers;
>>
>> peer | host_id
>> --------------+--------------------------------------
>> 192.168.1.18 | 09d24557-4e98-44c3-8c9d-53c4c31066e1
>> 192.168.1.22 | 09d24557-4e98-44c3-8c9d-53c4c31066e1
>> 192.168.1.9 | 76223d4c-9d9f-417f-be27-cebb791cddcc
>> 192.168.1.14 | d8017a03-7e4e-47b7-89b9-cd9ec472d74f
>> 192.168.1.12 | 719601e2-54a6-440e-a379-c9cf2dc20564
>>
>> Dead node and replacing node have different tokens in system.peers.
>>
>> I should add that I also tried decommission on a node that still
>> 192.168.1.18 in its peers. - it is still marked as "leaving" 5 days
>> later. Nothing in notetool netstats or nodetool compactionstats.
>>
>>
>> Thank you for taking the time to read this. Hope you can help.
>>
>> Alex
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>>
Links:
------
[1] http://192.168.1.18
[2]
http://downloads.sourceforge.net/cyclops-group/jmxterm-1.0-alpha-4-uber.jar
[3]
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsReplaceNode.html
