I have a dead node I need to remove from the cluster so that I can rebalance
among the existing servers (can't replace it for a while).
I used nodetool removetoken and it's been stuck in the "leaving" state for over
a day now. I've tried a rolling restart, which kicks of some streaming for a
while under netstats but now even that lists nothing going on.
I'm stuck on what to do next to get this node to finally leave so I can move
the tokens around.
Only error I see in the system log:
ERROR [Thread-209] 2011-12-11 01:40:34,347 AbstractCassandraDaemon.java (line
133) Fatal exception in thread Thread[Thread-209,5,main]
java.lang.AssertionError
at
org.apache.cassandra.db.compaction.LeveledManifest.promote(LeveledManifest.java:178)
at
org.apache.cassandra.db.compaction.LeveledCompactionStrategy.handleNotification(LeveledCompactionStrategy.java:141)
at
org.apache.cassandra.db.DataTracker.notifySSTablesChanged(DataTracker.java:481)
at org.apache.cassandra.db.DataTracker.replace(DataTracker.java:275)
at org.apache.cassandra.db.DataTracker.addSSTables(DataTracker.java:237)
at
org.apache.cassandra.db.DataTracker.addStreamedSSTable(DataTracker.java:242)
at
org.apache.cassandra.db.ColumnFamilyStore.addSSTable(ColumnFamilyStore.java:920)
at
org.apache.cassandra.streaming.StreamInSession.closeIfFinished(StreamInSession.java:141)
at
org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:103)
at
org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:184)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:81)
