[
https://issues.apache.org/jira/browse/CASSANDRA-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502397#comment-14502397
]
Study Hsueh edited comment on CASSANDRA-6542 at 4/20/15 2:03 PM:
-----------------------------------------------------------------
Also observed in 2.1.3 on CentOS 6.6
The nodes status Log.
Host: 192.168.1.13
------------------------------------------------------------------------------------------------------------------------------------
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.1.29 22.95 GB 256 ?
506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1
UN 192.168.1.28 19.68 GB 256 ?
ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1
UN 192.168.1.13 25.72 GB 256 ?
595ea5ef-cecf-44c7-aa7f-424648791751 rack1
DN 192.168.1.27 ? 256 ?
2ca22f3d-f8d8-4bde-8cdc-de649056cf9c rack1
UN 192.168.1.26 20.71 GB 256 ?
3c880801-8499-4b16-bce4-2bfbc79bed43 rack1
$ nodetool removenode force 2ca22f3d-f8d8-4bde-8cdc-de649056cf9c # nodetool
removenode hangs
$ nodetool removenode status
RemovalStatus: Removing token (-9132940871846770123). Waiting for replication
confirmation from [/192.168.1.29,/192.168.1.28,/192.168.1.26].
------------------------------------------------------------------------------------------------------------------------------------
Host: 192.168.1.28
------------------------------------------------------------------------------------------------------------------------------------
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.1.29 22.96 GB 256 ?
506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1
UN 192.168.1.28 19.69 GB 256 ?
ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1
UN 192.168.1.13 30.43 GB 256 ?
595ea5ef-cecf-44c7-aa7f-424648791751 rack1
UN 192.168.1.26 20.72 GB 256 ?
3c880801-8499-4b16-bce4-2bfbc79bed43 rack1
$ nodetool removenode status
RemovalStatus: No token removals in process.
------------------------------------------------------------------------------------------------------------------------------------
Host: 192.168.1.29
------------------------------------------------------------------------------------------------------------------------------------
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.1.29 22.96 GB 256 ?
506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1
UN 192.168.1.28 19.69 GB 256 ?
ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1
UN 192.168.1.13 30.43 GB 256 ?
595ea5ef-cecf-44c7-aa7f-424648791751 rack1
UN 192.168.1.26 20.72 GB 256 ?
3c880801-8499-4b16-bce4-2bfbc79bed43 rack1
$ nodetool removenode status
RemovalStatus: No token removals in process.
------------------------------------------------------------------------------------------------------------------------------------
Host: 192.168.1.26
------------------------------------------------------------------------------------------------------------------------------------
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.1.29 22.96 GB 256 ?
506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1
UN 192.168.1.28 19.69 GB 256 ?
ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1
UN 192.168.1.13 30.43 GB 256 ?
595ea5ef-cecf-44c7-aa7f-424648791751 rack1
UN 192.168.1.26 20.72 GB 256 ?
3c880801-8499-4b16-bce4-2bfbc79bed43 rack1
$ nodetool removenode status
RemovalStatus: No token removals in process.
------------------------------------------------------------------------------------------------------------------------------------
was (Author: study):
Also observed in 2.1.13 on CentOS 6.6
The nodes status Log.
Host: 192.168.1.13
------------------------------------------------------------------------------------------------------------------------------------
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.1.29 22.95 GB 256 ?
506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1
UN 192.168.1.28 19.68 GB 256 ?
ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1
UN 192.168.1.13 25.72 GB 256 ?
595ea5ef-cecf-44c7-aa7f-424648791751 rack1
DN 192.168.1.27 ? 256 ?
2ca22f3d-f8d8-4bde-8cdc-de649056cf9c rack1
UN 192.168.1.26 20.71 GB 256 ?
3c880801-8499-4b16-bce4-2bfbc79bed43 rack1
$ nodetool removenode force 2ca22f3d-f8d8-4bde-8cdc-de649056cf9c # nodetool
removenode hangs
$ nodetool removenode status
RemovalStatus: Removing token (-9132940871846770123). Waiting for replication
confirmation from [/192.168.1.29,/192.168.1.28,/192.168.1.26].
------------------------------------------------------------------------------------------------------------------------------------
Host: 192.168.1.28
------------------------------------------------------------------------------------------------------------------------------------
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.1.29 22.96 GB 256 ?
506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1
UN 192.168.1.28 19.69 GB 256 ?
ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1
UN 192.168.1.13 30.43 GB 256 ?
595ea5ef-cecf-44c7-aa7f-424648791751 rack1
UN 192.168.1.26 20.72 GB 256 ?
3c880801-8499-4b16-bce4-2bfbc79bed43 rack1
$ nodetool removenode status
RemovalStatus: No token removals in process.
------------------------------------------------------------------------------------------------------------------------------------
Host: 192.168.1.29
------------------------------------------------------------------------------------------------------------------------------------
$ nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.1.29 22.96 GB 256 ?
506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1
UN 192.168.1.28 19.69 GB 256 ?
ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1
UN 192.168.1.13 30.43 GB 256 ?
595ea5ef-cecf-44c7-aa7f-424648791751 rack1
UN 192.168.1.26 20.72 GB 256 ?
3c880801-8499-4b16-bce4-2bfbc79bed43 rack1
$ nodetool removenode status
RemovalStatus: No token removals in process.
------------------------------------------------------------------------------------------------------------------------------------
Host: 192.168.1.26
------------------------------------------------------------------------------------------------------------------------------------
nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID
Rack
UN 192.168.1.29 22.96 GB 256 ?
506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1
UN 192.168.1.28 19.69 GB 256 ?
ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1
UN 192.168.1.13 30.43 GB 256 ?
595ea5ef-cecf-44c7-aa7f-424648791751 rack1
UN 192.168.1.26 20.72 GB 256 ?
3c880801-8499-4b16-bce4-2bfbc79bed43 rack1
$ nodetool removenode status
RemovalStatus: No token removals in process.
------------------------------------------------------------------------------------------------------------------------------------
> nodetool removenode hangs
> -------------------------
>
> Key: CASSANDRA-6542
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6542
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: Ubuntu 12, 1.2.11 DSE
> Reporter: Eric Lubow
> Assignee: Yuki Morishita
>
> Running *nodetool removenode $host-id* doesn't actually remove the node from
> the ring. I've let it run anywhere from 5 minutes to 3 days and there are no
> messages in the log about it hanging or failing, the command just sits there
> running. So the regular response has been to run *nodetool removenode
> $host-id*, give it about 10-15 minutes and then run *nodetool removenode
> force*.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
