[jira] [Comment Edited] (CASSANDRA-6542) nodetool removenode hangs
[ https://issues.apache.org/jira/browse/CASSANDRA-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502397#comment-14502397 ] Study Hsueh edited comment on CASSANDRA-6542 at 4/20/15 2:03 PM: - Also observed in 2.1.3 on CentOS 6.6 The nodes status Log. Host: 192.168.1.13 $ nodetool status Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens OwnsHost ID Rack UN 192.168.1.29 22.95 GB 256 ? 506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1 UN 192.168.1.28 19.68 GB 256 ? ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1 UN 192.168.1.13 25.72 GB 256 ? 595ea5ef-cecf-44c7-aa7f-424648791751 rack1 DN 192.168.1.27 ? 256 ? 2ca22f3d-f8d8-4bde-8cdc-de649056cf9c rack1 UN 192.168.1.26 20.71 GB 256 ? 3c880801-8499-4b16-bce4-2bfbc79bed43 rack1 $ nodetool removenode force 2ca22f3d-f8d8-4bde-8cdc-de649056cf9c # nodetool removenode hangs $ nodetool removenode status RemovalStatus: Removing token (-9132940871846770123). Waiting for replication confirmation from [/192.168.1.29,/192.168.1.28,/192.168.1.26]. Host: 192.168.1.28 $ nodetool status Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens OwnsHost ID Rack UN 192.168.1.29 22.96 GB 256 ? 506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1 UN 192.168.1.28 19.69 GB 256 ? ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1 UN 192.168.1.13 30.43 GB 256 ? 595ea5ef-cecf-44c7-aa7f-424648791751 rack1 UN 192.168.1.26 20.72 GB 256 ? 3c880801-8499-4b16-bce4-2bfbc79bed43 rack1 $ nodetool removenode status RemovalStatus: No token removals in process. Host: 192.168.1.29 $ nodetool status Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens OwnsHost ID Rack UN 192.168.1.29 22.96 GB 256 ? 506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1 UN 192.168.1.28 19.69 GB 256 ? ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1 UN 192.168.1.13 30.43 GB 256 ? 595ea5ef-cecf-44c7-aa7f-424648791751 rack1 UN 192.168.1.26 20.72 GB 256 ? 3c880801-8499-4b16-bce4-2bfbc79bed43 rack1 $ nodetool removenode status RemovalStatus: No token removals in process. Host: 192.168.1.26 nodetool status Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens OwnsHost ID Rack UN 192.168.1.29 22.96 GB 256 ? 506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1 UN 192.168.1.28 19.69 GB 256 ? ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1 UN 192.168.1.13 30.43 GB 256 ? 595ea5ef-cecf-44c7-aa7f-424648791751 rack1 UN 192.168.1.26 20.72 GB 256 ? 3c880801-8499-4b16-bce4-2bfbc79bed43 rack1 $ nodetool removenode status RemovalStatus: No token removals in process. was (Author: study): Also observed in 2.1.13 on CentOS 6.6 The nodes status Log. Host: 192.168.1.13 $ nodetool status Datacenter: datacenter1 === Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens OwnsHost ID Rack UN 192.168.1.29 22.95 GB 256 ? 506910ae-a07b-4f74-8feb-a3f2b141dea5 rack1 UN 192.168.1.28 19.68 GB 256 ? ed79b6ee-cae0-48f9-a420-338058e1f2c5 rack1 UN
[jira] [Comment Edited] (CASSANDRA-6542) nodetool removenode hangs
[ https://issues.apache.org/jira/browse/CASSANDRA-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13881027#comment-13881027 ] Andrew Losey edited comment on CASSANDRA-6542 at 1/24/14 3:07 PM: -- This behavior is also seen in 1.2.12 on CentOS EL6 was (Author: and...@addthis.com): This behavior is also seen in 1.2.12. nodetool removenode hangs - Key: CASSANDRA-6542 URL: https://issues.apache.org/jira/browse/CASSANDRA-6542 Project: Cassandra Issue Type: Bug Components: Core Environment: Ubuntu 12, 1.2.11 DSE Reporter: Eric Lubow Assignee: Tyler Hobbs Running *nodetool removenode $host-id* doesn't actually remove the node from the ring. I've let it run anywhere from 5 minutes to 3 days and there are no messages in the log about it hanging or failing, the command just sits there running. So the regular response has been to run *nodetool removenode $host-id*, give it about 10-15 minutes and then run *nodetool removenode force*. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-6542) nodetool removenode hangs
[
https://issues.apache.org/jira/browse/CASSANDRA-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863911#comment-13863911
]
Justen Walker edited comment on CASSANDRA-6542 at 1/24/14 6:53 PM:
---
I've had this happen in 1.1.11.
Unfortunately this version does not have a {{removenode force}} command; you
have to use [unsafe JMX methods|https://gist.github.com/justenwalker/8338334]
to force-remove the node.
was (Author: justen_walker):
I've had this happen in 1.1.11.
Unfortunately this version does not have a {{removenode force}} command; you
have to use unsafe JMX methods to force-remove the node.
nodetool removenode hangs
-
Key: CASSANDRA-6542
URL: https://issues.apache.org/jira/browse/CASSANDRA-6542
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 12, 1.2.11 DSE
Reporter: Eric Lubow
Assignee: Tyler Hobbs
Running *nodetool removenode $host-id* doesn't actually remove the node from
the ring. I've let it run anywhere from 5 minutes to 3 days and there are no
messages in the log about it hanging or failing, the command just sits there
running. So the regular response has been to run *nodetool removenode
$host-id*, give it about 10-15 minutes and then run *nodetool removenode
force*.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-6542) nodetool removenode hangs
[
https://issues.apache.org/jira/browse/CASSANDRA-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13863911#comment-13863911
]
Justen Walker edited comment on CASSANDRA-6542 at 1/7/14 5:06 AM:
--
I've had this happen in 1.1.11.
Unfortunately this version does not have a {{removenode force}} command; you
have to use unsafe JMX methods to force-remove the node.
was (Author: justen_walker):
I've had this happen in 1.1.11.
Unfortunately this version does not have a {removenode force} command; you have
to use unsafe JMX methods to force-remove the node.
nodetool removenode hangs
-
Key: CASSANDRA-6542
URL: https://issues.apache.org/jira/browse/CASSANDRA-6542
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 12, 1.2.11 DSE
Reporter: Eric Lubow
Running *nodetool removenode $host-id* doesn't actually remove the node from
the ring. I've let it run anywhere from 5 minutes to 3 days and there are no
messages in the log about it hanging or failing, the command just sits there
running. So the regular response has been to run *nodetool removenode
$host-id*, give it about 10-15 minutes and then run *nodetool removenode
force*.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
