nodetool removetoken hang
-------------------------
Key: CASSANDRA-2974
URL: https://issues.apache.org/jira/browse/CASSANDRA-2974
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8.1
Reporter: Wojciech Meler
one node died - i tried to remove it with removetoken but it hanged:
RemovalStatus: Removing token (9452287970026068429538183539771339207). Waiting
for replication confirmation from [/10.0.3.78].
nodetool netstats doesn't show any streams:
# nodetool -h 10.0.3.65 -p 8080 netstats
Mode: Normal
Not sending any streams.
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 0 332578
Responses n/a 0 646405
# nodetool -h 10.0.3.66 -p 8080 netstats
Mode: Normal
Nothing streaming to /10.0.3.78
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 178 739797
Responses n/a 0 1294349
# nodetool -h 10.0.3.71 -p 8080 netstats
Mode: Normal
Not sending any streams.
Not receiving any streams.
Pool Name Active Pending Completed
Commands n/a 84 2031299
Responses n/a 0 357749
BTW. "Nothing streaming to /10.0.3.78" is quite funny :)
also nodetool ring show strange things - almost whole nodes report:
10.0.3.65 datacenter1 rack1 Up Normal 114.2 GB 5.56% 0
10.0.3.77 datacenter1 rack1 Down Leaving 158.09 GB 5.56%
9452287970026068429538183539771339207
10.0.3.71 datacenter1 rack1 Up Normal 196.76 GB 5.56%
18904575940052136859076367079542678414
10.0.3.66 datacenter1 rack1 Up Normal 178.95 GB 5.56%
28356863910078205288614550619314017621
10.0.3.78 datacenter1 rack1 Up Normal 227.05 GB 5.56%
37809151880104273718152734159085356828
10.0.3.72 datacenter1 rack1 Up Normal 110.83 GB 5.56%
47261439850130342147690917698856696035
10.0.3.67 datacenter1 rack1 Up Normal 117.45 GB 5.56%
56713727820156410577229101238628035242
10.0.3.79 datacenter1 rack1 Up Normal 138.62 GB 5.56%
66166015790182479006767284778399374449
10.0.3.73 datacenter1 rack1 Up Normal 110.49 GB 5.56%
75618303760208547436305468318170713656
10.0.3.68 datacenter1 rack1 Up Normal 114.82 GB 5.56%
85070591730234615865843651857942052863
10.0.3.80 datacenter1 rack1 Up Normal 145.51 GB 5.56%
94522879700260684295381835397713392070
10.0.3.74 datacenter1 rack1 Up Normal 113.63 GB 5.56%
103975167670286752724920018937484731277
10.0.3.69 datacenter1 rack1 Up Normal 111.24 GB 5.56%
113427455640312821154458202477256070484
10.0.3.81 datacenter1 rack1 Up Normal 142.12 GB 5.56%
122879743610338889583996386017027409691
10.0.3.75 datacenter1 rack1 Up Normal 110.87 GB 5.56%
132332031580364958013534569556798748898
10.0.3.70 datacenter1 rack1 Up Normal 113.21 GB 5.56%
141784319550391026443072753096570088105
10.0.3.82 datacenter1 rack1 Up Normal 163.29 GB 5.56%
151236607520417094872610936636341427312
10.0.3.76 datacenter1 rack1 Up Normal 112.68 GB 5.56%
160688895490443163302149120176112766519
but 2 nodes which should take responsibility for removed token says:
10.0.3.65 datacenter1 rack1 Up Normal 114.2 GB 5.56% 0
10.0.3.77 datacenter1 rack1 Up Leaving 158.09 GB 5.56%
9452287970026068429538183539771339207
10.0.3.71 datacenter1 rack1 Up Normal 196.76 GB 5.56%
18904575940052136859076367079542678414
10.0.3.66 datacenter1 rack1 Up Normal 178.95 GB 5.56%
28356863910078205288614550619314017621
10.0.3.78 datacenter1 rack1 Up Normal 227.05 GB 5.56%
37809151880104273718152734159085356828
10.0.3.72 datacenter1 rack1 Up Normal 110.83 GB 5.56%
47261439850130342147690917698856696035
10.0.3.67 datacenter1 rack1 Up Normal 117.45 GB 5.56%
56713727820156410577229101238628035242
10.0.3.79 datacenter1 rack1 Up Normal 138.62 GB 5.56%
66166015790182479006767284778399374449
10.0.3.73 datacenter1 rack1 Up Normal 110.49 GB 5.56%
75618303760208547436305468318170713656
10.0.3.68 datacenter1 rack1 Up Normal 114.82 GB 5.56%
85070591730234615865843651857942052863
10.0.3.80 datacenter1 rack1 Up Normal 145.51 GB 5.56%
94522879700260684295381835397713392070
10.0.3.74 datacenter1 rack1 Up Normal 113.63 GB 5.56%
103975167670286752724920018937484731277
10.0.3.69 datacenter1 rack1 Up Normal 111.24 GB 5.56%
113427455640312821154458202477256070484
10.0.3.81 datacenter1 rack1 Up Normal 142.12 GB 5.56%
122879743610338889583996386017027409691
10.0.3.75 datacenter1 rack1 Up Normal 110.87 GB 5.56%
132332031580364958013534569556798748898
10.0.3.70 datacenter1 rack1 Up Normal 113.21 GB 5.56%
141784319550391026443072753096570088105
10.0.3.82 datacenter1 rack1 Up Normal 163.29 GB 5.56%
151236607520417094872610936636341427312
10.0.3.76 datacenter1 rack1 Up Normal 112.68 GB 5.56%
160688895490443163302149120176112766519
they started to report it as UP after nodetool removetoken was called
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
