[jira] [Commented] (CASSANDRA-10991) Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
[
https://issues.apache.org/jira/browse/CASSANDRA-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15151132#comment-15151132
]
Carl Yeksigian commented on CASSANDRA-10991:
+1; not sure if this should still go in 2.1 since it's just cleaning up the
error message.
> Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
> --
>
> Key: CASSANDRA-10991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10991
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
>Reporter: mlowicki
>Assignee: Marcus Eriksson
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>
> I've C* cluster spread across 3 DCs. Running {{cleanup}} on all nodes in one
> DC always fails:
> {code}
> root@db1:~# nt cleanup system
> root@db1:~# nt cleanup sync
> root@db1:~# nt cleanup OpsCenter
> Aborted cleaning up atleast one column family in keyspace OpsCenter, check
> server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
> root@db1:~#
> {code}
> Checked two other DCs and running cleanup there works fine (it didn't fail
> immediately).
> Output from {{nodetool status}} from one node in problematic DC:
> {code}
> root@db1:~# nt status
> Datacenter: Amsterdam
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.3.162 518.54 GB 256 ?
> 50e606f5-e893-4a3b-86d3-1e5986dceea9 RAC1
> UN 10.210.3.230 532.63 GB 256 ?
> 7b8fc988-8a6a-4d94-ae84-ab9da9ab01e8 RAC1
> UN 10.210.3.161 538.82 GB 256 ?
> d44b0f6d-7933-4a7c-ba7b-f8648e038f85 RAC1
> UN 10.210.3.160 497.6 GB 256 ?
> e7332179-a47e-471d-bcd4-08c638ab9ea4 RAC1
> UN 10.210.3.224 334.25 GB 256 ?
> 92b0bd8c-0a5a-446a-83ea-2feea4988fe3 RAC1
> UN 10.210.3.118 518.34 GB 256 ?
> ebddeaf3-1433-4372-a4ca-9c7ba3d4a26b RAC1
> UN 10.210.3.221 516.57 GB 256 ?
> 44d67a49-5310-4ab5-b448-a44be350abf5 RAC1
> UN 10.210.3.117 493.83 GB 256 ?
> aae92956-82d6-421e-8f3f-22393ac7e5f7 RAC1
> Datacenter: Analytics
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.59.124 392.83 GB 320 ?
> f770a8cc-b7bf-44ac-8cc0-214d9228dfcd RAC1
> UN 10.210.59.151 411.9 GB 320 ?
> 3cc87422-0e43-4cd1-91bf-484f121be072 RAC1
> UN 10.210.58.132 309.8 GB 256 ?
> 84d94d13-28d3-4b49-a3d9-557ab47e79b9 RAC1
> UN 10.210.58.133 281.82 GB 256 ?
> 02bd2d02-41c5-4193-81b0-dee434adb0da RAC1
> UN 10.210.59.86 285.84 GB 256 ?
> bc6422ea-22e9-431a-ac16-c4c040f0c4e5 RAC1
> UN 10.210.59.84 331.06 GB 256 ?
> a798e6b0-3a84-4ec2-82bb-8474086cb315 RAC1
> UN 10.210.59.85 366.26 GB 256 ?
> 52699077-56cf-4c1e-b308-bf79a1644b7e RAC1
> Datacenter: Ashburn
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.195.15.176 534.51 GB 256 ?
> c6ac22df-c43a-4b25-b3b5-5e12ce9c69da RAC1
> UN 10.195.15.177 313.73 GB 256 ?
> eafa2a72-84a2-4cdc-a634-3c660acc6af8 RAC1
> UN 10.195.15.163 470.92 GB 256 ?
> bcd2a534-94c4-4406-8d16-c1fc26b41844 RAC1
> UN 10.195.15.162 539.82 GB 256 ?
> bb649cef-21de-4077-a35f-994319011a06 RAC1
> UN 10.195.15.182 499.64 GB 256 ?
> 6ce2d14d-9fb8-4494-8e97-3add05bd35de RAC1
> UN 10.195.15.167 508.48 GB 256 ?
> 6f359675-852a-4842-9ff2-bdc69e6b04a2 RAC1
> UN 10.195.15.166 490.28 GB 256 ?
> 1ec5d0c5-e8bd-4973-96d9-523de91d08c5 RAC1
> UN 10.195.15.183 447.78 GB 256 ?
> 824165b0-1f1b-40e8-9695-e2f596cb8611 RAC1
> Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless
> {code}
> Logs from one of the nodes where {{cleanup}} fails:
> {code}
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,942
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,970
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connect
[jira] [Commented] (CASSANDRA-10991) Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
[
https://issues.apache.org/jira/browse/CASSANDRA-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106873#comment-15106873
]
Marcus Eriksson commented on CASSANDRA-10991:
-
[~mlowicki] ok, then the patch fixes your issue - but it is not really a
problem currently either, just a bit ugly error message.
> Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
> --
>
> Key: CASSANDRA-10991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10991
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
>Reporter: mlowicki
>Assignee: Marcus Eriksson
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>
> I've C* cluster spread across 3 DCs. Running {{cleanup}} on all nodes in one
> DC always fails:
> {code}
> root@db1:~# nt cleanup system
> root@db1:~# nt cleanup sync
> root@db1:~# nt cleanup OpsCenter
> Aborted cleaning up atleast one column family in keyspace OpsCenter, check
> server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
> root@db1:~#
> {code}
> Checked two other DCs and running cleanup there works fine (it didn't fail
> immediately).
> Output from {{nodetool status}} from one node in problematic DC:
> {code}
> root@db1:~# nt status
> Datacenter: Amsterdam
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.3.162 518.54 GB 256 ?
> 50e606f5-e893-4a3b-86d3-1e5986dceea9 RAC1
> UN 10.210.3.230 532.63 GB 256 ?
> 7b8fc988-8a6a-4d94-ae84-ab9da9ab01e8 RAC1
> UN 10.210.3.161 538.82 GB 256 ?
> d44b0f6d-7933-4a7c-ba7b-f8648e038f85 RAC1
> UN 10.210.3.160 497.6 GB 256 ?
> e7332179-a47e-471d-bcd4-08c638ab9ea4 RAC1
> UN 10.210.3.224 334.25 GB 256 ?
> 92b0bd8c-0a5a-446a-83ea-2feea4988fe3 RAC1
> UN 10.210.3.118 518.34 GB 256 ?
> ebddeaf3-1433-4372-a4ca-9c7ba3d4a26b RAC1
> UN 10.210.3.221 516.57 GB 256 ?
> 44d67a49-5310-4ab5-b448-a44be350abf5 RAC1
> UN 10.210.3.117 493.83 GB 256 ?
> aae92956-82d6-421e-8f3f-22393ac7e5f7 RAC1
> Datacenter: Analytics
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.59.124 392.83 GB 320 ?
> f770a8cc-b7bf-44ac-8cc0-214d9228dfcd RAC1
> UN 10.210.59.151 411.9 GB 320 ?
> 3cc87422-0e43-4cd1-91bf-484f121be072 RAC1
> UN 10.210.58.132 309.8 GB 256 ?
> 84d94d13-28d3-4b49-a3d9-557ab47e79b9 RAC1
> UN 10.210.58.133 281.82 GB 256 ?
> 02bd2d02-41c5-4193-81b0-dee434adb0da RAC1
> UN 10.210.59.86 285.84 GB 256 ?
> bc6422ea-22e9-431a-ac16-c4c040f0c4e5 RAC1
> UN 10.210.59.84 331.06 GB 256 ?
> a798e6b0-3a84-4ec2-82bb-8474086cb315 RAC1
> UN 10.210.59.85 366.26 GB 256 ?
> 52699077-56cf-4c1e-b308-bf79a1644b7e RAC1
> Datacenter: Ashburn
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.195.15.176 534.51 GB 256 ?
> c6ac22df-c43a-4b25-b3b5-5e12ce9c69da RAC1
> UN 10.195.15.177 313.73 GB 256 ?
> eafa2a72-84a2-4cdc-a634-3c660acc6af8 RAC1
> UN 10.195.15.163 470.92 GB 256 ?
> bcd2a534-94c4-4406-8d16-c1fc26b41844 RAC1
> UN 10.195.15.162 539.82 GB 256 ?
> bb649cef-21de-4077-a35f-994319011a06 RAC1
> UN 10.195.15.182 499.64 GB 256 ?
> 6ce2d14d-9fb8-4494-8e97-3add05bd35de RAC1
> UN 10.195.15.167 508.48 GB 256 ?
> 6f359675-852a-4842-9ff2-bdc69e6b04a2 RAC1
> UN 10.195.15.166 490.28 GB 256 ?
> 1ec5d0c5-e8bd-4973-96d9-523de91d08c5 RAC1
> UN 10.195.15.183 447.78 GB 256 ?
> 824165b0-1f1b-40e8-9695-e2f596cb8611 RAC1
> Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless
> {code}
> Logs from one of the nodes where {{cleanup}} fails:
> {code}
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,942
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,970
> CompactionManager.java:388 - Cleanup cannot run before a node has jo
[jira] [Commented] (CASSANDRA-10991) Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
[
https://issues.apache.org/jira/browse/CASSANDRA-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106872#comment-15106872
]
mlowicki commented on CASSANDRA-10991:
--
{code}
cqlsh> desc keyspace "OpsCenter";
CREATE KEYSPACE "OpsCenter" WITH replication = {'class':
'NetworkTopologyStrategy', 'Amsterdam': '1', 'Ashburn': '1'} AND
durable_writes = true;
CREATE TABLE "OpsCenter".events_timeline (
key text,
column1 bigint,
value blob,
PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE
AND CLUSTERING ORDER BY (column1 ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = '{"info": "OpsCenter management data.", "version": [5, 2, 1]}'
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.25
AND speculative_retry = 'NONE';
CREATE TABLE "OpsCenter".settings (
key blob,
column1 blob,
value blob,
PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE
AND CLUSTERING ORDER BY (column1 ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
AND comment = '{"info": "OpsCenter management data.", "version": [5, 2, 1]}'
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
AND compression = {'sstable_compression':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 1.0
AND speculative_retry = 'NONE';
...
{code}
Ah I see that "Analytics" is missing in {{replication}}.
> Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
> --
>
> Key: CASSANDRA-10991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10991
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
>Reporter: mlowicki
>Assignee: Marcus Eriksson
> Fix For: 2.1.x, 2.2.x, 3.0.x, 3.x
>
>
> I've C* cluster spread across 3 DCs. Running {{cleanup}} on all nodes in one
> DC always fails:
> {code}
> root@db1:~# nt cleanup system
> root@db1:~# nt cleanup sync
> root@db1:~# nt cleanup OpsCenter
> Aborted cleaning up atleast one column family in keyspace OpsCenter, check
> server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
> root@db1:~#
> {code}
> Checked two other DCs and running cleanup there works fine (it didn't fail
> immediately).
> Output from {{nodetool status}} from one node in problematic DC:
> {code}
> root@db1:~# nt status
> Datacenter: Amsterdam
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.3.162 518.54 GB 256 ?
> 50e606f5-e893-4a3b-86d3-1e5986dceea9 RAC1
> UN 10.210.3.230 532.63 GB 256 ?
> 7b8fc988-8a6a-4d94-ae84-ab9da9ab01e8 RAC1
> UN 10.210.3.161 538.82 GB 256 ?
> d44b0f6d-7933-4a7c-ba7b-f8648e038f85 RAC1
> UN 10.210.3.160 497.6 GB 256 ?
> e7332179-a47e-471d-bcd4-08c638ab9ea4 RAC1
> UN 10.210.3.224 334.25 GB 256 ?
> 92b0bd8c-0a5a-446a-83ea-2feea4988fe3 RAC1
> UN 10.210.3.118 518.34 GB 256 ?
> ebddeaf3-1433-4372-a4ca-9c7ba3d4a26b RAC1
> UN 10.210.3.221 516.57 GB 256 ?
> 44d67a49-5310-4ab5-b448-a44be350abf5 RAC1
> UN 10.210.3.117 493.83 GB 256 ?
> aae92956-82d6-421e-8f3f-22393ac7e5f7 RAC1
> Datacenter: Analytics
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.59.124 392.83 GB 320 ?
> f770a8cc-b7bf-44ac-8cc0-214d9228dfcd RAC1
> UN 10.210.59.151 411.9 GB 320 ?
> 3cc87422-0e43-4cd1-91bf-484f121be072 RAC1
> UN 10.210.58.132 309.8 GB 256 ?
> 84d94d13-28d3-4b49-a3d9-557ab47e79b9 RAC1
> UN 10.21
[jira] [Commented] (CASSANDRA-10991) Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
[
https://issues.apache.org/jira/browse/CASSANDRA-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106787#comment-15106787
]
Marcus Eriksson commented on CASSANDRA-10991:
-
[~mlowicki] could you post 'describe OpsCenter'? Is it possible you have RF=0
in the DC where it fails like this?
> Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
> --
>
> Key: CASSANDRA-10991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10991
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
>Reporter: mlowicki
> Fix For: 2.1.12
>
>
> I've C* cluster spread across 3 DCs. Running {{cleanup}} on all nodes in one
> DC always fails:
> {code}
> root@db1:~# nt cleanup system
> root@db1:~# nt cleanup sync
> root@db1:~# nt cleanup OpsCenter
> Aborted cleaning up atleast one column family in keyspace OpsCenter, check
> server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
> root@db1:~#
> {code}
> Checked two other DCs and running cleanup there works fine (it didn't fail
> immediately).
> Output from {{nodetool status}} from one node in problematic DC:
> {code}
> root@db1:~# nt status
> Datacenter: Amsterdam
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.3.162 518.54 GB 256 ?
> 50e606f5-e893-4a3b-86d3-1e5986dceea9 RAC1
> UN 10.210.3.230 532.63 GB 256 ?
> 7b8fc988-8a6a-4d94-ae84-ab9da9ab01e8 RAC1
> UN 10.210.3.161 538.82 GB 256 ?
> d44b0f6d-7933-4a7c-ba7b-f8648e038f85 RAC1
> UN 10.210.3.160 497.6 GB 256 ?
> e7332179-a47e-471d-bcd4-08c638ab9ea4 RAC1
> UN 10.210.3.224 334.25 GB 256 ?
> 92b0bd8c-0a5a-446a-83ea-2feea4988fe3 RAC1
> UN 10.210.3.118 518.34 GB 256 ?
> ebddeaf3-1433-4372-a4ca-9c7ba3d4a26b RAC1
> UN 10.210.3.221 516.57 GB 256 ?
> 44d67a49-5310-4ab5-b448-a44be350abf5 RAC1
> UN 10.210.3.117 493.83 GB 256 ?
> aae92956-82d6-421e-8f3f-22393ac7e5f7 RAC1
> Datacenter: Analytics
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.59.124 392.83 GB 320 ?
> f770a8cc-b7bf-44ac-8cc0-214d9228dfcd RAC1
> UN 10.210.59.151 411.9 GB 320 ?
> 3cc87422-0e43-4cd1-91bf-484f121be072 RAC1
> UN 10.210.58.132 309.8 GB 256 ?
> 84d94d13-28d3-4b49-a3d9-557ab47e79b9 RAC1
> UN 10.210.58.133 281.82 GB 256 ?
> 02bd2d02-41c5-4193-81b0-dee434adb0da RAC1
> UN 10.210.59.86 285.84 GB 256 ?
> bc6422ea-22e9-431a-ac16-c4c040f0c4e5 RAC1
> UN 10.210.59.84 331.06 GB 256 ?
> a798e6b0-3a84-4ec2-82bb-8474086cb315 RAC1
> UN 10.210.59.85 366.26 GB 256 ?
> 52699077-56cf-4c1e-b308-bf79a1644b7e RAC1
> Datacenter: Ashburn
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.195.15.176 534.51 GB 256 ?
> c6ac22df-c43a-4b25-b3b5-5e12ce9c69da RAC1
> UN 10.195.15.177 313.73 GB 256 ?
> eafa2a72-84a2-4cdc-a634-3c660acc6af8 RAC1
> UN 10.195.15.163 470.92 GB 256 ?
> bcd2a534-94c4-4406-8d16-c1fc26b41844 RAC1
> UN 10.195.15.162 539.82 GB 256 ?
> bb649cef-21de-4077-a35f-994319011a06 RAC1
> UN 10.195.15.182 499.64 GB 256 ?
> 6ce2d14d-9fb8-4494-8e97-3add05bd35de RAC1
> UN 10.195.15.167 508.48 GB 256 ?
> 6f359675-852a-4842-9ff2-bdc69e6b04a2 RAC1
> UN 10.195.15.166 490.28 GB 256 ?
> 1ec5d0c5-e8bd-4973-96d9-523de91d08c5 RAC1
> UN 10.195.15.183 447.78 GB 256 ?
> 824165b0-1f1b-40e8-9695-e2f596cb8611 RAC1
> Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless
> {code}
> Logs from one of the nodes where {{cleanup}} fails:
> {code}
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,942
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,970
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-0
[jira] [Commented] (CASSANDRA-10991) Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
[
https://issues.apache.org/jira/browse/CASSANDRA-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090702#comment-15090702
]
mlowicki commented on CASSANDRA-10991:
--
Yes, all nodes in single DC have this problem. {{nodetool status}} looks the
same in all DCs.
> Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
> --
>
> Key: CASSANDRA-10991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10991
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
>Reporter: mlowicki
> Fix For: 2.1.12
>
>
> I've C* cluster spread across 3 DCs. Running {{cleanup}} on all nodes in one
> DC always fails:
> {code}
> root@db1:~# nt cleanup system
> root@db1:~# nt cleanup sync
> root@db1:~# nt cleanup OpsCenter
> Aborted cleaning up atleast one column family in keyspace OpsCenter, check
> server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
> root@db1:~#
> {code}
> Checked two other DCs and running cleanup there works fine (it didn't fail
> immediately).
> Output from {{nodetool status}} from one node in problematic DC:
> {code}
> root@db1:~# nt status
> Datacenter: Amsterdam
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.3.162 518.54 GB 256 ?
> 50e606f5-e893-4a3b-86d3-1e5986dceea9 RAC1
> UN 10.210.3.230 532.63 GB 256 ?
> 7b8fc988-8a6a-4d94-ae84-ab9da9ab01e8 RAC1
> UN 10.210.3.161 538.82 GB 256 ?
> d44b0f6d-7933-4a7c-ba7b-f8648e038f85 RAC1
> UN 10.210.3.160 497.6 GB 256 ?
> e7332179-a47e-471d-bcd4-08c638ab9ea4 RAC1
> UN 10.210.3.224 334.25 GB 256 ?
> 92b0bd8c-0a5a-446a-83ea-2feea4988fe3 RAC1
> UN 10.210.3.118 518.34 GB 256 ?
> ebddeaf3-1433-4372-a4ca-9c7ba3d4a26b RAC1
> UN 10.210.3.221 516.57 GB 256 ?
> 44d67a49-5310-4ab5-b448-a44be350abf5 RAC1
> UN 10.210.3.117 493.83 GB 256 ?
> aae92956-82d6-421e-8f3f-22393ac7e5f7 RAC1
> Datacenter: Analytics
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.59.124 392.83 GB 320 ?
> f770a8cc-b7bf-44ac-8cc0-214d9228dfcd RAC1
> UN 10.210.59.151 411.9 GB 320 ?
> 3cc87422-0e43-4cd1-91bf-484f121be072 RAC1
> UN 10.210.58.132 309.8 GB 256 ?
> 84d94d13-28d3-4b49-a3d9-557ab47e79b9 RAC1
> UN 10.210.58.133 281.82 GB 256 ?
> 02bd2d02-41c5-4193-81b0-dee434adb0da RAC1
> UN 10.210.59.86 285.84 GB 256 ?
> bc6422ea-22e9-431a-ac16-c4c040f0c4e5 RAC1
> UN 10.210.59.84 331.06 GB 256 ?
> a798e6b0-3a84-4ec2-82bb-8474086cb315 RAC1
> UN 10.210.59.85 366.26 GB 256 ?
> 52699077-56cf-4c1e-b308-bf79a1644b7e RAC1
> Datacenter: Ashburn
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.195.15.176 534.51 GB 256 ?
> c6ac22df-c43a-4b25-b3b5-5e12ce9c69da RAC1
> UN 10.195.15.177 313.73 GB 256 ?
> eafa2a72-84a2-4cdc-a634-3c660acc6af8 RAC1
> UN 10.195.15.163 470.92 GB 256 ?
> bcd2a534-94c4-4406-8d16-c1fc26b41844 RAC1
> UN 10.195.15.162 539.82 GB 256 ?
> bb649cef-21de-4077-a35f-994319011a06 RAC1
> UN 10.195.15.182 499.64 GB 256 ?
> 6ce2d14d-9fb8-4494-8e97-3add05bd35de RAC1
> UN 10.195.15.167 508.48 GB 256 ?
> 6f359675-852a-4842-9ff2-bdc69e6b04a2 RAC1
> UN 10.195.15.166 490.28 GB 256 ?
> 1ec5d0c5-e8bd-4973-96d9-523de91d08c5 RAC1
> UN 10.195.15.183 447.78 GB 256 ?
> 824165b0-1f1b-40e8-9695-e2f596cb8611 RAC1
> Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless
> {code}
> Logs from one of the nodes where {{cleanup}} fails:
> {code}
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,942
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,970
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:34,000
> CompactionMan
[jira] [Commented] (CASSANDRA-10991) Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
[
https://issues.apache.org/jira/browse/CASSANDRA-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090701#comment-15090701
]
Michael Shuler commented on CASSANDRA-10991:
I'm sorry, I may have mis-read that - all nodes in one of the DCs have this
problem?
> Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
> --
>
> Key: CASSANDRA-10991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10991
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
>Reporter: mlowicki
> Fix For: 2.1.12
>
>
> I've C* cluster spread across 3 DCs. Running {{cleanup}} on all nodes in one
> DC always fails:
> {code}
> root@db1:~# nt cleanup system
> root@db1:~# nt cleanup sync
> root@db1:~# nt cleanup OpsCenter
> Aborted cleaning up atleast one column family in keyspace OpsCenter, check
> server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
> root@db1:~#
> {code}
> Checked two other DCs and running cleanup there works fine (it didn't fail
> immediately).
> Output from {{nodetool status}} from one node in problematic DC:
> {code}
> root@db1:~# nt status
> Datacenter: Amsterdam
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.3.162 518.54 GB 256 ?
> 50e606f5-e893-4a3b-86d3-1e5986dceea9 RAC1
> UN 10.210.3.230 532.63 GB 256 ?
> 7b8fc988-8a6a-4d94-ae84-ab9da9ab01e8 RAC1
> UN 10.210.3.161 538.82 GB 256 ?
> d44b0f6d-7933-4a7c-ba7b-f8648e038f85 RAC1
> UN 10.210.3.160 497.6 GB 256 ?
> e7332179-a47e-471d-bcd4-08c638ab9ea4 RAC1
> UN 10.210.3.224 334.25 GB 256 ?
> 92b0bd8c-0a5a-446a-83ea-2feea4988fe3 RAC1
> UN 10.210.3.118 518.34 GB 256 ?
> ebddeaf3-1433-4372-a4ca-9c7ba3d4a26b RAC1
> UN 10.210.3.221 516.57 GB 256 ?
> 44d67a49-5310-4ab5-b448-a44be350abf5 RAC1
> UN 10.210.3.117 493.83 GB 256 ?
> aae92956-82d6-421e-8f3f-22393ac7e5f7 RAC1
> Datacenter: Analytics
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.59.124 392.83 GB 320 ?
> f770a8cc-b7bf-44ac-8cc0-214d9228dfcd RAC1
> UN 10.210.59.151 411.9 GB 320 ?
> 3cc87422-0e43-4cd1-91bf-484f121be072 RAC1
> UN 10.210.58.132 309.8 GB 256 ?
> 84d94d13-28d3-4b49-a3d9-557ab47e79b9 RAC1
> UN 10.210.58.133 281.82 GB 256 ?
> 02bd2d02-41c5-4193-81b0-dee434adb0da RAC1
> UN 10.210.59.86 285.84 GB 256 ?
> bc6422ea-22e9-431a-ac16-c4c040f0c4e5 RAC1
> UN 10.210.59.84 331.06 GB 256 ?
> a798e6b0-3a84-4ec2-82bb-8474086cb315 RAC1
> UN 10.210.59.85 366.26 GB 256 ?
> 52699077-56cf-4c1e-b308-bf79a1644b7e RAC1
> Datacenter: Ashburn
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.195.15.176 534.51 GB 256 ?
> c6ac22df-c43a-4b25-b3b5-5e12ce9c69da RAC1
> UN 10.195.15.177 313.73 GB 256 ?
> eafa2a72-84a2-4cdc-a634-3c660acc6af8 RAC1
> UN 10.195.15.163 470.92 GB 256 ?
> bcd2a534-94c4-4406-8d16-c1fc26b41844 RAC1
> UN 10.195.15.162 539.82 GB 256 ?
> bb649cef-21de-4077-a35f-994319011a06 RAC1
> UN 10.195.15.182 499.64 GB 256 ?
> 6ce2d14d-9fb8-4494-8e97-3add05bd35de RAC1
> UN 10.195.15.167 508.48 GB 256 ?
> 6f359675-852a-4842-9ff2-bdc69e6b04a2 RAC1
> UN 10.195.15.166 490.28 GB 256 ?
> 1ec5d0c5-e8bd-4973-96d9-523de91d08c5 RAC1
> UN 10.195.15.183 447.78 GB 256 ?
> 824165b0-1f1b-40e8-9695-e2f596cb8611 RAC1
> Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless
> {code}
> Logs from one of the nodes where {{cleanup}} fails:
> {code}
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,942
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,970
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:34,000
> Compaction
[jira] [Commented] (CASSANDRA-10991) Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
[
https://issues.apache.org/jira/browse/CASSANDRA-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090700#comment-15090700
]
Michael Shuler commented on CASSANDRA-10991:
{{nodetool status}} on the problematic node, itself; does that appear any
different than other nodes in the same DC?
> Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
> --
>
> Key: CASSANDRA-10991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10991
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
>Reporter: mlowicki
> Fix For: 2.1.12
>
>
> I've C* cluster spread across 3 DCs. Running {{cleanup}} on all nodes in one
> DC always fails:
> {code}
> root@db1:~# nt cleanup system
> root@db1:~# nt cleanup sync
> root@db1:~# nt cleanup OpsCenter
> Aborted cleaning up atleast one column family in keyspace OpsCenter, check
> server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
> root@db1:~#
> {code}
> Checked two other DCs and running cleanup there works fine (it didn't fail
> immediately).
> Output from {{nodetool status}} from one node in problematic DC:
> {code}
> root@db1:~# nt status
> Datacenter: Amsterdam
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.3.162 518.54 GB 256 ?
> 50e606f5-e893-4a3b-86d3-1e5986dceea9 RAC1
> UN 10.210.3.230 532.63 GB 256 ?
> 7b8fc988-8a6a-4d94-ae84-ab9da9ab01e8 RAC1
> UN 10.210.3.161 538.82 GB 256 ?
> d44b0f6d-7933-4a7c-ba7b-f8648e038f85 RAC1
> UN 10.210.3.160 497.6 GB 256 ?
> e7332179-a47e-471d-bcd4-08c638ab9ea4 RAC1
> UN 10.210.3.224 334.25 GB 256 ?
> 92b0bd8c-0a5a-446a-83ea-2feea4988fe3 RAC1
> UN 10.210.3.118 518.34 GB 256 ?
> ebddeaf3-1433-4372-a4ca-9c7ba3d4a26b RAC1
> UN 10.210.3.221 516.57 GB 256 ?
> 44d67a49-5310-4ab5-b448-a44be350abf5 RAC1
> UN 10.210.3.117 493.83 GB 256 ?
> aae92956-82d6-421e-8f3f-22393ac7e5f7 RAC1
> Datacenter: Analytics
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.59.124 392.83 GB 320 ?
> f770a8cc-b7bf-44ac-8cc0-214d9228dfcd RAC1
> UN 10.210.59.151 411.9 GB 320 ?
> 3cc87422-0e43-4cd1-91bf-484f121be072 RAC1
> UN 10.210.58.132 309.8 GB 256 ?
> 84d94d13-28d3-4b49-a3d9-557ab47e79b9 RAC1
> UN 10.210.58.133 281.82 GB 256 ?
> 02bd2d02-41c5-4193-81b0-dee434adb0da RAC1
> UN 10.210.59.86 285.84 GB 256 ?
> bc6422ea-22e9-431a-ac16-c4c040f0c4e5 RAC1
> UN 10.210.59.84 331.06 GB 256 ?
> a798e6b0-3a84-4ec2-82bb-8474086cb315 RAC1
> UN 10.210.59.85 366.26 GB 256 ?
> 52699077-56cf-4c1e-b308-bf79a1644b7e RAC1
> Datacenter: Ashburn
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.195.15.176 534.51 GB 256 ?
> c6ac22df-c43a-4b25-b3b5-5e12ce9c69da RAC1
> UN 10.195.15.177 313.73 GB 256 ?
> eafa2a72-84a2-4cdc-a634-3c660acc6af8 RAC1
> UN 10.195.15.163 470.92 GB 256 ?
> bcd2a534-94c4-4406-8d16-c1fc26b41844 RAC1
> UN 10.195.15.162 539.82 GB 256 ?
> bb649cef-21de-4077-a35f-994319011a06 RAC1
> UN 10.195.15.182 499.64 GB 256 ?
> 6ce2d14d-9fb8-4494-8e97-3add05bd35de RAC1
> UN 10.195.15.167 508.48 GB 256 ?
> 6f359675-852a-4842-9ff2-bdc69e6b04a2 RAC1
> UN 10.195.15.166 490.28 GB 256 ?
> 1ec5d0c5-e8bd-4973-96d9-523de91d08c5 RAC1
> UN 10.195.15.183 447.78 GB 256 ?
> 824165b0-1f1b-40e8-9695-e2f596cb8611 RAC1
> Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless
> {code}
> Logs from one of the nodes where {{cleanup}} fails:
> {code}
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,942
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,970
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-
[jira] [Commented] (CASSANDRA-10991) Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
[
https://issues.apache.org/jira/browse/CASSANDRA-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090699#comment-15090699
]
mlowicki commented on CASSANDRA-10991:
--
According to {{nodetool status}} and looking at metrics node joined the ring
many weeks ago.
> Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
> --
>
> Key: CASSANDRA-10991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10991
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
>Reporter: mlowicki
> Fix For: 2.1.12
>
>
> I've C* cluster spread across 3 DCs. Running {{cleanup}} on all nodes in one
> DC always fails:
> {code}
> root@db1:~# nt cleanup system
> root@db1:~# nt cleanup sync
> root@db1:~# nt cleanup OpsCenter
> Aborted cleaning up atleast one column family in keyspace OpsCenter, check
> server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
> root@db1:~#
> {code}
> Checked two other DCs and running cleanup there works fine (it didn't fail
> immediately).
> Output from {{nodetool status}} from one node in problematic DC:
> {code}
> root@db1:~# nt status
> Datacenter: Amsterdam
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.3.162 518.54 GB 256 ?
> 50e606f5-e893-4a3b-86d3-1e5986dceea9 RAC1
> UN 10.210.3.230 532.63 GB 256 ?
> 7b8fc988-8a6a-4d94-ae84-ab9da9ab01e8 RAC1
> UN 10.210.3.161 538.82 GB 256 ?
> d44b0f6d-7933-4a7c-ba7b-f8648e038f85 RAC1
> UN 10.210.3.160 497.6 GB 256 ?
> e7332179-a47e-471d-bcd4-08c638ab9ea4 RAC1
> UN 10.210.3.224 334.25 GB 256 ?
> 92b0bd8c-0a5a-446a-83ea-2feea4988fe3 RAC1
> UN 10.210.3.118 518.34 GB 256 ?
> ebddeaf3-1433-4372-a4ca-9c7ba3d4a26b RAC1
> UN 10.210.3.221 516.57 GB 256 ?
> 44d67a49-5310-4ab5-b448-a44be350abf5 RAC1
> UN 10.210.3.117 493.83 GB 256 ?
> aae92956-82d6-421e-8f3f-22393ac7e5f7 RAC1
> Datacenter: Analytics
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.59.124 392.83 GB 320 ?
> f770a8cc-b7bf-44ac-8cc0-214d9228dfcd RAC1
> UN 10.210.59.151 411.9 GB 320 ?
> 3cc87422-0e43-4cd1-91bf-484f121be072 RAC1
> UN 10.210.58.132 309.8 GB 256 ?
> 84d94d13-28d3-4b49-a3d9-557ab47e79b9 RAC1
> UN 10.210.58.133 281.82 GB 256 ?
> 02bd2d02-41c5-4193-81b0-dee434adb0da RAC1
> UN 10.210.59.86 285.84 GB 256 ?
> bc6422ea-22e9-431a-ac16-c4c040f0c4e5 RAC1
> UN 10.210.59.84 331.06 GB 256 ?
> a798e6b0-3a84-4ec2-82bb-8474086cb315 RAC1
> UN 10.210.59.85 366.26 GB 256 ?
> 52699077-56cf-4c1e-b308-bf79a1644b7e RAC1
> Datacenter: Ashburn
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.195.15.176 534.51 GB 256 ?
> c6ac22df-c43a-4b25-b3b5-5e12ce9c69da RAC1
> UN 10.195.15.177 313.73 GB 256 ?
> eafa2a72-84a2-4cdc-a634-3c660acc6af8 RAC1
> UN 10.195.15.163 470.92 GB 256 ?
> bcd2a534-94c4-4406-8d16-c1fc26b41844 RAC1
> UN 10.195.15.162 539.82 GB 256 ?
> bb649cef-21de-4077-a35f-994319011a06 RAC1
> UN 10.195.15.182 499.64 GB 256 ?
> 6ce2d14d-9fb8-4494-8e97-3add05bd35de RAC1
> UN 10.195.15.167 508.48 GB 256 ?
> 6f359675-852a-4842-9ff2-bdc69e6b04a2 RAC1
> UN 10.195.15.166 490.28 GB 256 ?
> 1ec5d0c5-e8bd-4973-96d9-523de91d08c5 RAC1
> UN 10.195.15.183 447.78 GB 256 ?
> 824165b0-1f1b-40e8-9695-e2f596cb8611 RAC1
> Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless
> {code}
> Logs from one of the nodes where {{cleanup}} fails:
> {code}
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,942
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,970
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:34,000
> CompactionMana
[jira] [Commented] (CASSANDRA-10991) Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
[
https://issues.apache.org/jira/browse/CASSANDRA-10991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090696#comment-15090696
]
Michael Shuler commented on CASSANDRA-10991:
Did you {{check server logs for more information}}?
> Cleanup OpsCenter keyspace fails - node thinks that didn't joined the ring yet
> --
>
> Key: CASSANDRA-10991
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10991
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.12, Debian Wheezy
>Reporter: mlowicki
> Fix For: 2.1.12
>
>
> I've C* cluster spread across 3 DCs. Running {{cleanup}} on all nodes in one
> DC always fails:
> {code}
> root@db1:~# nt cleanup system
> root@db1:~# nt cleanup sync
> root@db1:~# nt cleanup OpsCenter
> Aborted cleaning up atleast one column family in keyspace OpsCenter, check
> server logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:292)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:204)
> root@db1:~#
> {code}
> Checked two other DCs and running cleanup there works fine (it didn't fail
> immediately).
> Output from {{nodetool status}} from one node in problematic DC:
> {code}
> root@db1:~# nt status
> Datacenter: Amsterdam
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.3.162 518.54 GB 256 ?
> 50e606f5-e893-4a3b-86d3-1e5986dceea9 RAC1
> UN 10.210.3.230 532.63 GB 256 ?
> 7b8fc988-8a6a-4d94-ae84-ab9da9ab01e8 RAC1
> UN 10.210.3.161 538.82 GB 256 ?
> d44b0f6d-7933-4a7c-ba7b-f8648e038f85 RAC1
> UN 10.210.3.160 497.6 GB 256 ?
> e7332179-a47e-471d-bcd4-08c638ab9ea4 RAC1
> UN 10.210.3.224 334.25 GB 256 ?
> 92b0bd8c-0a5a-446a-83ea-2feea4988fe3 RAC1
> UN 10.210.3.118 518.34 GB 256 ?
> ebddeaf3-1433-4372-a4ca-9c7ba3d4a26b RAC1
> UN 10.210.3.221 516.57 GB 256 ?
> 44d67a49-5310-4ab5-b448-a44be350abf5 RAC1
> UN 10.210.3.117 493.83 GB 256 ?
> aae92956-82d6-421e-8f3f-22393ac7e5f7 RAC1
> Datacenter: Analytics
> =
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.210.59.124 392.83 GB 320 ?
> f770a8cc-b7bf-44ac-8cc0-214d9228dfcd RAC1
> UN 10.210.59.151 411.9 GB 320 ?
> 3cc87422-0e43-4cd1-91bf-484f121be072 RAC1
> UN 10.210.58.132 309.8 GB 256 ?
> 84d94d13-28d3-4b49-a3d9-557ab47e79b9 RAC1
> UN 10.210.58.133 281.82 GB 256 ?
> 02bd2d02-41c5-4193-81b0-dee434adb0da RAC1
> UN 10.210.59.86 285.84 GB 256 ?
> bc6422ea-22e9-431a-ac16-c4c040f0c4e5 RAC1
> UN 10.210.59.84 331.06 GB 256 ?
> a798e6b0-3a84-4ec2-82bb-8474086cb315 RAC1
> UN 10.210.59.85 366.26 GB 256 ?
> 52699077-56cf-4c1e-b308-bf79a1644b7e RAC1
> Datacenter: Ashburn
> ===
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- AddressLoad Tokens OwnsHost ID
> Rack
> UN 10.195.15.176 534.51 GB 256 ?
> c6ac22df-c43a-4b25-b3b5-5e12ce9c69da RAC1
> UN 10.195.15.177 313.73 GB 256 ?
> eafa2a72-84a2-4cdc-a634-3c660acc6af8 RAC1
> UN 10.195.15.163 470.92 GB 256 ?
> bcd2a534-94c4-4406-8d16-c1fc26b41844 RAC1
> UN 10.195.15.162 539.82 GB 256 ?
> bb649cef-21de-4077-a35f-994319011a06 RAC1
> UN 10.195.15.182 499.64 GB 256 ?
> 6ce2d14d-9fb8-4494-8e97-3add05bd35de RAC1
> UN 10.195.15.167 508.48 GB 256 ?
> 6f359675-852a-4842-9ff2-bdc69e6b04a2 RAC1
> UN 10.195.15.166 490.28 GB 256 ?
> 1ec5d0c5-e8bd-4973-96d9-523de91d08c5 RAC1
> UN 10.195.15.183 447.78 GB 256 ?
> 824165b0-1f1b-40e8-9695-e2f596cb8611 RAC1
> Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless
> {code}
> Logs from one of the nodes where {{cleanup}} fails:
> {code}
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,942
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:33,970
> CompactionManager.java:388 - Cleanup cannot run before a node has joined the
> ring
> INFO [RMI TCP Connection(158004)-10.210.59.86] 2016-01-09 15:58:34,000
> CompactionManager.java:388 - Cleanup cannot
