If you didn't explicitly remove a keyspace from one of your datacenters, the next most likely cause is that you have one keyspace that's NOT replicated to one of the datacenters. You can work around this by running 'nodetool cleanup <ks>' on all of your other keyspaces individually, skipping the one that isn't replicated to that datacenter.
On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Thanks Jeff, > > I have a C* cluster spread across multiple datacenter. > reason for cleanup : I added multiple nodes to cluster and need to run > cleanup on old nodes so that the redundant data is cleaned-up. > > On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa <jji...@apache.org> wrote: > >> >> >> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada < >> jaibheem...@gmail.com> wrote: >> > Hello, >> > >> > I am running into an issue where *nodetool cleanup *fails to cleanup >> data. >> > We are running 2.1.16 version of Cassandra. >> > >> > >> > [user@host ~]$ nodetool cleanup >> > Aborted cleaning up atleast one column family in keyspace user, check >> > server logs for more information. >> > Aborted cleaning up atleast one column family in keyspace org, check >> server >> > logs for more information. >> > error: nodetool failed, check server logs >> > -- StackTrace -- >> > java.lang.RuntimeException: nodetool failed, check server logs >> > at >> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294) >> > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206) >> > >> > *Logs:* >> > >> > INFO [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987 >> > CompactionManager.java:415 - Cleanup cannot run before a node has joined >> > the ring >> > INFO [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010 >> > CompactionManager.java:415 - Cleanup cannot run before a node has joined >> > the ring >> > >> > All the nodes in the cluster are up and running. We tried doing a >> rolling >> > restart of all nodes and no luck. >> > >> > After looking at the Cassandra JIRA : >> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the >> issue >> > is fixed with 2.2.6 and 3.0 version. >> > While we have plans to upgrade to the latest versions(which might take >> > longer time), does any know if there is any work around to mitigate the >> > issue? >> > >> >> Are you running multiple datacenters, and you just removed a specific >> datacenter from a keyspace (and that's why you want to run cleanup)? If >> that's the case, I fear the fix for 10991 isn't really going to fix it in >> the way you hope (we may need a follow-up jira). What you'll almost >> certainly need to do is remove the data on disk manually, which is quite >> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start >> each node in the datacenter for which you removed replication. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >> >