Re: Nodetool cleanup doesn't work

[Jai Bheemsen Rao Dhanwada]

ok thank you

On Thu, May 11, 2017 at 1:11 PM, Jeff Jirsa  wrote:

> No, it's not expected, but it's pretty obvious from reading the code
> what'll happen. Opened https://issues.apache.org/jira/browse/CASSANDRA-
> 13526
>
>
>
>
>
> On Thu, May 11, 2017 at 12:53 PM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Yes I have many keyspaces which are not spread across all the data
>> centers(expected by design).
>> In this case, is this the expected behavior cleanup will not work for all
>> the keyspaces(nodetool cleanup)? is it going to be fixed in the latest
>> versions?
>>
>> P.S: Thanks for the tip, I can workaround this by "nodetool cleanup
>> keyspacename"
>>
>> On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa  wrote:
>>
>>> If you didn't explicitly remove a keyspace from one of your datacenters,
>>> the next most likely cause is that you have one keyspace that's NOT
>>> replicated to one of the datacenters. You can work around this by running
>>> 'nodetool cleanup ' on all of your other keyspaces individually,
>>> skipping the one that isn't replicated to that datacenter.
>>>
>>>
>>>
>>> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>>
 Thanks Jeff,

 I have a C* cluster spread across multiple datacenter.
 reason for cleanup : I added multiple nodes to cluster and need to run
 cleanup on old nodes so that the redundant data is cleaned-up.

 On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:

>
>
> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
> > Hello,
> >
> > I am running into an issue where *nodetool cleanup *fails to cleanup
> data.
> > We are running 2.1.16 version of Cassandra.
> >
> >
> > [user@host ~]$ nodetool cleanup
> > Aborted cleaning up atleast one column family in keyspace user, check
> > server logs for more information.
> > Aborted cleaning up atleast one column family in keyspace org, check
> server
> > logs for more information.
> > error: nodetool failed, check server logs
> > -- StackTrace --
> > java.lang.RuntimeException: nodetool failed, check server logs
> > at
> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool
> .java:294)
> > at org.apache.cassandra.tools.Nod
> eTool.main(NodeTool.java:206)
> >
> > *Logs:*
> >
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
> > CompactionManager.java:415 - Cleanup cannot run before a node has
> joined
> > the ring
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
> > CompactionManager.java:415 - Cleanup cannot run before a node has
> joined
> > the ring
> >
> > All the nodes in the cluster are up and running. We tried doing a
> rolling
> > restart of all nodes and no luck.
> >
> > After looking at the Cassandra JIRA :
> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like
> the issue
> > is fixed with 2.2.6 and 3.0 version.
> > While we have plans to upgrade to the latest versions(which might
> take
> > longer time), does any know if there is any work around to mitigate
> the
> > issue?
> >
>
> Are you running multiple datacenters, and you just removed a specific
> datacenter from a keyspace (and that's why you want to run cleanup)? If
> that's the case, I fear the fix for 10991 isn't really going to fix it in
> the way you hope (we may need a follow-up jira). What you'll almost
> certainly need to do is remove the data on disk manually, which is quite
> unfortunate as it'll require you to 
> stop+delete-data-for-that-keyspace+start
> each node in the datacenter for which you removed replication.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

>>>
>>
>

Re: Nodetool cleanup doesn't work

[Jeff Jirsa]

No, it's not expected, but it's pretty obvious from reading the code
what'll happen. Opened https://issues.apache.org/jira/browse/CASSANDRA-13526





On Thu, May 11, 2017 at 12:53 PM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Yes I have many keyspaces which are not spread across all the data
> centers(expected by design).
> In this case, is this the expected behavior cleanup will not work for all
> the keyspaces(nodetool cleanup)? is it going to be fixed in the latest
> versions?
>
> P.S: Thanks for the tip, I can workaround this by "nodetool cleanup
> keyspacename"
>
> On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa  wrote:
>
>> If you didn't explicitly remove a keyspace from one of your datacenters,
>> the next most likely cause is that you have one keyspace that's NOT
>> replicated to one of the datacenters. You can work around this by running
>> 'nodetool cleanup ' on all of your other keyspaces individually,
>> skipping the one that isn't replicated to that datacenter.
>>
>>
>>
>> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>>> Thanks Jeff,
>>>
>>> I have a C* cluster spread across multiple datacenter.
>>> reason for cleanup : I added multiple nodes to cluster and need to run
>>> cleanup on old nodes so that the redundant data is cleaned-up.
>>>
>>> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:
>>>


 On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
 jaibheem...@gmail.com> wrote:
 > Hello,
 >
 > I am running into an issue where *nodetool cleanup *fails to cleanup
 data.
 > We are running 2.1.16 version of Cassandra.
 >
 >
 > [user@host ~]$ nodetool cleanup
 > Aborted cleaning up atleast one column family in keyspace user, check
 > server logs for more information.
 > Aborted cleaning up atleast one column family in keyspace org, check
 server
 > logs for more information.
 > error: nodetool failed, check server logs
 > -- StackTrace --
 > java.lang.RuntimeException: nodetool failed, check server logs
 > at
 > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool
 .java:294)
 > at org.apache.cassandra.tools.Nod
 eTool.main(NodeTool.java:206)
 >
 > *Logs:*
 >
 > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
 > CompactionManager.java:415 - Cleanup cannot run before a node has
 joined
 > the ring
 > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
 > CompactionManager.java:415 - Cleanup cannot run before a node has
 joined
 > the ring
 >
 > All the nodes in the cluster are up and running. We tried doing a
 rolling
 > restart of all nodes and no luck.
 >
 > After looking at the Cassandra JIRA :
 > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
 issue
 > is fixed with 2.2.6 and 3.0 version.
 > While we have plans to upgrade to the latest versions(which might take
 > longer time), does any know if there is any work around to mitigate
 the
 > issue?
 >

 Are you running multiple datacenters, and you just removed a specific
 datacenter from a keyspace (and that's why you want to run cleanup)? If
 that's the case, I fear the fix for 10991 isn't really going to fix it in
 the way you hope (we may need a follow-up jira). What you'll almost
 certainly need to do is remove the data on disk manually, which is quite
 unfortunate as it'll require you to 
 stop+delete-data-for-that-keyspace+start
 each node in the datacenter for which you removed replication.

 -
 To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: user-h...@cassandra.apache.org


>>>
>>
>

Re: Nodetool cleanup doesn't work

[Jai Bheemsen Rao Dhanwada]

Yes I have many keyspaces which are not spread across all the data
centers(expected by design).
In this case, is this the expected behavior cleanup will not work for all
the keyspaces(nodetool cleanup)? is it going to be fixed in the latest
versions?

P.S: Thanks for the tip, I can workaround this by "nodetool cleanup
keyspacename"

On Thu, May 11, 2017 at 12:11 PM, Jeff Jirsa  wrote:

> If you didn't explicitly remove a keyspace from one of your datacenters,
> the next most likely cause is that you have one keyspace that's NOT
> replicated to one of the datacenters. You can work around this by running
> 'nodetool cleanup ' on all of your other keyspaces individually,
> skipping the one that isn't replicated to that datacenter.
>
>
>
> On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Thanks Jeff,
>>
>> I have a C* cluster spread across multiple datacenter.
>> reason for cleanup : I added multiple nodes to cluster and need to run
>> cleanup on old nodes so that the redundant data is cleaned-up.
>>
>> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:
>>
>>>
>>>
>>> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
>>> jaibheem...@gmail.com> wrote:
>>> > Hello,
>>> >
>>> > I am running into an issue where *nodetool cleanup *fails to cleanup
>>> data.
>>> > We are running 2.1.16 version of Cassandra.
>>> >
>>> >
>>> > [user@host ~]$ nodetool cleanup
>>> > Aborted cleaning up atleast one column family in keyspace user, check
>>> > server logs for more information.
>>> > Aborted cleaning up atleast one column family in keyspace org, check
>>> server
>>> > logs for more information.
>>> > error: nodetool failed, check server logs
>>> > -- StackTrace --
>>> > java.lang.RuntimeException: nodetool failed, check server logs
>>> > at
>>> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
>>> > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
>>> >
>>> > *Logs:*
>>> >
>>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
>>> > CompactionManager.java:415 - Cleanup cannot run before a node has
>>> joined
>>> > the ring
>>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
>>> > CompactionManager.java:415 - Cleanup cannot run before a node has
>>> joined
>>> > the ring
>>> >
>>> > All the nodes in the cluster are up and running. We tried doing a
>>> rolling
>>> > restart of all nodes and no luck.
>>> >
>>> > After looking at the Cassandra JIRA :
>>> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
>>> issue
>>> > is fixed with 2.2.6 and 3.0 version.
>>> > While we have plans to upgrade to the latest versions(which might take
>>> > longer time), does any know if there is any work around to mitigate the
>>> > issue?
>>> >
>>>
>>> Are you running multiple datacenters, and you just removed a specific
>>> datacenter from a keyspace (and that's why you want to run cleanup)? If
>>> that's the case, I fear the fix for 10991 isn't really going to fix it in
>>> the way you hope (we may need a follow-up jira). What you'll almost
>>> certainly need to do is remove the data on disk manually, which is quite
>>> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start
>>> each node in the datacenter for which you removed replication.
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>>
>>>
>>
>

Re: Nodetool cleanup doesn't work

[Jeff Jirsa]

If you didn't explicitly remove a keyspace from one of your datacenters,
the next most likely cause is that you have one keyspace that's NOT
replicated to one of the datacenters. You can work around this by running
'nodetool cleanup ' on all of your other keyspaces individually,
skipping the one that isn't replicated to that datacenter.



On Thu, May 11, 2017 at 11:19 AM, Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Thanks Jeff,
>
> I have a C* cluster spread across multiple datacenter.
> reason for cleanup : I added multiple nodes to cluster and need to run
> cleanup on old nodes so that the redundant data is cleaned-up.
>
> On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:
>
>>
>>
>> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>> > Hello,
>> >
>> > I am running into an issue where *nodetool cleanup *fails to cleanup
>> data.
>> > We are running 2.1.16 version of Cassandra.
>> >
>> >
>> > [user@host ~]$ nodetool cleanup
>> > Aborted cleaning up atleast one column family in keyspace user, check
>> > server logs for more information.
>> > Aborted cleaning up atleast one column family in keyspace org, check
>> server
>> > logs for more information.
>> > error: nodetool failed, check server logs
>> > -- StackTrace --
>> > java.lang.RuntimeException: nodetool failed, check server logs
>> > at
>> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
>> > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
>> >
>> > *Logs:*
>> >
>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
>> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
>> > the ring
>> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
>> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
>> > the ring
>> >
>> > All the nodes in the cluster are up and running. We tried doing a
>> rolling
>> > restart of all nodes and no luck.
>> >
>> > After looking at the Cassandra JIRA :
>> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
>> issue
>> > is fixed with 2.2.6 and 3.0 version.
>> > While we have plans to upgrade to the latest versions(which might take
>> > longer time), does any know if there is any work around to mitigate the
>> > issue?
>> >
>>
>> Are you running multiple datacenters, and you just removed a specific
>> datacenter from a keyspace (and that's why you want to run cleanup)? If
>> that's the case, I fear the fix for 10991 isn't really going to fix it in
>> the way you hope (we may need a follow-up jira). What you'll almost
>> certainly need to do is remove the data on disk manually, which is quite
>> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start
>> each node in the datacenter for which you removed replication.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: user-h...@cassandra.apache.org
>>
>>
>

Re: Nodetool cleanup doesn't work

[Jai Bheemsen Rao Dhanwada]

Thanks Jeff,

I have a C* cluster spread across multiple datacenter.
reason for cleanup : I added multiple nodes to cluster and need to run
cleanup on old nodes so that the redundant data is cleaned-up.

On Thu, May 11, 2017 at 11:08 AM, Jeff Jirsa  wrote:

>
>
> On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
> > Hello,
> >
> > I am running into an issue where *nodetool cleanup *fails to cleanup
> data.
> > We are running 2.1.16 version of Cassandra.
> >
> >
> > [user@host ~]$ nodetool cleanup
> > Aborted cleaning up atleast one column family in keyspace user, check
> > server logs for more information.
> > Aborted cleaning up atleast one column family in keyspace org, check
> server
> > logs for more information.
> > error: nodetool failed, check server logs
> > -- StackTrace --
> > java.lang.RuntimeException: nodetool failed, check server logs
> > at
> > org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
> > at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
> >
> > *Logs:*
> >
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
> > the ring
> > INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
> > CompactionManager.java:415 - Cleanup cannot run before a node has joined
> > the ring
> >
> > All the nodes in the cluster are up and running. We tried doing a rolling
> > restart of all nodes and no luck.
> >
> > After looking at the Cassandra JIRA :
> > https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the
> issue
> > is fixed with 2.2.6 and 3.0 version.
> > While we have plans to upgrade to the latest versions(which might take
> > longer time), does any know if there is any work around to mitigate the
> > issue?
> >
>
> Are you running multiple datacenters, and you just removed a specific
> datacenter from a keyspace (and that's why you want to run cleanup)? If
> that's the case, I fear the fix for 10991 isn't really going to fix it in
> the way you hope (we may need a follow-up jira). What you'll almost
> certainly need to do is remove the data on disk manually, which is quite
> unfortunate as it'll require you to stop+delete-data-for-that-keyspace+start
> each node in the datacenter for which you removed replication.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>

Re: Nodetool cleanup doesn't work

[Jeff Jirsa]

On 2017-05-10 22:44 (-0700), Jai Bheemsen Rao Dhanwada  
wrote: 
> Hello,
> 
> I am running into an issue where *nodetool cleanup *fails to cleanup data.
> We are running 2.1.16 version of Cassandra.
> 
> 
> [user@host ~]$ nodetool cleanup
> Aborted cleaning up atleast one column family in keyspace user, check
> server logs for more information.
> Aborted cleaning up atleast one column family in keyspace org, check server
> logs for more information.
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)
> 
> *Logs:*
> 
> INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
> CompactionManager.java:415 - Cleanup cannot run before a node has joined
> the ring
> INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
> CompactionManager.java:415 - Cleanup cannot run before a node has joined
> the ring
> 
> All the nodes in the cluster are up and running. We tried doing a rolling
> restart of all nodes and no luck.
> 
> After looking at the Cassandra JIRA :
> https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the issue
> is fixed with 2.2.6 and 3.0 version.
> While we have plans to upgrade to the latest versions(which might take
> longer time), does any know if there is any work around to mitigate the
> issue?
> 

Are you running multiple datacenters, and you just removed a specific 
datacenter from a keyspace (and that's why you want to run cleanup)? If that's 
the case, I fear the fix for 10991 isn't really going to fix it in the way you 
hope (we may need a follow-up jira). What you'll almost certainly need to do is 
remove the data on disk manually, which is quite unfortunate as it'll require 
you to stop+delete-data-for-that-keyspace+start each node in the datacenter for 
which you removed replication. 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Nodetool cleanup doesn't work

[Jai Bheemsen Rao Dhanwada]

Hello,

I am running into an issue where *nodetool cleanup *fails to cleanup data.
We are running 2.1.16 version of Cassandra.


[user@host ~]$ nodetool cleanup
Aborted cleaning up atleast one column family in keyspace user, check
server logs for more information.
Aborted cleaning up atleast one column family in keyspace org, check server
logs for more information.
error: nodetool failed, check server logs
-- StackTrace --
java.lang.RuntimeException: nodetool failed, check server logs
at
org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:294)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:206)

*Logs:*

INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:07,987
CompactionManager.java:415 - Cleanup cannot run before a node has joined
the ring
INFO  [RMI TCP Connection(17)-x.x.x.x] 2017-05-05 04:04:08,010
CompactionManager.java:415 - Cleanup cannot run before a node has joined
the ring

All the nodes in the cluster are up and running. We tried doing a rolling
restart of all nodes and no luck.

After looking at the Cassandra JIRA :
https://issues.apache.org/jira/browse/CASSANDRA-10991 looks like the issue
is fixed with 2.2.6 and 3.0 version.
While we have plans to upgrade to the latest versions(which might take
longer time), does any know if there is any work around to mitigate the
issue?