Re: Error when running nodetool cleanup after adding a new node to a cluster

[Jerold Kinder]

While you folks are solving problems, I'm looking for expertise that sorely
needed at the Architect level that can consult and mentor.  Sorry for the
interruption but our need is getting desperate.  Thanks for your patience.

Regards

On Fri, Feb 10, 2017 at 2:14 AM, Srinath Reddy  wrote:

> The nodetool cleanup ran successfully after setting the CLASSPATH
> variable to the kubernetes-cassandra.jar.
>
> Thanks.
>
> On 09-Feb-2017, at 2:23 PM, Srinath Reddy  wrote:
>
> Alex,
>
> Thanks for reply.  I will try the workaround and post an update.
>
> Regards,
>
> Srinath Reddy
>
> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy  wrote:
>
>> Hi,
>>
>> Trying to re-balacne a Cassandra cluster after adding a new node and I'm
>> getting this error when running nodetool cleanup. The Cassandra cluster
>> is running in a Kubernetes cluster.
>>
>> Cassandra version is 2.2.8
>>
>> nodetool cleanup
>> error: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for
>> stacktrace.
>> -- StackTrace --
>> org.apache.cassandra.exceptions.ConfigurationException:
>> io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for
>> stacktrace.
>> at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(D
>> atabaseDescriptor.java:676)
>> at org.apache.cassandra.config.DatabaseDescriptor.(Data
>> baseDescriptor.java:119)
>> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
>> at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(No
>> deProbe.java:262)
>> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
>> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
>> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>>
>
> Hi,
>
> From the above stacktrace it looks like you're hitting the following TODO
> item:
>
> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e
> 9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282
>
> That is, nodetool needs to know concurrent_compactors setting's value
> before starting cleanup, but doesn't use JMX and tries to parse the
> configuration file instead.  That fails because your custom SeedProvider
> class is not on classpath for nodetool.
>
> A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be
> found by java when running nodetool script, see https://github.com/apache/
> cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108
>
> Proper fix: get rid of the TODO and really query the value using JMX,
> especially since the latest tick-tock release of Cassandra (3.10) added a
> way to modify it with JMX.
>
> --
> Alex
>
>
>
>

Re: Error when running nodetool cleanup after adding a new node to a cluster

[Srinath Reddy]

The nodetool cleanup ran successfully after setting the CLASSPATH variable to 
the kubernetes-cassandra.jar.

Thanks.

> On 09-Feb-2017, at 2:23 PM, Srinath Reddy  wrote:
> 
> Alex,
> 
> Thanks for reply.  I will try the workaround and post an update.
> 
> Regards,
> 
> Srinath Reddy
> 
>> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin > > wrote:
>> 
>> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy > > wrote:
>> Hi,
>> 
>> Trying to re-balacne a Cassandra cluster after adding a new node and I'm 
>> getting this error when running nodetool cleanup. The Cassandra cluster is 
>> running in a Kubernetes cluster.
>> 
>> Cassandra version is 2.2.8
>> 
>> nodetool cleanup
>> error: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for stacktrace.
>> -- StackTrace --
>> org.apache.cassandra.exceptions.ConfigurationException: 
>> io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for stacktrace.
>>  at 
>> org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
>>  at 
>> org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:119)
>>  at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
>>  at 
>> org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
>>  at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
>>  at 
>> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
>>  at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>> 
>> Hi,
>> 
>> From the above stacktrace it looks like you're hitting the following TODO 
>> item:
>> 
>> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282
>>  
>> 
>> 
>> That is, nodetool needs to know concurrent_compactors setting's value before 
>> starting cleanup, but doesn't use JMX and tries to parse the configuration 
>> file instead.  That fails because your custom SeedProvider class is not on 
>> classpath for nodetool.
>> 
>> A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be found 
>> by java when running nodetool script, see 
>> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108
>>  
>> 
>> 
>> Proper fix: get rid of the TODO and really query the value using JMX, 
>> especially since the latest tick-tock release of Cassandra (3.10) added a 
>> way to modify it with JMX.
>> 
>> --
>> Alex
> 



signature.asc
Description: Message signed with OpenPGP

Re: Error when running nodetool cleanup after adding a new node to a cluster

[Srinath Reddy]

Alex,

Thanks for reply.  I will try the workaround and post an update.

Regards,

Srinath Reddy

> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin  
> wrote:
> 
> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy  > wrote:
> Hi,
> 
> Trying to re-balacne a Cassandra cluster after adding a new node and I'm 
> getting this error when running nodetool cleanup. The Cassandra cluster is 
> running in a Kubernetes cluster.
> 
> Cassandra version is 2.2.8
> 
> nodetool cleanup
> error: io.k8s.cassandra.KubernetesSeedProvider
> Fatal configuration error; unable to start server.  See log for stacktrace.
> -- StackTrace --
> org.apache.cassandra.exceptions.ConfigurationException: 
> io.k8s.cassandra.KubernetesSeedProvider
> Fatal configuration error; unable to start server.  See log for stacktrace.
>   at 
> org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
>   at 
> org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:119)
>   at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
>   at 
> org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
>   at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
>   at 
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
>   at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
> 
> Hi,
> 
> From the above stacktrace it looks like you're hitting the following TODO 
> item:
> 
> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282
>  
> 
> 
> That is, nodetool needs to know concurrent_compactors setting's value before 
> starting cleanup, but doesn't use JMX and tries to parse the configuration 
> file instead.  That fails because your custom SeedProvider class is not on 
> classpath for nodetool.
> 
> A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be found 
> by java when running nodetool script, see 
> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108
>  
> 
> 
> Proper fix: get rid of the TODO and really query the value using JMX, 
> especially since the latest tick-tock release of Cassandra (3.10) added a way 
> to modify it with JMX.
> 
> --
> Alex



signature.asc
Description: Message signed with OpenPGP

Re: Error when running nodetool cleanup after adding a new node to a cluster

[Oleksandr Shulgin]

On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy  wrote:

> Hi,
>
> Trying to re-balacne a Cassandra cluster after adding a new node and I'm
> getting this error when running nodetool cleanup. The Cassandra cluster
> is running in a Kubernetes cluster.
>
> Cassandra version is 2.2.8
>
> nodetool cleanup
> error: io.k8s.cassandra.KubernetesSeedProvider
> Fatal configuration error; unable to start server.  See log for stacktrace.
> -- StackTrace --
> org.apache.cassandra.exceptions.ConfigurationException: io.k8s.cassandra.
> KubernetesSeedProvider
> Fatal configuration error; unable to start server.  See log for stacktrace.
> at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(
> DatabaseDescriptor.java:676)
> at org.apache.cassandra.config.DatabaseDescriptor.(
> DatabaseDescriptor.java:119)
> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
> at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(
> NodeProbe.java:262)
> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>

Hi,

>From the above stacktrace it looks like you're hitting the following TODO
item:

https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282

That is, nodetool needs to know concurrent_compactors setting's value
before starting cleanup, but doesn't use JMX and tries to parse the
configuration file instead.  That fails because your custom SeedProvider
class is not on classpath for nodetool.

A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be
found by java when running nodetool script, see
https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108

Proper fix: get rid of the TODO and really query the value using JMX,
especially since the latest tick-tock release of Cassandra (3.10) added a
way to modify it with JMX.

--
Alex

Re: Error when running nodetool cleanup after adding a new node to a cluster

[Srinath Reddy]

Yes, I ran the nodetool cleanup on the other nodes and got the error.

Thanks.

> On 09-Feb-2017, at 11:12 AM, Harikrishnan Pillai  
> wrote:
> 
> The cleanup has to run on other nodes
> 
> Sent from my iPhone
> 
> On Feb 8, 2017, at 9:14 PM, Srinath Reddy  > wrote:
> 
>> Hi,
>> 
>> Trying to re-balacne a Cassandra cluster after adding a new node and I'm 
>> getting this error when running nodetool cleanup. The Cassandra cluster is 
>> running in a Kubernetes cluster.
>> 
>> Cassandra version is 2.2.8
>> 
>> nodetool cleanup
>> error: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for stacktrace.
>> -- StackTrace --
>> org.apache.cassandra.exceptions.ConfigurationException: 
>> io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for stacktrace.
>> at 
>> org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
>> at 
>> org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:119)
>> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
>> at 
>> org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
>> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
>> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
>> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>> 
>> nodetool status
>> Datacenter: datacenter1
>> ===
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> --  Address  Load   Tokens   Owns (effective)  Host ID   
>> Rack
>> UN  10.244.3.4   6.91 GB256  60.8% 
>> bad1c6c6-8c2e-4f0c-9aea-0d63b451e7a1  rack1
>> UN  10.244.0.3   6.22 GB256  60.2% 
>> 936cb0c0-d14f-4ddd-bfde-3865b922e267  rack1
>> UN  10.244.1.3   6.12 GB256  59.4% 
>> 0cb43711-b155-449c-83ba-00ed2a97affe  rack1
>> UN  10.244.4.3   632.43 MB  256  57.8% 
>> 55095c75-26df-4180-9004-9fabf88faacc  rack1
>> UN  10.244.2.10  6.08 GB256  61.8% 
>> 32e32bd2-364f-4b6f-b13a-8814164ed160  rack1
>> 
>> 
>> Any suggestions on what is needed to re-balance the cluster after adding the 
>> new node? I have run nodetool repair but not able to run nodetool cleanup.
>> 
>> Thanks.
>> 
>> 



signature.asc
Description: Message signed with OpenPGP

Re: Error when running nodetool cleanup after adding a new node to a cluster

[Harikrishnan Pillai]

The cleanup has to run on other nodes

Sent from my iPhone

On Feb 8, 2017, at 9:14 PM, Srinath Reddy 
mailto:ksre...@gmail.com>> wrote:

Hi,

Trying to re-balacne a Cassandra cluster after adding a new node and I'm 
getting this error when running nodetool cleanup. The Cassandra cluster is 
running in a Kubernetes cluster.

Cassandra version is 2.2.8

nodetool cleanup
error: io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
-- StackTrace --
org.apache.cassandra.exceptions.ConfigurationException: 
io.k8s.cassandra.KubernetesSeedProvider
Fatal configuration error; unable to start server.  See log for stacktrace.
at 
org.apache.cassandra.config.DatabaseDescriptor.applyConfig(DatabaseDescriptor.java:676)
at 
org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:119)
at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(NodeProbe.java:262)
at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)

nodetool status
Datacenter: datacenter1
===
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address  Load   Tokens   Owns (effective)  Host ID  
 Rack
UN  10.244.3.4   6.91 GB256  60.8% 
bad1c6c6-8c2e-4f0c-9aea-0d63b451e7a1  rack1
UN  10.244.0.3   6.22 GB256  60.2% 
936cb0c0-d14f-4ddd-bfde-3865b922e267  rack1
UN  10.244.1.3   6.12 GB256  59.4% 
0cb43711-b155-449c-83ba-00ed2a97affe  rack1
UN  10.244.4.3   632.43 MB  256  57.8% 
55095c75-26df-4180-9004-9fabf88faacc  rack1
UN  10.244.2.10  6.08 GB256  61.8% 
32e32bd2-364f-4b6f-b13a-8814164ed160  rack1


Any suggestions on what is needed to re-balance the cluster after adding the 
new node? I have run nodetool repair but not able to run nodetool cleanup.

Thanks.