While you folks are solving problems, I'm looking for expertise that sorely needed at the Architect level that can consult and mentor. Sorry for the interruption but our need is getting desperate. Thanks for your patience.
Regards On Fri, Feb 10, 2017 at 2:14 AM, Srinath Reddy <ksre...@gmail.com> wrote: > The nodetool cleanup ran successfully after setting the CLASSPATH > variable to the kubernetes-cassandra.jar. > > Thanks. > > On 09-Feb-2017, at 2:23 PM, Srinath Reddy <ksre...@gmail.com> wrote: > > Alex, > > Thanks for reply. I will try the workaround and post an update. > > Regards, > > Srinath Reddy > > On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin < > oleksandr.shul...@zalando.de> wrote: > > On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy <ksre...@gmail.com> wrote: > >> Hi, >> >> Trying to re-balacne a Cassandra cluster after adding a new node and I'm >> getting this error when running nodetool cleanup. The Cassandra cluster >> is running in a Kubernetes cluster. >> >> Cassandra version is 2.2.8 >> >> nodetool cleanup >> error: io.k8s.cassandra.KubernetesSeedProvider >> Fatal configuration error; unable to start server. See log for >> stacktrace. >> -- StackTrace -- >> org.apache.cassandra.exceptions.ConfigurationException: >> io.k8s.cassandra.KubernetesSeedProvider >> Fatal configuration error; unable to start server. See log for >> stacktrace. >> at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(D >> atabaseDescriptor.java:676) >> at org.apache.cassandra.config.DatabaseDescriptor.<clinit>(Data >> baseDescriptor.java:119) >> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256) >> at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(No >> deProbe.java:262) >> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55) >> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244) >> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158) >> > > Hi, > > From the above stacktrace it looks like you're hitting the following TODO > item: > > https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e > 9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282 > > That is, nodetool needs to know concurrent_compactors setting's value > before starting cleanup, but doesn't use JMX and tries to parse the > configuration file instead. That fails because your custom SeedProvider > class is not on classpath for nodetool. > > A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be > found by java when running nodetool script, see https://github.com/apache/ > cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108 > > Proper fix: get rid of the TODO and really query the value using JMX, > especially since the latest tick-tock release of Cassandra (3.10) added a > way to modify it with JMX. > > -- > Alex > > > >
