Re: Error when running nodetool cleanup after adding a new node to a cluster

2017-02-13 Thread Jerold Kinder 
While you folks are solving problems, I'm looking for expertise that sorely
needed at the Architect level that can consult and mentor.  Sorry for the
interruption but our need is getting desperate.  Thanks for your patience.

Regards

On Fri, Feb 10, 2017 at 2:14 AM, Srinath Reddy  wrote:

> The nodetool cleanup ran successfully after setting the CLASSPATH
> variable to the kubernetes-cassandra.jar.
>
> Thanks.
>
> On 09-Feb-2017, at 2:23 PM, Srinath Reddy  wrote:
>
> Alex,
>
> Thanks for reply.  I will try the workaround and post an update.
>
> Regards,
>
> Srinath Reddy
>
> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin <
> oleksandr.shul...@zalando.de> wrote:
>
> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy  wrote:
>
>> Hi,
>>
>> Trying to re-balacne a Cassandra cluster after adding a new node and I'm
>> getting this error when running nodetool cleanup. The Cassandra cluster
>> is running in a Kubernetes cluster.
>>
>> Cassandra version is 2.2.8
>>
>> nodetool cleanup
>> error: io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for
>> stacktrace.
>> -- StackTrace --
>> org.apache.cassandra.exceptions.ConfigurationException:
>> io.k8s.cassandra.KubernetesSeedProvider
>> Fatal configuration error; unable to start server.  See log for
>> stacktrace.
>> at org.apache.cassandra.config.DatabaseDescriptor.applyConfig(D
>> atabaseDescriptor.java:676)
>> at org.apache.cassandra.config.DatabaseDescriptor.(Data
>> baseDescriptor.java:119)
>> at org.apache.cassandra.tools.NodeProbe.checkJobs(NodeProbe.java:256)
>> at org.apache.cassandra.tools.NodeProbe.forceKeyspaceCleanup(No
>> deProbe.java:262)
>> at org.apache.cassandra.tools.nodetool.Cleanup.execute(Cleanup.java:55)
>> at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:244)
>> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:158)
>>
>
> Hi,
>
> From the above stacktrace it looks like you're hitting the following TODO
> item:
>
> https://github.com/apache/cassandra/blob/98d74ed998706e9e047dc0f7886a1e
> 9b18df3ce9/src/java/org/apache/cassandra/tools/NodeProbe.java#L282
>
> That is, nodetool needs to know concurrent_compactors setting's value
> before starting cleanup, but doesn't use JMX and tries to parse the
> configuration file instead.  That fails because your custom SeedProvider
> class is not on classpath for nodetool.
>
> A workaround: make sure io.k8s.cassandra.KubernetesSeedProvider can be
> found by java when running nodetool script, see https://github.com/apache/
> cassandra/blob/98d74ed998706e9e047dc0f7886a1e9b18df3ce9/bin/nodetool#L108
>
> Proper fix: get rid of the TODO and really query the value using JMX,
> especially since the latest tick-tock release of Cassandra (3.10) added a
> way to modify it with JMX.
>
> --
> Alex
>
>
>
>

Re: compaction falling behind

2017-02-13 Thread Harikrishnan Pillai 
If your compaction strategy is Leveled the number of sstables in each level is 
a good indication that compactions  are keeping up.


From: Ben Bromhead 
Sent: Monday, February 13, 2017 1:49:05 PM
To: user
Subject: Re: compaction falling behind


You can do so in two ways:

1) direct observation:
You can keep an eye on the number of pending compactions. This will fluctuate 
with load, compaction strategy, ongoing repairs and nodes bootstrapping but 
generally the pattern is it should trend towards 0.

There have been a number of bugs in past versions of Cassandra whereby the 
number of pending compactions is not reported correctly, so depending on what 
version of Cassandra you run this could impact you.

2) Indirect observation
You can keep an eye on metrics that healthy compaction will directly contribute 
to. These include the number of sstables per read histogram, estimated 
droppable tombstones, tombstones per read etc. You should keep an eye on these 
things anyway as they can often show you areas where you can fine tune 
compaction or your data model.

Everything exposed by nodetool is consumable via JMX which is great to plug 
into your metrics/monitoring/observability system :)

On Mon, 13 Feb 2017 at 13:23 John Sanda 
> wrote:
What is a good way to determine whether or not compaction is falling behind? I 
read a couple things earlier that suggest nodetool compactionstats might not be 
the most reliable thing to use.



- John
--
Ben Bromhead
CTO | Instaclustr
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: compaction falling behind

2017-02-13 Thread Ben Bromhead 
You can do so in two ways:

1) direct observation:
You can keep an eye on the number of pending compactions. This will
fluctuate with load, compaction strategy, ongoing repairs and nodes
bootstrapping but generally the pattern is it should trend towards 0.

There have been a number of bugs in past versions of Cassandra whereby the
number of pending compactions is not reported correctly, so depending on
what version of Cassandra you run this could impact you.

2) Indirect observation
You can keep an eye on metrics that healthy compaction will directly
contribute to. These include the number of sstables per read histogram,
estimated droppable tombstones, tombstones per read etc. You should keep an
eye on these things anyway as they can often show you areas where you can
fine tune compaction or your data model.

Everything exposed by nodetool is consumable via JMX which is great to plug
into your metrics/monitoring/observability system :)

On Mon, 13 Feb 2017 at 13:23 John Sanda  wrote:

> What is a good way to determine whether or not compaction is falling
> behind? I read a couple things earlier that suggest nodetool
> compactionstats might not be the most reliable thing to use.
>
>
>
> - John
>
-- 
Ben Bromhead
CTO | Instaclustr 
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

compaction falling behind

2017-02-13 Thread John Sanda 
What is a good way to determine whether or not compaction is falling
behind? I read a couple things earlier that suggest nodetool
compactionstats might not be the most reliable thing to use.


- John

Re: Why does Cassandra recommends Oracle JVM instead of OpenJDK?

2017-02-13 Thread Edward Capriolo 
On Monday, February 13, 2017, Brice Dutheil  wrote:

> The Android battle is another thing that I wouldn't consider for OracleJDK
> / OpenJDK.
> While I do like what Google did from a technical point of view, Google may
> have overstepped fair use (or not – I don't know). Anyway Sun didn't like
> what Google did, they probably considered going to court at that time.
>
>
>
>
> -- Brice
>
> On Mon, Feb 13, 2017 at 10:20 AM, kurt greaves  > wrote:
>
>> are people actually trying to imply that Google is less evil than oracle?
>> what is this shill fest
>>
>>
>> On 12 Feb. 2017 8:24 am, "Kant Kodali" > > wrote:
>>
>> Saw this one today...
>>
>> https://news.ycombinator.com/item?id=13624062
>>
>> On Tue, Jan 3, 2017 at 6:27 AM, Eric Evans > > wrote:
>>
>>> On Mon, Jan 2, 2017 at 2:26 PM, Edward Capriolo >> > wrote:
>>> > Lets be clear:
>>> > What I am saying is avoiding being loose with the word "free"
>>> >
>>> > https://en.wikipedia.org/wiki/Free_software_license
>>> >
>>> > Many things with the JVM are free too. Most importantly it is free to
>>> use.
>>> >
>>> > https://www.java.com/en/download/faq/distribution.xml
>>> >
>>> > As it relates to this conversation: I am not aware of anyone running
>>> > Cassandra that has modified upstream JVM to make Cassandra run
>>> > better/differently *. Thus the license around the Oracle JVM is roughly
>>> > meaningless to the user/developer of cassandra.
>>> >
>>> > * The only group I know that took an action to modify upstream was
>>> Acunu.
>>> > They had released a modified Linux Kernel with a modified Apache
>>> Cassandra.
>>> > http://cloudtweaks.com/2011/02/data-storage-startup-acunu-ra
>>> ises-3-6-million-to-launch-its-first-product/.
>>> > That product no longer exists.
>>> >
>>> > "I don't how to read any of this.  It sounds like you're saying that a
>>> > JVM is something that cannot be produced as a Free Software project,"
>>> >
>>> > What I am saying is something like the JVM "could" be produced as a
>>> "free
>>> > software project". However, the argument that I was making is that the
>>> > popular viable languages/(including vms or runtime to use them) today
>>> > including Java, C#, Go, Swift are developed by the largest tech
>>> companies in
>>> > the world, and as such I do believe a platform would be viable.
>>> Specifically
>>> > I believe without Oracle driving Java OpenJDK would not be viable.
>>> >
>>> > There are two specific reasons.
>>> > 1) I do not see large costly multi-year initiatives like G1 happening
>>> > 2) Without guidance/leadership that sun/oracle I do not see new
>>> features
>>> > that change the language like lambda's and try multi-catch happening
>>> in a
>>> > sane way.
>>> >
>>> > I expanded upon #2 be discussing my experience with standards like c++
>>> 11,
>>> > 14,17 and attempting to take compiling working lambda code on linux
>>> GCC to
>>> > microsoft visual studio and having it not compile. In my opinion, Java
>>> only
>>> > wins because as a platform it is very portable as both source and
>>> binary
>>> > code. Without leadership on that front I believe that over time the
>>> language
>>> > would suffer.
>>>
>>> I realize that you're trying to be pragmatic about all of this, but
>>> what I don't think you realize, is that so am I.
>>>
>>> Java could change hands at any time (it has once already), or Oracle
>>> leadership could decide to go in a different direction.  Imagine for
>>> example that they relicensed it to exclude use by orientation or
>>> religion, Cassandra would implicitly carry these restrictions as well.
>>> Imagine that they decided to provide a back-door to the NSA, Cassandra
>>> would then also contain such a back-door.  These might sound
>>> hypothetical, but there is plenty of precedent here.
>>>
>>> OpenJDK benefits from the same resources and leadership from Oracle
>>> that you value, but is licensed and distributed in a way that
>>> safeguards us from a day when Oracle becomes less benevolent, (if that
>>> were to happen, some other giant company could assume the mantle of
>>> leadership).
>>>
>>> All I'm really suggesting is that we at least soften our requirement
>>> on the Oracle JVM, and perhaps perform some test runs in CI against
>>> OpenJDK.  Actively discouraging people from using the Free Software
>>> alternative here, one that is working well for many, isn't the
>>> behavior I'd normally expect from a Free Software project.
>>>
>>> --
>>> Eric Evans
>>> john.eric.ev...@gmail.com
>>> 
>>>
>>
>>
>>
>

Re: UnknownColumnFamilyException after removing all Cassandra data

2017-02-13 Thread Jacob Shadix 
The node will not bootstrap if it is listed as a seed node.

-- Jacob Shadix

On Tue, Feb 7, 2017 at 12:16 PM, Simone Franzini 
wrote:

> To further add to my previous answer, the node in question is a seed node,
> so it did not bootstrap.
> Should I remove it from the list of seed nodes and then try to restart it?
>
> Simone Franzini, PhD
>
> http://www.linkedin.com/in/simonefranzini
>
> On Tue, Feb 7, 2017 at 9:43 AM, Simone Franzini 
> wrote:
>
>> This is exactly what I did on the second node. If this is not the correct
>> / best procedure to adopt in these cases, please advise:
>>
>> 1. Removed all the data, including the system table (rm -rf data/
>> commitlog/ saved_caches).
>> 2. Configured the node to replace itself, by adding the following line to
>> cassandra-env.sh: JVM_OPTS="$JVM_OPTS -Dcassandra.replace_address=> own IP address>"
>> 3. Start the node.
>>
>> Noticeably, I did not do nodetool decommission or removenode. Is that the
>> recommended approach?
>>
>> Given what I did, I am mystified as to what the problem is. If I query
>> the system.schema_columnfamilies on the affected node, all CF IDs are
>> there. Same goes for the only other node that is currently up. Also, the
>> other node that is currently up has data for all those CF IDs in the data
>> folder.
>>
>>
>> Simone Franzini, PhD
>>
>> http://www.linkedin.com/in/simonefranzini
>>
>> On Tue, Feb 7, 2017 at 5:39 AM, kurt greaves 
>> wrote:
>>
>>> The node is trying to communicate with another node, potentially
>>> streaming data, and is receiving files/data for an "unknown column family".
>>> That is, it doesn't know about the CF with the id
>>> e36415b6-95a7-368c-9ac0-ae0ac774863d.
>>> If you deleted some columnfamilies but not all the system keyspace and
>>> restarted the node I'd expect this error to occur. Or I suppose if you
>>> didn't decommission the node properly before blowing the data away and
>>> restarting.
>>>
>>> You'll have to give us more information on what your exact steps were on
>>> this 2nd node:
>>>
>>> When you say deleted all Cassandra data, did this include the system
>>> tables? Were your steps to delete all the data and then just restart the
>>> node? Did you remove the node from the cluster prior to deleting the data
>>> and restarting it (nodetool decommission/removenode? Did the node rejoin
>>> the cluster or did it have to bootstrap?
>>>
>>>
>>>
>>
>

Re: Why does Cassandra recommends Oracle JVM instead of OpenJDK?

2017-02-13 Thread Brice Dutheil 
The Android battle is another thing that I wouldn't consider for OracleJDK
/ OpenJDK.
While I do like what Google did from a technical point of view, Google may
have overstepped fair use (or not – I don't know). Anyway Sun didn't like
what Google did, they probably considered going to court at that time.




-- Brice

On Mon, Feb 13, 2017 at 10:20 AM, kurt greaves  wrote:

> are people actually trying to imply that Google is less evil than oracle?
> what is this shill fest
>
>
> On 12 Feb. 2017 8:24 am, "Kant Kodali"  wrote:
>
> Saw this one today...
>
> https://news.ycombinator.com/item?id=13624062
>
> On Tue, Jan 3, 2017 at 6:27 AM, Eric Evans 
> wrote:
>
>> On Mon, Jan 2, 2017 at 2:26 PM, Edward Capriolo 
>> wrote:
>> > Lets be clear:
>> > What I am saying is avoiding being loose with the word "free"
>> >
>> > https://en.wikipedia.org/wiki/Free_software_license
>> >
>> > Many things with the JVM are free too. Most importantly it is free to
>> use.
>> >
>> > https://www.java.com/en/download/faq/distribution.xml
>> >
>> > As it relates to this conversation: I am not aware of anyone running
>> > Cassandra that has modified upstream JVM to make Cassandra run
>> > better/differently *. Thus the license around the Oracle JVM is roughly
>> > meaningless to the user/developer of cassandra.
>> >
>> > * The only group I know that took an action to modify upstream was
>> Acunu.
>> > They had released a modified Linux Kernel with a modified Apache
>> Cassandra.
>> > http://cloudtweaks.com/2011/02/data-storage-startup-acunu-ra
>> ises-3-6-million-to-launch-its-first-product/.
>> > That product no longer exists.
>> >
>> > "I don't how to read any of this.  It sounds like you're saying that a
>> > JVM is something that cannot be produced as a Free Software project,"
>> >
>> > What I am saying is something like the JVM "could" be produced as a
>> "free
>> > software project". However, the argument that I was making is that the
>> > popular viable languages/(including vms or runtime to use them) today
>> > including Java, C#, Go, Swift are developed by the largest tech
>> companies in
>> > the world, and as such I do believe a platform would be viable.
>> Specifically
>> > I believe without Oracle driving Java OpenJDK would not be viable.
>> >
>> > There are two specific reasons.
>> > 1) I do not see large costly multi-year initiatives like G1 happening
>> > 2) Without guidance/leadership that sun/oracle I do not see new features
>> > that change the language like lambda's and try multi-catch happening in
>> a
>> > sane way.
>> >
>> > I expanded upon #2 be discussing my experience with standards like c++
>> 11,
>> > 14,17 and attempting to take compiling working lambda code on linux GCC
>> to
>> > microsoft visual studio and having it not compile. In my opinion, Java
>> only
>> > wins because as a platform it is very portable as both source and binary
>> > code. Without leadership on that front I believe that over time the
>> language
>> > would suffer.
>>
>> I realize that you're trying to be pragmatic about all of this, but
>> what I don't think you realize, is that so am I.
>>
>> Java could change hands at any time (it has once already), or Oracle
>> leadership could decide to go in a different direction.  Imagine for
>> example that they relicensed it to exclude use by orientation or
>> religion, Cassandra would implicitly carry these restrictions as well.
>> Imagine that they decided to provide a back-door to the NSA, Cassandra
>> would then also contain such a back-door.  These might sound
>> hypothetical, but there is plenty of precedent here.
>>
>> OpenJDK benefits from the same resources and leadership from Oracle
>> that you value, but is licensed and distributed in a way that
>> safeguards us from a day when Oracle becomes less benevolent, (if that
>> were to happen, some other giant company could assume the mantle of
>> leadership).
>>
>> All I'm really suggesting is that we at least soften our requirement
>> on the Oracle JVM, and perhaps perform some test runs in CI against
>> OpenJDK.  Actively discouraging people from using the Free Software
>> alternative here, one that is working well for many, isn't the
>> behavior I'd normally expect from a Free Software project.
>>
>> --
>> Eric Evans
>> john.eric.ev...@gmail.com
>>
>
>
>

Re: Why does Cassandra recommends Oracle JVM instead of OpenJDK?

2017-02-13 Thread kurt greaves 
are people actually trying to imply that Google is less evil than oracle?
what is this shill fest

On 12 Feb. 2017 8:24 am, "Kant Kodali"  wrote:

Saw this one today...

https://news.ycombinator.com/item?id=13624062

On Tue, Jan 3, 2017 at 6:27 AM, Eric Evans 
wrote:

> On Mon, Jan 2, 2017 at 2:26 PM, Edward Capriolo 
> wrote:
> > Lets be clear:
> > What I am saying is avoiding being loose with the word "free"
> >
> > https://en.wikipedia.org/wiki/Free_software_license
> >
> > Many things with the JVM are free too. Most importantly it is free to
> use.
> >
> > https://www.java.com/en/download/faq/distribution.xml
> >
> > As it relates to this conversation: I am not aware of anyone running
> > Cassandra that has modified upstream JVM to make Cassandra run
> > better/differently *. Thus the license around the Oracle JVM is roughly
> > meaningless to the user/developer of cassandra.
> >
> > * The only group I know that took an action to modify upstream was Acunu.
> > They had released a modified Linux Kernel with a modified Apache
> Cassandra.
> > http://cloudtweaks.com/2011/02/data-storage-startup-acunu-ra
> ises-3-6-million-to-launch-its-first-product/.
> > That product no longer exists.
> >
> > "I don't how to read any of this.  It sounds like you're saying that a
> > JVM is something that cannot be produced as a Free Software project,"
> >
> > What I am saying is something like the JVM "could" be produced as a "free
> > software project". However, the argument that I was making is that the
> > popular viable languages/(including vms or runtime to use them) today
> > including Java, C#, Go, Swift are developed by the largest tech
> companies in
> > the world, and as such I do believe a platform would be viable.
> Specifically
> > I believe without Oracle driving Java OpenJDK would not be viable.
> >
> > There are two specific reasons.
> > 1) I do not see large costly multi-year initiatives like G1 happening
> > 2) Without guidance/leadership that sun/oracle I do not see new features
> > that change the language like lambda's and try multi-catch happening in a
> > sane way.
> >
> > I expanded upon #2 be discussing my experience with standards like c++
> 11,
> > 14,17 and attempting to take compiling working lambda code on linux GCC
> to
> > microsoft visual studio and having it not compile. In my opinion, Java
> only
> > wins because as a platform it is very portable as both source and binary
> > code. Without leadership on that front I believe that over time the
> language
> > would suffer.
>
> I realize that you're trying to be pragmatic about all of this, but
> what I don't think you realize, is that so am I.
>
> Java could change hands at any time (it has once already), or Oracle
> leadership could decide to go in a different direction.  Imagine for
> example that they relicensed it to exclude use by orientation or
> religion, Cassandra would implicitly carry these restrictions as well.
> Imagine that they decided to provide a back-door to the NSA, Cassandra
> would then also contain such a back-door.  These might sound
> hypothetical, but there is plenty of precedent here.
>
> OpenJDK benefits from the same resources and leadership from Oracle
> that you value, but is licensed and distributed in a way that
> safeguards us from a day when Oracle becomes less benevolent, (if that
> were to happen, some other giant company could assume the mantle of
> leadership).
>
> All I'm really suggesting is that we at least soften our requirement
> on the Oracle JVM, and perhaps perform some test runs in CI against
> OpenJDK.  Actively discouraging people from using the Free Software
> alternative here, one that is working well for many, isn't the
> behavior I'd normally expect from a Free Software project.
>
> --
> Eric Evans
> john.eric.ev...@gmail.com
>