Re: Restarting nodes and reported load

Am I the only one thinking 3TB is way too much data for a single node on a VM? On Tue, May 30, 2017 at 10:36 PM, Daniel Steuernol wrote: > I don't believe incremental repair is enabled, I have never enabled it on > the cluster, and unless it's the default then it is off.

Re: Restarting nodes and reported load

My question is about cassandra, ultimately I'm trying to figure out why our clusters performance degrades approximately every 6 days. I noticed that the load as reported by nodetool status was very high, but that might be unrelated to the problem. A restart solves the performance problem.I've

Re: [Cassandra] Ignoring interval time

The debug output is from the failure detector in the gossip module. Code can be found here https://github.com/apache/cassandra/blob/8b3a60b9a7dbefeecc06bace617279612ec7092d/src/java/org/apache/cassandra/gms/FailureDetector.java#L450-L474 . The debug logging above is reporting around an

Re: Restarting nodes and reported load

Unless there is a bug, snapshots are excluded (they are not HDFS anyway!) from nodetool status. Out of curiousity, is incremenatal repair enabled? This is almost certainly a rat hole, but there was an issue a few releases back where load would only increase until the node was restarted. Had been

Re: Restarting nodes and reported load

Did you notice that HDFS is the distributed file system used? *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* *“All men dream, but not equally. Those who dream by night in the dusty recesses of their minds wake up in the day to find it was vanity, but the dreamers

Re: How to know when repair repaired something?

On 30 May 2017, at 21:11, Varun Gupta wrote: I am missing the point, why do you want to re-trigger the process post repair. Repair will sync the data correctly. Sorry - I mis-represented that. I want to trigger something else, not repair. I am investigating a CQRS/Event Sourced pattern

Re: Restarting nodes and reported load

No degradation. *Daemeon C.M. ReiydelleUSA (+1) 415.501.0198London (+44) (0) 20 8144 9872* *“All men dream, but not equally. Those who dream by night in the dusty recesses of their minds wake up in the day to find it was vanity, but the dreamers of the day are dangerous men, for they may

Netty SSL memory leak

I have Cassandra 3.0.9 cluster that is hitting OutOfMemoryErrors with byte buffer allocation. The stack trace looks like: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:694) ~[na:1.8.0_131] at java.nio.DirectByteBuffer.(DirectByteBuffer.java:123)

Re: Restarting nodes and reported load

This isn't an HDFS mailing list. On Tue, May 30, 2017 at 2:14 PM daemeon reiydelle wrote: > no, 3tb is small. 30-50tb of hdfs space is typical these days per hdfs > node. Depends somewhat on whether there is a mix of more and less > frequently accessed data. But even storing

Re: Restarting nodes and reported load

Daniel - my comment wasn't to you, it was in response to Daemeon. > no, 3tb is small. 30-50tb of hdfs space is typical these days per hdfs node Jon On Tue, May 30, 2017 at 2:30 PM Daniel Steuernol wrote: > My question is about cassandra, ultimately I'm trying to figure

Re: Restarting nodes and reported load

I don't believe incremental repair is enabled, I have never enabled it on the cluster, and unless it's the default then it is off. Also I don't see a setting in cassandra.yaml for it. On May 30 2017, at 1:10 pm, daemeon reiydelle wrote:

Re: Restarting nodes and reported load

OK, thanks. So there was a bug in a prior version of C*, symptoms were: Nodetool would show increasing load utilization over time. Stopping and restarting C* nodes would reset the storage back to what one would expect on that node, for a while, then it would creep upwards again, until the

Re: Restarting nodes and reported load

incremental backup is set to false in the config file, also I have set  snapshot_before_compaction and auto_snapshot to false as well. I ran nodetool clearsnapshot, but before doing that I ran nodetool listsnapshots and it listed a bunch of snapshots. I would have expected that to be empty because

Re: Restarting nodes and reported load

That does sound like what's happening, did performance degrade as the reported load increased? On May 30 2017, at 1:52 pm, daemeon reiydelle wrote: OK, thanks.So there was a bug in a prior version of

Re: Restarting nodes and reported load

You're the only one I see in the thread that's made any reference to HDFS. The OP even noted that his question is about C*, not HDFS. On Tue, May 30, 2017 at 2:59 PM daemeon reiydelle wrote: > Did you notice that HDFS is the distributed file system used? > > > > > > *Daemeon

[Cassandra] Ignoring interval time

Hi All, Please let me know why this debug log is coming: DEBUG [GossipStage:1] 2017-05-30 15:01:31,496 FailureDetector.java:456 - Ignoring interval time of 2000686406 for /XXX.XX.XXX.204 DEBUG [GossipStage:1] 2017-05-30 15:01:34,497 FailureDetector.java:456 - Ignoring interval time of

org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey

Hi All, I am getting below exception in debug.log. DEBUG [ReadRepairStage:636754] 2017-05-30 14:49:44,259 ReadCallback.java:234 - Digest mismatch: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey(4329955402556695061,

Is DataStax's DSE better than cassandra's free open source for a newbie developer's good start for cassandra?

Dear sir, Good evening, this is Georgelin from the biggest market of ShangHai, China, I have known how to download an odd-number(bug fixed) version cassandra source but not a rpm package. would you like to give me a step by step guiding from compiling, distributing compiled classes to

Re: org.apache.cassandra.service.DigestMismatchException: Mismatch for key DecoratedKey

This blog post (http://thelastpickle.com/blog/2011/05/15/Deletes-and-Tombstones.html ) provides good explenation on the exception in your debug log. Regards, Akhil > On 30/05/2017, at 9:29 PM, Abhishek Kumar Maheshwari >

Re: Is DataStax's DSE better than cassandra's free open source for a newbie developer's good start for cassandra?

Hello, DSE is commercial and costs money to use in production. More info from DataStax: http://www.datastax.com/products/subscriptions RPMs are currently not available for the latest version. There is 3.0.13 but newer than that are not available

Re: Restarting nodes and reported load

When you say "the load rises ... ", could you clarify what you mean by "load"? That has a specific Linux term, and in e.g. Cloudera Manager. But in neither case would that be relevant to transient or persisted disk. Am I missing something? On Tue, May 30, 2017 at 10:18 AM, tommaso barbugli

Re: Restarting nodes and reported load

3-4 TB per node or in total? On Tue, May 30, 2017 at 6:48 PM, Daniel Steuernol wrote: > I should also mention that I am running cassandra 3.10 on the cluster > > > > On May 29 2017, at 9:43 am, Daniel Steuernol > wrote: > >> The cluster is running

Re: Restarting nodes and reported load

It's 3-4TB per node, and by load rises, I'm talking about load as reported by nodetool status. On May 30 2017, at 10:25 am, daemeon reiydelle wrote: When you say "the load rises ... ", could you

Re: Restarting nodes and reported load

I should also mention that I am running cassandra 3.10 on the cluster On May 29 2017, at 9:43 am, Daniel Steuernol wrote: The cluster is running with RF=3, right now each node is storing about 3-4

Re: How to know when repair repaired something?

I am missing the point, why do you want to re-trigger the process post repair. Repair will sync the data correctly. On Mon, May 29, 2017 at 8:07 AM, Jan Algermissen wrote: > Hi, > > is it possible to extract from repair logs the writetime of the writes > that needed

Re: Restarting nodes and reported load

Can you please check if you have incremental backup enabled and snapshots are occupying the space. run nodetool clearsnapshot command. On Tue, May 30, 2017 at 11:12 AM, Daniel Steuernol wrote: > It's 3-4TB per node, and by load rises, I'm talking about load as reported >

Re: [Cassandra] Ignoring interval time

Can you please check Cassandra Stats, if cluster is under too much load. This is the symptom, not the root cause. On Tue, May 30, 2017 at 2:33 AM, Abhishek Kumar Maheshwari < abhishek.maheshw...@timesinternet.in> wrote: > Hi All, > > > > Please let me know why this debug log is coming: > > > >

Re: Restarting nodes and reported load

no, 3tb is small. 30-50tb of hdfs space is typical these days per hdfs node. Depends somewhat on whether there is a mix of more and less frequently accessed data. But even storing only hot data, never saw anything less than 20tb hdfs per node. *Daemeon C.M. ReiydelleUSA (+1)