Re: Unable to remove dead node from cluster.

A few other folks have reported issues with lingering dead nodes on large clusters - Jason Brown *just* gave an excellent gossip presentation at the summit regarding gossip optimizations for large clusters. Gossip is in the process of being refactored (here's at least one of the issues:

Re: Seeing null pointer exception 2.0.14 after purging gossip state

could it be related to CASSANDRA-9180 which was fixed in 2.0.15? although it really behaves like CASSANDRA-10231 which I don’t see any reference to it being in 2.0.x > On Sep 24,

Re: High read latency

if you run: nodetool cfhistograms On the given table and that will tell you how wide your rows are getting. At some point you can get wide enough rows that just the physics of retrieving them all take some time. > On Sep 25, 2015, at 9:21 AM, sai krishnam raju potturi

Re: memory usage problem of Metadata.tokenMap.tokenToHost

In practice there are not many good reasons to use that many keyspaces and tables. If the use case is multi tenancy then you’re almost always better off just using a combination of version tables and tenantId to give you flexibility as well as separation of client data. If you have that many

How to remove huge files with all expired data sooner?

Hi I have a table where I set TTL to only 7 days for all records and we keep pumping records in every day. In general, I would expect all data files for that table to have timestamps less than, say 8 or 9 days old, giving the system some time to work its magic. However, I see some files more

Re: How to tune Cassandra or Java Driver to get lower latency when there are a lot of writes?

Hi Ryan, As I said, saveToCassandra doesn't support "DELETE". This is why I modified the code of spark-cassandra-connector to allow me have DELETEs. What I change is how to bind a RDD row into a batch of CQL preparedStatements. On Fri, Sep 25, 2015 at 7:22 AM, Ryan Svihla

Re: High read latency

Please find histogram attached. On Fri, Sep 25, 2015 at 12:20 PM, Ryan Svihla wrote: > if everything is in ram there could be a number of issues unrelated to > Cassandra and there could be hardware limitations or contention problems. > Otherwise cell count can really deeply

Re: High read latency

I understand that but everything is in RAM (my data dir is tmpfs) and my row is not that wide approx. less than 5MB in size. So my question is if everything is in RAM then why does it take 43ms latency? On Fri, Sep 25, 2015 at 7:54 AM, Ryan Svihla wrote: > if you run: > >

Re: To batch or not to batch: A question for fast inserts

I think my main point is still, unlogged token aware batches are great, but if you’re writes are large enough, they may actually hurt rather than help, and likewise if your writes are too small, async only is likely only going to hurt. I’d say the average user I’ve had to help (with my

Re: High read latency

if everything is in ram there could be a number of issues unrelated to Cassandra and there could be hardware limitations or contention problems. Otherwise cell count can really deeply impact reads, all ram or not, and some of this is because of the nature of GC and some of it is the age of the

Re: To batch or not to batch: A question for fast inserts

Yep, my approach is definitely naive to hotspotting. If someone had that trouble, they could exhaust the iterator out of getReplicas() and distribute their writes more evenly (which might result in better statement distribution, but wouldn't change the workload on the cluster). In the end

Re: Unable to remove dead node from cluster.

The NPE throws when node tried to handleStateLeft, because it can not find the tokens associated with the node, can we just ignore the NPE and continue to remove the endpoint from the ring? On Fri, Sep 25, 2015 at 10:52 AM, Dikang Gu wrote: > @Jeff, yeah, I run the nodetool

Re: Unable to remove dead node from cluster.

@Jeff, yeah, I run the nodetool grep, and in my case, some nodes return "301", and some nodes return "300". And 300 is the correct number of nodes in my cluster. So it does look like an inconsistent issue, can you open a jira for this? Also, I'm looking for a quick fix/patch for this. On Fri,

Re: To batch or not to batch: A question for fast inserts

> compaction usually is the limiter for most clusters, so the difference between async versus unlogged batch ends up being minor or worse..non existent cause the hardware and data model combination result in compaction being the main throttle. If your number of records to load per second is

Running Cassandra on Java 8 u60..

Any issues with running Cassandra 2.0.16 on Java 8? I remember there is long term advice on not changing the GC but not the underlying version of Java. Thoughts? -- We’re hiring if you know of any awesome Java Devops or Linux Operations Engineers! Founder/CEO Spinn3r.com Location: *San

Re: memory usage problem of Metadata.tokenMap.tokenToHost

On Sun, Sep 20, 2015 at 9:22 AM, joseph gao wrote: >My application uses 2000+ keyspaces, and will dynamically create > keyspaces and tables. > While I agree with your observation (and think you should file a ticket at issues.apache.org and let the list know the URL)

Re: Running Cassandra on Java 8 u60..

I think those were referring to Java7 and G1GC (early versions were buggy). Cheers, Stefano On Fri, Sep 25, 2015 at 5:08 PM, Kevin Burton wrote: > Any issues with running Cassandra 2.0.16 on Java 8? I remember there is > long term advice on not changing the GC but not the

Re: Running Cassandra on Java 8 u60..

We saw no problems with 8u60. From: on behalf of Kevin Burton Reply-To: "user@cassandra.apache.org" Date: Friday, September 25, 2015 at 5:08 PM To: "user@cassandra.apache.org" Subject: Running Cassandra on Java 8 u60.. Any issues with running Cassandra 2.0.16 on

Re: Unable to remove dead node from cluster.

Apparently this was reported back in May: https://issues.apache.org/jira/browse/CASSANDRA-9510 - Jeff From: Dikang Gu Reply-To: "user@cassandra.apache.org" Date: Friday, September 25, 2015 at 11:31 AM To: cassandra Subject: Re: Unable to remove dead node from cluster. The NPE throws when

Re: High read latency

Jaydeep; since your primary key involves a clustering column, you may be having pretty wide rows. The read would be sequential. The latency could be acceptable, if the read were to involve really wide rows. If your primary key was like ((a,b)) without the clustering column, it's like reading a

Re: How to tune Cassandra or Java Driver to get lower latency when there are a lot of writes?

Why aren’t you using saveToCassandra (https://github.com/datastax/spark-cassandra-connector/blob/master/doc/5_saving.md )? They have a number of locality aware optimizations that will probably exceed your by

Re: To batch or not to batch: A question for fast inserts

Generally this is all correct but I cannot emphasize enough how much this “just depends” and today I generally move people to async inserts first before trying to micro-optimize some things to keep in mind. compaction usually is the limiter for most clusters, so the difference between async