Re: When are hints written?

HI Bo; you raised 2 questions: 20% system utilization Hints 20% system utilization: For a node or a cluster to have 20% utilization is Normal during peak write operation. Hints: hints are written when a node is unreachable;C* 3.0 has a complete over haul in the way hints have

RE: Problem Replacing a Dead Node

Mir; You can take a node out of the cluster with nodetool decommission to a live node, or nodetool removetoken (to any other machine) to remove a dead one. This will assign the ranges the old node was responsible for to other nodes, and replicate the appropriate data there. If decommission is

Re: Combining two clusters/keyspaces into single cluster

HI ; Your objective is add the Keyspace 2 to cluster 1. The documentation link being referred to is to add a new datacenter [not applicable to you]. You need to : a. take a snapshot of keyspace 2 on cluster2 b. use sstable loader to copy the keyspace2 onto cluster 1 c. run a

Re: Problem Replacing a Dead Node

I will try a rolling restart to see whether that helps. The replacement node is pingable from other cassandra nodes. I also was able to telnet to the storage port (7000) of the replacement node as well from another node. cqlsh doesn't work on the new node. When does gossip settle? Is there anyway

RE: Problem Replacing a Dead Node

Reusing the bootstrapping node could have caused this, but hard to tell. Since you have only 7 nodes, have you tried doing a few rolling restarts of all nodes to let gossip settle ? Also, the node is pingable from other nodes even though it says Unreachable below. Correct ? Based on nodetool

Combining two clusters/keyspaces into single cluster

Hey Folks, I've been looking through various documentations, but I'm either overlooking something obvious or not wording it correctly, but the gist of my problem is this: I have two cassandra clusters, with two separate keyspaces on EC2. We'll call them as follows: *cluster1* (DC name, cluster

Re: Limit 1

As far as I know, the answer is yes, however it is unlikely that the cursor will have to probe very far to find a valid row unless your data is highly bursty. The key cache (assuming you have it enabled) will allow the query to skip unrelated rows in its search. However I would caution against

Re: Problem Replacing a Dead Node

Here is a bit more detail of the whole situation. I am hoping someone can help me out here. We have a seven node cluster. One the nodes started to have issues but it was running. We decided to add a new node, and remove the problematic node after the new node joins. However, the new node did not

Re: Problem Replacing a Dead Node

Hi Jeff, thanks for getting back to me. I have gone through the output of nodetool netstats, and it seems all the streams are 100% completed as per the output. What else do you think is going wrong? -Mir On Thu, Apr 21, 2016 at 10:27 AM, Jeff Jirsa wrote: > The

Re: What does Cassandra use (JNI?) that triggers GCLocker Initiated GCs?

We're running G1 at the moment, both young and mixed collections. On Thursday, April 21, 2016 11:07 AM, Jake Luciani wrote: What kind of collection? if its par new I wouldn't worry. On Thu, Apr 21, 2016 at 2:02 PM, Sotirios Delimanolis wrote:

Re: What does Cassandra use (JNI?) that triggers GCLocker Initiated GCs?

What kind of collection? if its par new I wouldn't worry. On Thu, Apr 21, 2016 at 2:02 PM, Sotirios Delimanolis wrote: > Should this be of any concern? Are the corresponding threads spending too > long in this JNI critical region and delaying GC? > > I don't get that

Re: Problem Replacing a Dead Node

Hi Anubhav, thanks for getting back to me. here is the information that you requested. datastax agent is running on the node. However, in the agent log I see ERROR [clojure-agent-send-off-pool-4] 2016-04-21 17:51:46,055 Can't connect to Cassandra (All host(s) tried for query failed (tried:

Re: What does Cassandra use (JNI?) that triggers GCLocker Initiated GCs?

Should this be of any concern? Are the corresponding threads spending too long in this JNI critical region and delaying GC? I don't get that impression at all from the GC log timings. They're very reasonable. On Thursday, April 21, 2016 10:57 AM, Jake Luciani wrote:

Re: What does Cassandra use (JNI?) that triggers GCLocker Initiated GCs?

It's only used by the Snappy and LZ4 Compressors On Thu, Apr 21, 2016 at 1:54 PM, Sotirios Delimanolis wrote: > According to this Oracle document > , GCLocker > Initiated GC > > is triggered when a JNI

What does Cassandra use (JNI?) that triggers GCLocker Initiated GCs?

According to this Oracle document, GCLocker Initiated GC is triggered when a JNI critical region was released. GC is blocked when any thread is in the JNI Critical region.If GC was requested during that period, that GC is invoked after all the threads come out of the JNI critical

Re: Problem Replacing a Dead Node

The keyspace with RF=1 may lose data, but isn’t blocking the replacement. The most likely cause of the delay is hung streaming. Run `nodetool netstats` on the joining (replacement) node. Do the byte counters change? If not, streaming is hung, and you’ll likely need to restart the process. If

RE: Problem Replacing a Dead Node

Is the datastax-agent running fine on the node ? What does nodetool status and system.log show ? From: Mir Tanvir Hossain [mailto:mir.tanvir.hoss...@gmail.com] Sent: Thursday, April 21, 2016 10:02 AM To: user@cassandra.apache.org Subject: Problem Replacing a Dead Node Hi, I am trying to replace

Problem Replacing a Dead Node

Hi, I am trying to replace a dead node with by following https://docs.datastax.com/en/cassandra/2.0/cassandra/operations/ops_replace_node_t.html. It's been 3 full days since the replacement node started, and the node is still not showing up as part of the cluster on OpsCenter. I was wondering

Re: Alternative approach to setting up new DC

Jens; I am unsure that you need to enable Replication & also use the sstable loader. You could load the data into the new DC and susbsequently alter the keyspace to replicate from the older DC. Cheers Jan On Thu, 4/21/16, Jens Rantil

Unable to reliably count keys on a thrift CF

Hi guys. I've been struggling for the last days to find a reliable and stable way to count keys in a thrift column family. My idea is to basically iterate the whole ring using the token function, as documented here: https://docs.datastax.com/en/cql/3.1/cql/cql_using/paging_c.html in batches of

Re: Alternative approach to setting up new DC

Hi, I never got any response here, but just wanted to share that I went to a Cassandra meet-up in Stockholm yesterday where I talked to two knowledgable Cassandra people that verified that the approach below should work. The most important thing is that the backup must be fully imported before

Re: When are hints written?

Hi again Bo, I assume this is the piece of documentation you are referring to? http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_about_hh_c.html?scroll=concept_ds_ifg_jqx_zj__performance > If a replica node is overloaded or unavailable, and the failure detector has not yet marked it

Re: A few misbehaving nodes

On 2016-04-19 15:54, sai krishnam raju potturi wrote: hi; do we see any hung process like Repairs on those 3 nodes? what does "nodetool netstats" show?? No hung process from what I can see. root@cssa02-06:~# nodetool tpstats Pool NameActive Pending Completed

RE: Cassandra 2.0.x OOM during bootstrap

Hi, all, Here is some more information on before the OOM happened on the rebooted node in a 2-node test cluster: 1. It seems the schema version has changed on the rebooted node after reboot, i.e. Before reboot, Node 1: DEBUG [MigrationStage:1] 2016-04-19 11:09:42,326

Re: When are hints written?

Hi Jens, Thank you for the tip! ALL would definitely cure our hints issue, but as you note, it is not optimal as we are unable to take down nodes without clients failing. I am most probably overlooking something in the documentation, but I cannot see any description of when hints are written