Re: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees

s://www.instaclustr.com/platform/ > > > > > > > > > > > > *From: *Fd Habash > *Reply-To: *"user@cassandra.apache.org" > *Date: *Wednesday, 1 May 2019 at 06:18 > *To: *"user@cassandra.apache.org" > *Subject: *Bootstrapping to Replace a

Re: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees

Thank you. Range movement is one reason this is enforced when adding a new node. But, what about forcing a consistent bootstrap i.e. bootstrapping from primary owner of the range and not a secondary replica. How’s consistent bootstrap enforced when replacing a dead node. - Thank you

Re: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees

/platform/ From: Fd Habash Reply-To: "user@cassandra.apache.org" Date: Wednesday, 1 May 2019 at 06:18 To: "user@cassandra.apache.org" Subject: Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees Reviewing the documentation & based on my

Bootstrapping to Replace a Dead Node vs. Adding a New Node: Consistency Guarantees

Reviewing the documentation & based on my testing, using C* 2.2.8, I was not able to extend the cluster by adding multiple nodes simultaneously. I got an error message … Other bootstrapping/leaving/moving nodes detected, cannot bootstrap while cassandra.consistent.rangemovement is true I unde

Re: Problem adding a new node to a cluster

Definitely upgrade to 3.11.1. On Sun, Dec 17, 2017 at 8:54 PM Pradeep Chhetri wrote: > Hello Kurt, > > I realized it was because of RAM shortage which caused the issue. I bumped > up the memory of the machine and node bootstrap started but this time i hit > this bug of cassandra 3.9: > > https://

Re: Problem adding a new node to a cluster

Hello Kurt, I realized it was because of RAM shortage which caused the issue. I bumped up the memory of the machine and node bootstrap started but this time i hit this bug of cassandra 3.9: https://issues.apache.org/jira/browse/CASSANDRA-12905 I tried running nodetool bootstrap resume multiple t

Re: Problem adding a new node to a cluster

You haven't provided enough logs for us to really tell what's wrong. I suggest running *nodetool netstats* *| grep -v 100% *to see if any streams are still ongoing, and also running *nodetool compactionstats -H* to see if there are any index builds the node might be waiting for prior to joining the

Problem adding a new node to a cluster

Hello all, I am trying to add a 4th node to a 3-node cluster which is using SimpleSnitch. But this new node is stuck in Joining state for last 20 hours. We have around 10GB data per node with RF as 3. Its mostly stuck in redistributing index summaries phase. Here are the logs: https://gist.gith

Re: Adding a New Node

Thanks Kurt! That sorted things in my head. Much appreciated! On Tue, Oct 24, 2017 at 12:29 PM, kurt greaves wrote: > Your node shouldn't show up in DC1 in nodetool status from the other > nodes, this implies a configuration problem. Sounds like you haven't added > the new node to all the exi

Re: Adding a New Node

Your node shouldn't show up in DC1 in nodetool status from the other nodes, this implies a configuration problem. Sounds like you haven't added the new node to all the existing nodes cassandra-topology.properties file. You don't need to do a rolling restart with PropertyFileSnitch, it should reload

Adding a New Node

Hi Everyone, I have 2 DCs (v2.0.14) with the following topology.properties: DC1: xxx11=DC1:RAC1 xxx12=DC1:RAC1 xxx13=DC1:RAC1 xxx14=DC1:RAC1 xxx15=DC1:RAC1 DC2: yyy11=DC2:RAC1 yyy12=DC2:RAC1 yyy13=DC2:RAC1 yyy14=DC2:RAC1 yyy15=DC2:RAC1 # default for unknown nodes default=DC1:RAC1 Now let's s

Re: Adding a new node with the double of disk space

important factors, as well as the future growth of > the cluster and the expected size of any additional nodes. > > > > > > Sean Durity > > > > From: Jeff Jirsa [mailto:jji...@gmail.com] > Sent: Thursday, August 17, 2017 4:20 PM > To: cassandra

Re: RE: Adding a new node with the double of disk space

st 17, 2017 4:20 PM To: cassandra Subject: Re: Adding a new node with the double of disk space   If you really double the hardware in every way, it's PROBABLY reasonable to double num_tokens. It won't be quite the same as doubling all-the-things, because you still have a single

RE: Adding a new node with the double of disk space

growth of the cluster and the expected size of any additional nodes. Sean Durity From: Jeff Jirsa [mailto:jji...@gmail.com] Sent: Thursday, August 17, 2017 4:20 PM To: cassandra Subject: Re: Adding a new node with the double of disk space If you really double the hardware in every way, it&#

Re: Adding a new node with the double of disk space

I would preferably spin 2 JVMs inside the same hardware (if you double everything) than having to deal with what Jeff stated. Also certain operations are not really found of a large number of vnodes (eg. repair). There was a lot of improvements in the 3.x release cycle, but I do still tend to redu

Re: Adding a new node with the double of disk space

If you really double the hardware in every way, it's PROBABLY reasonable to double num_tokens. It won't be quite the same as doubling all-the-things, because you still have a single JVM, and you'll still have to deal with GC as you're now reading twice as much and generating twice as much garbage,

Re: Adding a new node with the double of disk space

Are you saying if a node had double the hardware capacity in every way it would be a bad idea to up num_tokens? I thought that was the whole idea of that setting though? On Thu, Aug 17, 2017 at 9:52 AM, Carlos Rolo wrote: > No. > > If you would double all the hardware on that node vs the others

Re: Adding a new node with the double of disk space

No. If you would double all the hardware on that node vs the others would still be a bad idea. Keep the cluster uniform vnodes wise. Regards, Carlos Juzarte Rolo Cassandra Consultant / Datastax Certified Architect / Cassandra MVP Pythian - Love your data rolo@pythian | Twitter: @cjrolo | Skype

Adding a new node with the double of disk space

Hi all, I need to add a new node to my cluster but this time the new node will have the double of disk space comparing to the other nodes. I'm using the default vnodes (num_tokens: 256). To fully use the disk space in the new node I just have to configure num_tokens: 512? Thanks in advance. -

Re: Adding a New Node With The Same IP of an Old Node

Thanks Nitan! Eventually it was a firewall issue related to the Centos7 node. Once fixed, the rolling restart resolved the issue completely. Thanks again! Shalom Sagges DBA T: +972-74-700-4035

Re: Adding a New Node With The Same IP of an Old Node

Try rolling restart of cluster as solution for schema version mismatch. Sent from my iPhone > On Jul 4, 2017, at 8:31 AM, Shalom Sagges wrote: > > Hi Experts, > > My plan is to upgrade the C* nodes' OS from Centos6 to Centos7. > Since an upgrade wasn't recommended I needed to install new ma

Adding a New Node With The Same IP of an Old Node

Hi Experts, My plan is to upgrade the C* nodes' OS from Centos6 to Centos7. Since an upgrade wasn't recommended I needed to install new machines with Centos7 and join them to the cluster. I didn't want to decommission/bootstrap dozens of nodes, so I decided to do the following: - Create a new

Re: Error when running nodetool cleanup after adding a new node to a cluster

rds, > > Srinath Reddy > > On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin < > oleksandr.shul...@zalando.de> wrote: > > On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy wrote: > >> Hi, >> >> Trying to re-balacne a Cassandra cluster after adding a new node

Re: Error when running nodetool cleanup after adding a new node to a cluster

; Srinath Reddy > >> On 09-Feb-2017, at 1:44 PM, Oleksandr Shulgin > <mailto:oleksandr.shul...@zalando.de>> wrote: >> >> On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy > <mailto:ksre...@gmail.com>> wrote: >> Hi, >> >> Trying to re-balac

Re: Error when running nodetool cleanup after adding a new node to a cluster

re-balacne a Cassandra cluster after adding a new node and I'm > getting this error when running nodetool cleanup. The Cassandra cluster is > running in a Kubernetes cluster. > > Cassandra version is 2.2.8 > > nodetool cleanup > error: io.k8s.cassandra.KubernetesSeedProv

Re: Error when running nodetool cleanup after adding a new node to a cluster

On Thu, Feb 9, 2017 at 6:13 AM, Srinath Reddy wrote: > Hi, > > Trying to re-balacne a Cassandra cluster after adding a new node and I'm > getting this error when running nodetool cleanup. The Cassandra cluster > is running in a Kubernetes cluster. > > Cassandra ver

Re: Error when running nodetool cleanup after adding a new node to a cluster

o:ksre...@gmail.com>> wrote: > >> Hi, >> >> Trying to re-balacne a Cassandra cluster after adding a new node and I'm >> getting this error when running nodetool cleanup. The Cassandra cluster is >> running in a Kubernetes clus

Re: Error when running nodetool cleanup after adding a new node to a cluster

The cleanup has to run on other nodes Sent from my iPhone On Feb 8, 2017, at 9:14 PM, Srinath Reddy mailto:ksre...@gmail.com>> wrote: Hi, Trying to re-balacne a Cassandra cluster after adding a new node and I'm getting this error when running nodetool cleanup. The Cassandra

Error when running nodetool cleanup after adding a new node to a cluster

Hi, Trying to re-balacne a Cassandra cluster after adding a new node and I'm getting this error when running nodetool cleanup. The Cassandra cluster is running in a Kubernetes cluster. Cassandra version is 2.2.8 nodetool cleanup error: io.k8s.cassandra.KubernetesSeedProvider

Re: Read operations freeze for a few second while adding a new node

Hi Lorand, Do you see any different gc pattern during these 20 seconds? In 2.0.x, memtable create lot of heap pressure. So in a way, reads are not isolated from writes. Frankly speaking, I would have accepted 20 second slowness as scaling is one time activity. But may be your business case doesnt

Re: Read operations freeze for a few second while adding a new node

ct: Read operations freeze for a few second while adding a new node Hi, We are struggling with a problem that when adding nodes around 5% read operations freeze (aka time out after 1 second) for a few seconds (10-20 seconds). It might not seems much, but at the order of 200k requests per second

Re: Read operations freeze for a few second while adding a new node

If you've got a read heavy workload you should check out http://blakeeggleston.com/cassandra-tuning-the-jvm-for-read-heavy-workloads.html On Thu, Jan 28, 2016 at 8:11 AM Lorand Kasler wrote: > Hi, > > We are struggling with a problem that when adding nodes around 5% read > operations freeze (a

Read operations freeze for a few second while adding a new node

Hi, We are struggling with a problem that when adding nodes around 5% read operations freeze (aka time out after 1 second) for a few seconds (10-20 seconds). It might not seems much, but at the order of 200k requests per second that's quite big of disruption. It is well documented and known that

Re: [MASSMAIL]Re: Error while adding a new node.

thanks for the reply.!! I will update it to 2.1.7 and checkout. On Thu, Jul 2, 2015 at 6:59 PM, Carlos Rolo wrote: > Marco you should also avoid 2.1.5 and 2.1.6 because of > https://issues.apache.org/jira/browse/CASSANDRA-9549 > > I know (And often don't recommend last versions, I'm still recomm

Re: [MASSMAIL]Re: Error while adding a new node.

Marco you should also avoid 2.1.5 and 2.1.6 because of https://issues.apache.org/jira/browse/CASSANDRA-9549 I know (And often don't recommend last versions, I'm still recommending 2.0.x series unless someone is already in 2.1.x) but given the above bug, 2.1.7 is the best option. Regards, Carlos

Re: [MASSMAIL]Re: Error while adding a new node.

The recommended version to use is 2.1.5 because, like you Carlos said, 2.1.6 and 2.1.7 are very new to consider them like stable. On 02/07/15 08:55, Carlos Rolo wrote: Indeed you should upgrade to 2.1.7. And then report if you are still facing problems. Versions up to 2.1.5 (in the 2.1.x seri

Re: Error while adding a new node.

Indeed you should upgrade to 2.1.7. And then report if you are still facing problems. Versions up to 2.1.5 (in the 2.1.x series) are not considered stable. Regards, Carlos Juzarte Rolo Cassandra Consultant Pythian - Love your data rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/car

Re: Error while adding a new node.

any help? On Thu, Jul 2, 2015 at 6:18 AM, Neha Trivedi wrote: > also: > root@cas03:~# sudo service cassandra start > root@cas03:~# lsof -n | grep java | wc -l > 5315 > root@cas03:~# lsof -n | grep java | wc -l > 977317 > root@cas03:~# lsof -n | grep java | wc -l > 880240 > root@cas03:~# lsof -n

Re: Error while adding a new node.

also: root@cas03:~# sudo service cassandra start root@cas03:~# lsof -n | grep java | wc -l 5315 root@cas03:~# lsof -n | grep java | wc -l 977317 root@cas03:~# lsof -n | grep java | wc -l 880240 root@cas03:~# lsof -n | grep java | wc -l 882402 On Wed, Jul 1, 2015 at 6:31 PM, Neha Trivedi wrote:

Re: Stream failure while adding a new node

il.com] Envoyé : mardi 30 juin 2015 15:18 À : user@cassandra.apache.org Objet : Re: Stream failure while adding a new node   Hi David,   Are you sure you ran the repair entirely (9 days + repair logs ok on opscenter server) before adding the 10th node ? This is important to avoid pote

RE: Stream failure while adding a new node

ystem.com/> [cid:image003.png@01D0B41D.54B89AA0] De : Alain RODRIGUEZ [mailto:arodr...@gmail.com] Envoyé : mardi 30 juin 2015 15:18 À : user@cassandra.apache.org Objet : Re: Stream failure while adding a new node Hi David, Are you sure you ran the repair entirely (9 days + repair logs ok on opsce

Re: Error while adding a new node.

One of the column family has SStable count as under : SSTable count: 98506 Can it be because of 2.1.3 version of cassandra.. I found this : https://issues.apache.org/jira/browse/CASSANDRA-8964 regards Neha On Wed, Jul 1, 2015 at 5:40 PM, Jason Wee wrote: > nodetool cfstats? > > On Wed, Jul 1,

Re: Error while adding a new node.

nodetool cfstats? On Wed, Jul 1, 2015 at 8:08 PM, Neha Trivedi wrote: > Hey.. > nodetool compactionstats > pending tasks: 0 > > no pending tasks. > > Dont have opscenter. how do I monitor sstables? > > > On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ > wrote: > >> You also might want to check

Re: Error while adding a new node.

Hey.. nodetool compactionstats pending tasks: 0 no pending tasks. Dont have opscenter. how do I monitor sstables? On Wed, Jul 1, 2015 at 4:28 PM, Alain RODRIGUEZ wrote: > You also might want to check if you have compactions pending (Opscenter / > nodetool compactionstats). > > Also you can mo

Re: Error while adding a new node.

You also might want to check if you have compactions pending (Opscenter / nodetool compactionstats). Also you can monitor the number of sstables. C*heers Alain 2015-07-01 11:53 GMT+02:00 Neha Trivedi : > Thanks I will checkout. > I increased the ulimit to 10, but I am getting the same erro

Re: Error while adding a new node.

Thanks I will checkout. I increased the ulimit to 10, but I am getting the same error, but after a while. regards Neha On Wed, Jul 1, 2015 at 2:22 PM, Alain RODRIGUEZ wrote: > Just check the process owner to be sure (top, htop, ps, ...) > > > http://docs.datastax.com/en/cassandra/2.0/cassan

Re: Error while adding a new node.

Just check the process owner to be sure (top, htop, ps, ...) http://docs.datastax.com/en/cassandra/2.0/cassandra/install/installRecommendSettings.html#reference_ds_sxl_gf3_2k__user-resource-limits C*heers, Alain 2015-07-01 7:33 GMT+02:00 Neha Trivedi : > Arun, > I am logging on to Server as ro

Re: Error while adding a new node.

Arun, I am logging on to Server as root and running (sudo service cassandra start) regards Neha On Wed, Jul 1, 2015 at 11:00 AM, Neha Trivedi wrote: > Thanks Arun ! I will try and get back ! > > On Wed, Jul 1, 2015 at 10:32 AM, Arun wrote: > >> Looks like you have too many open files issue. In

Re: Error while adding a new node.

Thanks Arun ! I will try and get back ! On Wed, Jul 1, 2015 at 10:32 AM, Arun wrote: > Looks like you have too many open files issue. Increase the ulimit for the > user. > > If you are starting the cassandra daemon using user cassandra, increase > the ulimit for that user. > > > > On Jun 30, 20

Re: Error while adding a new node.

Looks like you have too many open files issue. Increase the ulimit for the user. If you are starting the cassandra daemon using user cassandra, increase the ulimit for that user. > On Jun 30, 2015, at 21:16, Neha Trivedi wrote: > > Hello, > I have a 4 node cluster with SimpleSnitch. > Cassan

Error while adding a new node.

Hello, I have a 4 node cluster with SimpleSnitch. Cassandra : Cassandra 2.1.3 I am trying to add a new node (cassandra 2.1.7) and I get the following error. ERROR [STREAM-IN-] 2015-06-30 05:13:48,516 JVMStabilityInspector.java:94 - JVM state determined to be unstable. Exiting forcefully due to:

Re: Stream failure while adding a new node

Hi David, Are you sure you ran the repair entirely (9 days + repair logs ok on opscenter server) before adding the 10th node ? This is important to avoid potential data loss ! Did you set auto_bootstrap to true on this 10th node ? C*heers, Alain 2015-06-29 14:54 GMT+02:00 David CHARBONNIER :

Stream failure while adding a new node

Hi, We're using Cassandra 2.0.8.39 through Datastax Enterprise 4.5.1 with a 9 nodes cluster. We need to add a few new nodes to the cluster but we're experiencing an issue we don't know how to solve. Here is exactly what we did : - We had 8 nodes and need to add a few ones - W

Re: Problem in adding a new node

Do you use a 32 bit JVM ? If so I refer you to the following thread: http://mail-archives.apache.org/mod_mbox/cassandra-user/201204.mbox/%3ccaldd-zgthksc2bikp3h4trjxo5vcnhkl2wpwclsf+d9sqty...@mail.gmail.com%3E In short, avoids 32 bits, but if you really cannot, set commitlog_total_space_in_mb to a

RE: Problem in adding a new node

Yes I gave the ip address of the 1st node in the seeds option Thanks and Regards Prakrati From: MOHD ARSHAD SALEEM [mailto:marshadsal...@tataelxsi.co.in] Sent: Friday, June 08, 2012 12:51 PM To: user@cassandra.apache.org Subject: RE: Problem in adding a new node Hi, the node which you added in

RE: Problem in adding a new node

in adding a new node Dear all, I had a 1 node cluster of Cassandra. Then I added one more node to it and started Cassandra on it. I got the following error: INFO 12:44:49,588 Loading persisted ring state ERROR 12:44:49,613 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError

Problem in adding a new node

Dear all, I had a 1 node cluster of Cassandra. Then I added one more node to it and started Cassandra on it. I got the following error: INFO 12:44:49,588 Loading persisted ring state ERROR 12:44:49,613 Exception in thread Thread[COMMIT-LOG-ALLOCATOR,5,main] java.io.IOError: java.io.IOException:

Re: Adding a new node to Cassandra cluster

eloper - Big Data(I&D)| 9731648376 | > www.mu-sigma.com > > ** ** > > *From:* samal [mailto:samalgo...@gmail.com] > *Sent:* Monday, June 04, 2012 3:12 PM > > *To:* user@cassandra.apache.org > *Subject:* Re: Adding a new node to Cassandra cluster > > ** **

RE: Adding a new node to Cassandra cluster

algo...@gmail.com] Sent: Monday, June 04, 2012 3:12 PM To: user@cassandra.apache.org Subject: Re: Adding a new node to Cassandra cluster If you use thrift API, you have to maintain lot of low level code by yourself which is already being polished by HLC hector, pycassa also with HLC your can ea

Re: Adding a new node to Cassandra cluster

;D)| 9731648376 | >> www.mu-sigma.com >> >> ** ** >> >> *From:* R. Verlangen [mailto:ro...@us2.nl] >> *Sent:* Monday, June 04, 2012 2:44 PM >> *To:* user@cassandra.apache.org >> *Subject:* Re: Adding a new node to Cassandra cluster >>

Re: Adding a new node to Cassandra cluster

t; Date: Mon, 4 Jun 2012 02:30:40 -0700 To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: Adding a new node to Cassandra cluster You might consider using a higher level client (like Hector indeed). If you don't

Re: Adding a new node to Cassandra cluster

a.com > > ** ** > > *From:* R. Verlangen [mailto:ro...@us2.nl] > *Sent:* Monday, June 04, 2012 2:44 PM > *To:* user@cassandra.apache.org > *Subject:* Re: Adding a new node to Cassandra cluster > > ** ** > > Hi there, > > ** ** > > When you speak

RE: Adding a new node to Cassandra cluster

Sent: Monday, June 04, 2012 2:44 PM To: user@cassandra.apache.org Subject: Re: Adding a new node to Cassandra cluster Hi there, When you speak to one node it will internally redirect the request to the proper node (local / external): but you won't be able to failover on a crash of the local

Re: Adding a new node to Cassandra cluster

Hi there, When you speak to one node it will internally redirect the request to the proper node (local / external): but you won't be able to failover on a crash of the localhost. For adding another node to the connection pool you should take a look at the documentation of your java client. Good l

Adding a new node to Cassandra cluster

Dear all I successfully added a new node to my cluster so now it's a 2 node cluster. But how do I mention it in my Java code as when I am retrieving data its retrieving only for one node that I am specifying in the localhost. How do I specify more than one node in the localhost. Please help me

Re: Adding a new node to already existing single-node-cluster cassandra

Sounds similar to http://www.mail-archive.com/user@cassandra.apache.org/msg20926.html Are you able to try adding the node again with logging set to DEBUG (in /etc/cassandra/log4j-server.properties) . (Please make sure the system directory is empty (/var/lib/cassandra/data/system) *NOTE* do not

Adding a new node to already existing single-node-cluster cassandra

Hello, I have been trying to add a node to single node cluster of Cassandra (1.0.8) but I always get following error: INFO 17:50:35,555 JOINING: schema complete, ready to bootstrap INFO 17:50:35,556 JOINING: getting bootstrap token ERROR 17:50:35,557 Exception encountered during startup java.lan

Re: Could not reach schema agreement when adding a new node.

from "Migration". Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 24/09/2011, at 10:10 PM, Dikang Gu wrote: > I found this in the system.log when adding a new node to the cluster. > > Anyone familiar with this? >

Could not reach schema agreement when adding a new node.

I found this in the system.log when adding a new node to the cluster. Anyone familiar with this? ERROR [HintedHandoff:2] 2011-09-24 18:01:30,498 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[HintedHandoff:2,1,main] java.lang.RuntimeException: java.lang.RuntimeException

Re: Adding a new node

Gossip should help them converge on the truth. Can you give an example of the different views from nodetool ring ? Also check the logs to see if there is anything been logged about endpoints. Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://

Re: Adding a new node

Thanks for the pointer. I restarted entire cluster and started nodes at the same time. However, I still see the issue. The view is not consistant. Am running 0.7.5. In general, if a node with bad ring view starts first, then I guess the restart also doesnt help as it might be propagating its view

Re: Adding a new node

It is possible to change IP address of a node, background http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/change-node-IP-address-td6197607.html If you have already bought a new node back with a different IP and the nodes in the cluster have different views of the ring (nodetoo

Adding a new node

Hi, I am trying to bring up a new node (with different IP) to replace a dead node on cassandra 0.7.5. Rather than bootstrap, I am copying the SSTable files to the new node(backed up files) as my data runs into several GB. Although the node successfully joins the ring, some of the ring nodes sti