Databasde connections based Cassandra users

[Chuck Reynolds]

Is there a way to see who is connected to Cassandra based on a Cassandra user?

Invalid Gossip generation with Cassandra 2.1.12

[Chuck Reynolds]

Calling all super heros.

I have a long standing Cassandra 2.1.12 ring that has an occasional node that 
gets restarted and then is flagged with the invalid gossip generation error 
leaving him down in nodetool status but the logs make it look like the nodes is 
ok.

It’s only when I look at the other nodes logs that I see there are pointing to 
the invalid gossip generation error and marking him down.

I know this issue has been fixed in 2.1.13 but how can I Install the fixed 
version and roll restart on a production ring without seeing the error?

Any help would be greatly appreciated

Re: system_auth replication factor in Cassandra 2.1

[Chuck Reynolds]

 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-08-30 10:51:25.003000 
| xx.xx.xx.116 |178
   Read 
1 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-08-30 10:51:25.003000 
| xx.xx.xx.116 |186
   Read 
1 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-08-30 10:51:25.003000 
| xx.xx.xx.116 |191
   Read 
1 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-08-30 10:51:25.003000 
| xx.xx.xx.116 |194
   Read 
1 live and 0 tombstone cells [SharedPool-Worker-1] | 2017-08-30 10:51:25.004000 
| xx.xx.xx.116 |198

Scanned 5 rows and matched 5 [SharedPool-Worker-1] | 2017-08-30 10:51:25.004000 
| xx.xx.xx.116 |224

Enqueuing response to /xx.xx.xx.113 [SharedPool-Worker-1] | 2017-08-30 
10:51:25.004000 | xx.xx.xx.116 |240
 Sending REQUEST_RESPONSE message to 
/xx.xx.xx.113 [MessagingService-Outgoing-/xx.xx.xx.113] | 2017-08-30 
10:51:25.004000 | xx.xx.xx.116 |302
 
Enqueuing request to /xx.xx.xx.116 [SharedPool-Worker-2] | 2017-08-30 
10:51:25.014000 | xx.xx.xx.113 | 601103
 Submitted 1 concurrent range 
requests covering 63681 ranges [SharedPool-Worker-2] | 2017-08-30 
10:51:25.014000 | xx.xx.xx.113 | 601120
  Sending PAGED_RANGE message to 
/xx.xx.xx.116 [MessagingService-Outgoing-/xx.xx.xx.116] | 2017-08-30 
10:51:25.015000 | xx.xx.xx.113 | 601190
  REQUEST_RESPONSE message received from 
/xx.xx.xx.116 [MessagingService-Incoming-/xx.xx.xx.116] | 2017-08-30 
10:51:25.015000 | xx.xx.xx.113 | 601771
 Processing 
response from /xx.xx.xx.116 [SharedPool-Worker-1] | 2017-08-30 10:51:25.015000 
| xx.xx.xx.113 | 601824

  Request complete | 2017-08-30 10:51:25.014874 
| xx.xx.xx.113 | 601874


From: Oleksandr Shulgin <oleksandr.shul...@zalando.de>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wednesday, August 30, 2017 at 10:42 AM
To: User <user@cassandra.apache.org>
Subject: Re: system_auth replication factor in Cassandra 2.1

On Wed, Aug 30, 2017 at 6:40 PM, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:
How many users do you have (or expect to be found in system_auth.users)?
  5 users.
What are the current RF for system_auth and consistency level you are using in 
cqlsh?
 135 in one DC and 227 in the other DC.  Consistency level one

Still very surprising...

Did you try to obtain a trace of a timing-out query (with TRACING ON)?
Tracing timeout even though I increased it to 120 seconds.

Even if cqlsh doesn't print the trace because of timeout, you should be still 
able to find something in system_traces.

--
Alex

Re: system_auth replication factor in Cassandra 2.1

[Chuck Reynolds]

How many users do you have (or expect to be found in system_auth.users)?
  5 users.
What are the current RF for system_auth and consistency level you are using in 
cqlsh?
 135 in one DC and 227 in the other DC.  Consistency level one
Did you try to obtain a trace of a timing-out query (with TRACING ON)?
Tracing timeout even though I increased it to 120 seconds.

From: Oleksandr Shulgin <oleksandr.shul...@zalando.de>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wednesday, August 30, 2017 at 10:19 AM
To: User <user@cassandra.apache.org>
Subject: Re: system_auth replication factor in Cassandra 2.1

On Wed, Aug 30, 2017 at 5:50 PM, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:
So I’ve read that if your using authentication in Cassandra 2.1 that your 
replication factor should match the number of nodes in your datacenter.

Is that true?

I have two datacenter cluster, 135 nodes in datacenter 1 & 227 nodes in an AWS 
datacenter.

Why do I want to replicate the system_auth table that many times?

What are the benefits and disadvantages of matching the number of nodes as 
opposed to the standard replication factor of 3?


The reason I’m asking the question is because it seems like I’m getting a lot 
of authentication errors now and they seem to happen more under load.

Also, querying the system_auth table from cqlsh to get the users seems to now 
timeout.

This is surprising.

How many users do you have (or expect to be found in system_auth.users)?   What 
are the current RF for system_auth and consistency level you are using in 
cqlsh?  Did you try to obtain a trace of a timing-out query (with TRACING ON)?

Regards,
--
Oleksandr "Alex" Shulgin | Database Engineer | Zalando SE | Tel: +49 176 
127-59-707

Re: system_auth replication factor in Cassandra 2.1

[Chuck Reynolds]

So I tried to run a repair with the following on one of the server.
nodetool repair system_auth -pr –local

After two hours it hadn’t finished.  I had to kill the repair because of 
another issue and haven’t tried again.

Why would such a small table take so long to repair?

Also what would happen if I set the RF back to a lower number like 5?


Thanks
From: <li...@beobal.com> on behalf of Sam Tunnicliffe <s...@beobal.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wednesday, August 30, 2017 at 10:10 AM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: system_auth replication factor in Cassandra 2.1

It's a better rule of thumb to use an RF of 3 to 5 per DC and this is what the 
docs now suggest: 
http://cassandra.apache.org/doc/latest/operating/security.html#authentication
Out of the box, the system_auth keyspace is setup with SimpleStrategy and RF=1 
so that it works on any new system including dev & test clusters, but obviously 
that's no use for a production system.

Regarding the increased rate of authentication errors: did you run repair after 
changing the RF? Auth queries are done at CL.LOCAL_ONE, so if you haven't 
repaired, the data for the user logging in will probably not be where it should 
be. The exception to this is the default "cassandra" user, queries for that 
user are done at CL.QUORUM, which will indeed lead to timeouts and 
authentication errors with a very high RF. It's recommended to only use that 
default user to bootstrap the setup of your own users & superusers, the link 
above also has info on this.

Thanks,
Sam


On 30 August 2017 at 16:50, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:
So I’ve read that if your using authentication in Cassandra 2.1 that your 
replication factor should match the number of nodes in your datacenter.

Is that true?

I have two datacenter cluster, 135 nodes in datacenter 1 & 227 nodes in an AWS 
datacenter.

Why do I want to replicate the system_auth table that many times?

What are the benefits and disadvantages of matching the number of nodes as 
opposed to the standard replication factor of 3?


The reason I’m asking the question is because it seems like I’m getting a lot 
of authentication errors now and they seem to happen more under load.

Also, querying the system_auth table from cqlsh to get the users seems to now 
timeout.


Any help would be greatly appreciated.

Thanks

system_auth replication factor in Cassandra 2.1

[Chuck Reynolds]

So I’ve read that if your using authentication in Cassandra 2.1 that your 
replication factor should match the number of nodes in your datacenter.

Is that true?

I have two datacenter cluster, 135 nodes in datacenter 1 & 227 nodes in an AWS 
datacenter.

Why do I want to replicate the system_auth table that many times?

What are the benefits and disadvantages of matching the number of nodes as 
opposed to the standard replication factor of 3?


The reason I’m asking the question is because it seems like I’m getting a lot 
of authentication errors now and they seem to happen more under load.

Also, querying the system_auth table from cqlsh to get the users seems to now 
timeout.


Any help would be greatly appreciated.

Thanks

Re: UnauthorizedException: user has no select permissions when quering Cassandra

[Chuck Reynolds]

Yes.  We’ve tried them all.

From: Akhil Mehra <akhilme...@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Tuesday, August 29, 2017 at 4:06 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: Re: UnauthorizedException: user has no select permissions when quering 
Cassandra

What consistency are you running the query with? Does the query timeout even 
with a consistency of one?


On 30/08/2017, at 9:49 AM, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:


We have that keyspace replicated to the same number of nodes in the ring.

Right now I can’t even run select * from system_auth.users without it timeing 
out.

From: Akhil Mehra <akhilme...@gmail.com<mailto:akhilme...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Tuesday, August 29, 2017 at 3:46 PM
To: user <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: UnauthorizedException: user has no select permissions when quering 
Cassandra

https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureConfigInternalAuth.html

Re: UnauthorizedException: user has no select permissions when quering Cassandra

[Chuck Reynolds]

We have that keyspace replicated to the same number of nodes in the ring.

Right now I can’t even run select * from system_auth.users without it timeing 
out.

From: Akhil Mehra 
Reply-To: "user@cassandra.apache.org" 
Date: Tuesday, August 29, 2017 at 3:46 PM
To: user 
Subject: Re: UnauthorizedException: user has no select permissions when quering 
Cassandra

https://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureConfigInternalAuth.html

UnauthorizedException: user has no select permissions when quering Cassandra

[Chuck Reynolds]

I’m receiving the following error when quering a table that I know the user has 
super user rights to.

It only happens about 10% of the time.

com.datastax.driver.core.exceptions.UnauthorizedException: User test has no 
SELECT permission on  or any of its parents.

Measuring Read Repairs

[Chuck Reynolds]

I’m running queries based on the token ranges to initiate read repairs across 
datacenter.

Example query with CONSISTENCY set to ALL
SELECT token(test_guid), test_guid FROM test_table WHERE 
token(test_guid)>6546138161478345924 AND token(test_guid)<6571069709219758671;

Is there a way to tell if read repairs are happening based on this query?

I’m not seeing anything in the logs.

Re: Upgrade requirements for upgrading from cassandra 2.1.x to 2.2.x

[Chuck Reynolds]

Anyone?

From: "Chuck (me) Reynolds" 
Reply-To: "user@cassandra.apache.org" 
Date: Tuesday, August 22, 2017 at 9:40 AM
To: "user@cassandra.apache.org" 
Subject: Upgrade requirements for upgrading from cassandra 2.1.x to 2.2.x

Where can I find requirements to upgrade from Cassandra 2.1.x to 2.2.x?

I would like to know things like do I have to do an SStable upgrade or not.


Thanks

Upgrade requirements for upgrading from cassandra 2.1.x to 2.2.x

[Chuck Reynolds]

Where can I find requirements to upgrade from Cassandra 2.1.x to 2.2.x?

I would like to know things like do I have to do an SStable upgrade or not.


Thanks

Re: JOB | Permanent Database Engineer (the Netherlands)

[Chuck Reynolds]

Tell me more.

On 8/8/17, 10:44 PM, "James Tobin"  wrote:

Hello, I'm working with an employer that is looking to hire someone to
become their lead on Apache Cassandra in the Netherlands.
Consequently I had hoped that some members of this mailing list may
like to discuss further off-list using "JamesBTobin (at) Gmail (dot)
com".  Kind regards, James

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Different data size between datacenters

[Chuck Reynolds]

Keyspace has WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': 
'3', 'us-east-productiondata': '3'}  AND durable_writes = true;

From: Jeff Jirsa <jji...@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Monday, August 7, 2017 at 2:51 PM
To: cassandra <user@cassandra.apache.org>
Subject: Re: Different data size between datacenters

And when you say the data size is smaller, you mean per node? Or sum of all 
nodes in the datacenter?

With 185 hosts in AWS vs 135 in your DC, I would expect your DC hosts to have  
30% less data per host than AWS.

If instead they have twice as much, it sounds like it's balancing by # of 
tokens instead, which may be an indication that you're somehow using 
SimpleStrategy, or your NetworkTopologyStrategy is somehow misconfigured for 
one or more keyspaces.

Can you paste your keyspace replication strategy lines, anonymized as needed?


On Mon, Aug 7, 2017 at 1:46 PM, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:
Yes to the NetworkTopologyStrategy.

From: Jeff Jirsa <jji...@gmail.com<mailto:jji...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Monday, August 7, 2017 at 2:39 PM
To: cassandra <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Different data size between datacenters

You're using NetworkTopologyStrategy and not SimpleStrategy, correct?


On Mon, Aug 7, 2017 at 11:50 AM, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:
I have a cluster that spans two datacenters running Cassandra 2.1.12.  135 
nodes in my data center and about 185 in AWS.

The size of the second data center (AWS) is quite a bit smaller.  Replication 
is the same in both datacenters.  Is there a logical explanation for this?

thanks

Re: Different data size between datacenters

[Chuck Reynolds]

Yes it’s the total size.

Could it be that tombstones or data that nodes no longer own is not being 
copied/streamed to the data center in AWS?

From: Jeff Jirsa <jji...@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Monday, August 7, 2017 at 2:51 PM
To: cassandra <user@cassandra.apache.org>
Subject: Re: Different data size between datacenters

And when you say the data size is smaller, you mean per node? Or sum of all 
nodes in the datacenter?

With 185 hosts in AWS vs 135 in your DC, I would expect your DC hosts to have  
30% less data per host than AWS.

If instead they have twice as much, it sounds like it's balancing by # of 
tokens instead, which may be an indication that you're somehow using 
SimpleStrategy, or your NetworkTopologyStrategy is somehow misconfigured for 
one or more keyspaces.

Can you paste your keyspace replication strategy lines, anonymized as needed?


On Mon, Aug 7, 2017 at 1:46 PM, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:
Yes to the NetworkTopologyStrategy.

From: Jeff Jirsa <jji...@gmail.com<mailto:jji...@gmail.com>>
Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Monday, August 7, 2017 at 2:39 PM
To: cassandra <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Different data size between datacenters

You're using NetworkTopologyStrategy and not SimpleStrategy, correct?


On Mon, Aug 7, 2017 at 11:50 AM, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:
I have a cluster that spans two datacenters running Cassandra 2.1.12.  135 
nodes in my data center and about 185 in AWS.

The size of the second data center (AWS) is quite a bit smaller.  Replication 
is the same in both datacenters.  Is there a logical explanation for this?

thanks

Re: Different data size between datacenters

[Chuck Reynolds]

So we have the default 256 in our datacenter and 128 in AWS.

From: "ZAIDI, ASAD A" <az1...@att.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Monday, August 7, 2017 at 1:36 PM
To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Subject: RE: Different data size between datacenters

Are you using same number of token/vnodes in both data centers?

From: Chuck Reynolds [mailto:creyno...@ancestry.com]
Sent: Monday, August 07, 2017 1:51 PM
To: user@cassandra.apache.org
Subject: Different data size between datacenters

I have a cluster that spans two datacenters running Cassandra 2.1.12.  135 
nodes in my data center and about 185 in AWS.

The size of the second data center (AWS) is quite a bit smaller.  Replication 
is the same in both datacenters.  Is there a logical explanation for this?

thanks

Re: Different data size between datacenters

[Chuck Reynolds]

Yes to the NetworkTopologyStrategy.

From: Jeff Jirsa <jji...@gmail.com>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Monday, August 7, 2017 at 2:39 PM
To: cassandra <user@cassandra.apache.org>
Subject: Re: Different data size between datacenters

You're using NetworkTopologyStrategy and not SimpleStrategy, correct?


On Mon, Aug 7, 2017 at 11:50 AM, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:
I have a cluster that spans two datacenters running Cassandra 2.1.12.  135 
nodes in my data center and about 185 in AWS.

The size of the second data center (AWS) is quite a bit smaller.  Replication 
is the same in both datacenters.  Is there a logical explanation for this?

thanks

Different data size between datacenters

[Chuck Reynolds]

I have a cluster that spans two datacenters running Cassandra 2.1.12.  135 
nodes in my data center and about 185 in AWS.

The size of the second data center (AWS) is quite a bit smaller.  Replication 
is the same in both datacenters.  Is there a logical explanation for this?

thanks

Can I have multiple datacenter with different versions of Cassandra

[Chuck Reynolds]

I have a need to create another datacenter and upgrade my existing Cassandra 
from 2.1.13 to Cassandra 3.0.9.

Can I do this as one step?  Create a new Cassandra ring that is version 3.0.9 
and replicate the data from an existing ring that is Cassandra 2.1.13?

After replicating to the new ring if possible them I would upgrade the old ring 
to Cassandra 3.0.9

Random slow read times in Cassandra

[Chuck Reynolds]

I have a large Cassandra 2.1.13 ring (60 nodes) in AWS that has consistently 
random high read times.  In general most reads are under 10 milliseconds but 
with in the 30 request there is usually a read time that is a couple of seconds.

Instance type: r4.8xlarge
EBS GP2 volumes, 3tb with 9000 IOPS
30 Gig Heap

Data per node is about 170 gigs

The keyspace is an id & a blob.  When I check the data the slow reads don’t 
seem to have anything to do with size of the blobs

This system has repairs run once a weeks because it takes a lot of updates.

The client makes a call and does 30 request serially to Cassandra and the 
response times look like this in milliseconds.

What could make these so slow and what can I do to diagnosis this?


Responses

Get Person time: 3 319746229:9009:66
Get Person time: 7 1830093695:9009:66
Get Person time: 4 30072253:9009:66
Get Person time: 4 2303790089:9009:66
Get Person time: 2 156792066:9009:66
Get Person time: 8 491230624:9009:66
Get Person time: 7 284904599:9009:66
Get Person time: 4 600370489:9009:66
Get Person time: 2 281007386:9009:66
Get Person time: 4 971178094:9009:66
Get Person time: 1 1322259885:9009:66
Get Person time: 2 1937958542:9009:66
Get Person time: 9 286536648:9009:66
Get Person time: 9 1835633470:9009:66
Get Person time: 2 300867513:9009:66
Get Person time: 3 178975468:9009:66
Get Person time: 2900 293043081:9009:66
Get Person time: 8 214913830:9009:66
Get Person time: 2 1956710764:9009:66
Get Person time: 4 237673776:9009:66
Get Person time: 17 68942206:9009:66
Get Person time: 1800 20072145:9009:66
Get Person time: 2 304698506:9009:66
Get Person time: 2 308177320:9009:66
Get Person time: 2 998436038:9009:66
Get Person time: 10 1036890112:9009:66
Get Person time: 1 1629649548:9009:66
Get Person time: 6 1595339706:9009:66
Get Person time: 4 1079637599:9009:66
Get Person time: 3 556342855:9009:66


Get Person time: 5 1856382256:9009:66
Get Person time: 3 1891737174:9009:66
Get Person time: 2 1179373651:9009:66
Get Person time: 2 1482602756:9009:66
Get Person time: 3 1236458510:9009:66
Get Person time: 11 1003159823:9009:66
Get Person time: 2 1264952556:9009:66
Get Person time: 2 1662234295:9009:66
Get Person time: 1 246108569:9009:66
Get Person time: 5 1709881651:9009:66
Get Person time: 3213 11878078:9009:66
Get Person time: 2 112866483:9009:66
Get Person time: 2 201870153:9009:66
Get Person time: 6 227696684:9009:66
Get Person time: 2 1946780190:9009:66
Get Person time: 2 2197987101:9009:66
Get Person time: 18 1838959725:9009:66
Get Person time: 3 1782937802:9009:66
Get Person time: 3 1692530939:9009:66
Get Person time: 9 1765654196:9009:66
Get Person time: 2 1597757121:9009:66
Get Person time: 2 1853127153:9009:66
Get Person time: 3 1533599253:9009:66
Get Person time: 6 1693244112:9009:66
Get Person time: 6 82047537:9009:66
Get Person time: 2 96221961:9009:66
Get Person time: 4 98202209:9009:66
Get Person time: 9 12952388:9009:66
Get Person time: 2 300118652:9009:66
Get Person time: 10 78801084:9009:66


Get Person time: 13 1856424913:9009:66
Get Person time: 2 255814186:9009:66
Get Person time: 2 1183397424:9009:66
Get Person time: 5 1828603730:9009:66
Get Person time: 9 132965919:9009:66
Get Person time: 4 1616190071:9009:66
Get Person time: 2 15929337:9009:66
Get Person time: 10 297005427:9009:66
Get Person time: 2 1306460047:9009:66
Get Person time: 5 620139216:9009:66
Get Person time: 2 1364349058:9009:66
Get Person time: 3 629543403:9009:66
Get Person time: 5 1299827034:9009:66
Get Person time: 4 1593205912:9009:66
Get Person time: 2 1755460077:9009:66
Get Person time: 2 1906388666:9009:66
Get Person time: 1 1838653952:9009:66
Get Person time: 2 2249662508:9009:66
Get Person time: 3 1931708432:9009:66
Get Person time: 2 2177004948:9009:66
Get Person time: 2 2042756682:9009:66
Get Person time: 5 41764865:9009:66
Get Person time: 4023 1733384704:9009:66
Get Person time: 1 1614842189:9009:66
Get Person time: 2 2194211396:9009:66
Get Person time: 3 1711330834:9009:66
Get Person time: 2 2264849689:9009:66
Get Person time: 3 1819027970:9009:66
Get Person time: 2 1978614851:9009:66
Get Person time: 1 1863483129:9009:66

Re: Disconnecting two data centers

[Chuck Reynolds]

I was hoping I could do the following

· Change seeds

· Change the topology back to simply

· Stop nodes in datacenter 2

· Remove nodes in datacenter 2

· Restart nodes in datacenter 2

Somehow Cassandra holds on to the information about who was in the cluster.

What if I also changed the cluster name in the Cassandra.yaml before restarting?

Is the data bound to the cluster name?

From: Ryan Svihla <r...@foundev.pro>
Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
Date: Wednesday, March 8, 2017 at 7:38 AM
To: user <user@cassandra.apache.org>
Subject: Re: Disconnecting two data centers

it's a bit tricky and I don't advise it, but the typical pattern is (say you 
have DC1 and DC2):

1. partition the data centers from one another..kill the routing however you 
can (firewall, etc)
2. while partitioned log onto DC1 alter schema so that DC2 is not replicating), 
repeat for other.
2a. If using propertyfilesnitch remove the DC2 from all the DC1 property files 
and vice versa
2b. change the seeds setting in the cassandra.yaml accordingly (DC1 yaml's 
shouldn't have any seeds from DC2, etc)
3. rolling restart to account for this.
4,. run repair (not even sure how necessary this step is, but after doing RF 
changes I do this to prevent hiccups)

I've done this a couple of times but really failing all of that, the more well 
supported and harder to mess up but more work approach is:

1. Set DC2 to RF 0
2. remove all nodes from DC2
3. change yamls for seed files (update property file if need be)
4. create new cluster in DC2,
5. use sstableloader to stream DC1 data to DC2.

On Wed, Mar 8, 2017 at 8:13 AM, Chuck Reynolds 
<creyno...@ancestry.com<mailto:creyno...@ancestry.com>> wrote:
I’m running C* 2.1.13 and I have two rings that are replicating data from our 
data center to one in AWS.

We would like to keep both of them for a while but we have a need to disconnect 
them.  How can this be done?



--

Thanks,
Ryan Svihla

Disconnecting two data centers

[Chuck Reynolds]

I’m running C* 2.1.13 and I have two rings that are replicating data from our 
data center to one in AWS.

We would like to keep both of them for a while but we have a need to disconnect 
them.  How can this be done?