Re: Node replacement strategy with AWS EBS

[Hannu Kröger]

Hi,

So, if it works, great.

auto_bootstrap false is not needed when you have system keyspace as also
mentioned in the article. Now you are likely to have different tokens then
the previous node (unless those were manually configured to match the old
node) and repair and cleanup are needed to get that node to “right” state.
But if the tokens were configured ok then repair & cleanup are not needed.

Cheers,
Hannu

On 14 June 2017 at 13:37:29, Rutvij Bhatt (rut...@sense.com) wrote:

Thanks again for your help! To summarize for anyone who stumbles onto this
in the future, this article covers the procedure well:
https://www.eventbrite.com/engineering/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/

It is more or less what Hannu suggested.

I carried out the following steps:
1. safely stop the cassandra instance (nodetool drain + service cassandra
stop)
2. Shut down the ec2 instance.
3. detach the storage volume from old instance.
4. attach to new instance.
5. point cassandra configuration on new instance to this drive and set
auto_bootstrap: false
6. start cassandra on new instance. Once it has established connection with
peers, you will notice that it takes over the token ranges on its own.
Doing a select on the system.peers table will show that the old node is
gone.
7. Run nodetool repair if need be.

On Tue, Jun 13, 2017 at 1:01 PM Rutvij Bhatt  wrote:

> Nevermind, I misunderstood the first link. In this case, the replacement
> would just be leaving the listen_address as is (to
> InetAddress.getLocalHost()) and just start the new instance up as you
> pointed out in your original answer Hannu.
>
> Thanks.
>
> On Tue, Jun 13, 2017 at 12:35 PM Rutvij Bhatt  wrote:
>
>> Hannu/Nitan,
>>
>> Thanks for your help so far! From what you said in your first response, I
>> can get away with just attaching the EBS volume to Cassandra and starting
>> it with the old node's private IP as my listen_address because it will take
>> over the token assignment from the old node using the data files? With
>> regards to "Cassandra automatically realizes that have just effectively
>> changed IP address.", it says in the first link to change this manually to
>> the desired address - does this not apply in my case if I'm replacing the
>> old node?
>>
>> As for the plan I outlined earlier, is this more for DR scenarios where I
>> have lost a node due to hardware failure and I need to recover the data in
>> a safe manner by requesting a stream from the other replicas?  Am I
>> understanding this right?
>>
>>
>> On Tue, Jun 13, 2017 at 11:59 AM Hannu Kröger  wrote:
>>
>>> Hello,
>>>
>>> So the local information about tokens is stored in the system keyspace.
>>> Also the host id and all that.
>>>
>>> Also documented here:
>>>
>>> https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE
>>>
>>> If for any reason that causes issues, you can also check this:
>>> https://issues.apache.org/jira/browse/CASSANDRA-8382
>>>
>>> If you copy all cassandra data, you are on the safe side. Good point in
>>> the links is that if you have IP addresses in topolgy or other files, then
>>> update those as well.
>>>
>>> Hannu
>>>
>>> On 13 June 2017 at 11:53:13, Nitan Kainth (ni...@bamlabs.com) wrote:
>>>
>>> Hannu,
>>>
>>> "Cassandra automatically realizes that have just effectively changed IP
>>> address” —> are you sure C* will take care of IP change as is? How will it
>>> know which token range to be assigned to this new IP address?
>>>
>>> On Jun 13, 2017, at 10:51 AM, Hannu Kröger  wrote:
>>>
>>> Cassandra automatically realizes that have just effectively changed IP
>>> address
>>>
>>>
>>>

Re: Node replacement strategy with AWS EBS

[Rutvij Bhatt]

Thanks again for your help! To summarize for anyone who stumbles onto this
in the future, this article covers the procedure well:
https://www.eventbrite.com/engineering/changing-the-ip-address-of-a-cassandra-node-with-auto_bootstrapfalse/

It is more or less what Hannu suggested.

I carried out the following steps:
1. safely stop the cassandra instance (nodetool drain + service cassandra
stop)
2. Shut down the ec2 instance.
3. detach the storage volume from old instance.
4. attach to new instance.
5. point cassandra configuration on new instance to this drive and set
auto_bootstrap: false
6. start cassandra on new instance. Once it has established connection with
peers, you will notice that it takes over the token ranges on its own.
Doing a select on the system.peers table will show that the old node is
gone.
7. Run nodetool repair if need be.

On Tue, Jun 13, 2017 at 1:01 PM Rutvij Bhatt  wrote:

> Nevermind, I misunderstood the first link. In this case, the replacement
> would just be leaving the listen_address as is (to
> InetAddress.getLocalHost()) and just start the new instance up as you
> pointed out in your original answer Hannu.
>
> Thanks.
>
> On Tue, Jun 13, 2017 at 12:35 PM Rutvij Bhatt  wrote:
>
>> Hannu/Nitan,
>>
>> Thanks for your help so far! From what you said in your first response, I
>> can get away with just attaching the EBS volume to Cassandra and starting
>> it with the old node's private IP as my listen_address because it will take
>> over the token assignment from the old node using the data files? With
>> regards to "Cassandra automatically realizes that have just effectively
>> changed IP address.", it says in the first link to change this manually to
>> the desired address - does this not apply in my case if I'm replacing the
>> old node?
>>
>> As for the plan I outlined earlier, is this more for DR scenarios where I
>> have lost a node due to hardware failure and I need to recover the data in
>> a safe manner by requesting a stream from the other replicas?  Am I
>> understanding this right?
>>
>>
>> On Tue, Jun 13, 2017 at 11:59 AM Hannu Kröger  wrote:
>>
>>> Hello,
>>>
>>> So the local information about tokens is stored in the system keyspace.
>>> Also the host id and all that.
>>>
>>> Also documented here:
>>>
>>> https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE
>>>
>>> If for any reason that causes issues, you can also check this:
>>> https://issues.apache.org/jira/browse/CASSANDRA-8382
>>>
>>> If you copy all cassandra data, you are on the safe side. Good point in
>>> the links is that if you have IP addresses in topolgy or other files, then
>>> update those as well.
>>>
>>> Hannu
>>>
>>> On 13 June 2017 at 11:53:13, Nitan Kainth (ni...@bamlabs.com) wrote:
>>>
>>> Hannu,
>>>
>>> "Cassandra automatically realizes that have just effectively changed IP
>>> address” —> are you sure C* will take care of IP change as is? How will it
>>> know which token range to be assigned to this new IP address?
>>>
>>> On Jun 13, 2017, at 10:51 AM, Hannu Kröger  wrote:
>>>
>>> Cassandra automatically realizes that have just effectively changed IP
>>> address
>>>
>>>
>>>

Re: Node replacement strategy with AWS EBS

[John Hughes]

In aws, I just grow the cluster 2x, then shrink away the old nodes via
decommission. Mind you I am not dealing with TBs of data, just hundreds of
gigs. Also, I have deployment automated with Cloud Formation and Priam.
YMMV.

On Tue, Jun 13, 2017 at 2:22 PM Cogumelos Maravilha <
cogumelosmaravi...@sapo.pt> wrote:

> Simplest way of all, if you are using RF>=2 simple terminate the old
> instance and create a new one.
> Cheers.
>
>
> On 13-06-2017 18:01, Rutvij Bhatt wrote:
>
> Nevermind, I misunderstood the first link. In this case, the replacement
> would just be leaving the listen_address as is (to
> InetAddress.getLocalHost()) and just start the new instance up as you
> pointed out in your original answer Hannu.
>
> Thanks.
>
> On Tue, Jun 13, 2017 at 12:35 PM Rutvij Bhatt  wrote:
>
>> Hannu/Nitan,
>>
>> Thanks for your help so far! From what you said in your first response, I
>> can get away with just attaching the EBS volume to Cassandra and starting
>> it with the old node's private IP as my listen_address because it will take
>> over the token assignment from the old node using the data files? With
>> regards to "Cassandra automatically realizes that have just effectively
>> changed IP address.", it says in the first link to change this manually to
>> the desired address - does this not apply in my case if I'm replacing the
>> old node?
>>
>> As for the plan I outlined earlier, is this more for DR scenarios where I
>> have lost a node due to hardware failure and I need to recover the data in
>> a safe manner by requesting a stream from the other replicas?  Am I
>> understanding this right?
>>
>>
>> On Tue, Jun 13, 2017 at 11:59 AM Hannu Kröger  wrote:
>>
>>> Hello,
>>>
>>> So the local information about tokens is stored in the system keyspace.
>>> Also the host id and all that.
>>>
>>> Also documented here:
>>>
>>> https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE
>>>
>>> If for any reason that causes issues, you can also check this:
>>> https://issues.apache.org/jira/browse/CASSANDRA-8382
>>>
>>> If you copy all cassandra data, you are on the safe side. Good point in
>>> the links is that if you have IP addresses in topolgy or other files, then
>>> update those as well.
>>>
>>> Hannu
>>>
>>> On 13 June 2017 at 11:53:13, Nitan Kainth (ni...@bamlabs.com) wrote:
>>>
>>> Hannu,
>>>
>>> "Cassandra automatically realizes that have just effectively changed IP
>>> address” —> are you sure C* will take care of IP change as is? How will it
>>> know which token range to be assigned to this new IP address?
>>>
>>> On Jun 13, 2017, at 10:51 AM, Hannu Kröger  wrote:
>>>
>>> Cassandra automatically realizes that have just effectively changed IP
>>> address
>>>
>>>
>>>
>

Re: Node replacement strategy with AWS EBS

[Cogumelos Maravilha]

Simplest way of all, if you are using RF>=2 simple terminate the old
instance and create a new one.

Cheers.

On 13-06-2017 18:01, Rutvij Bhatt wrote:
> Nevermind, I misunderstood the first link. In this case, the
> replacement would just be leaving the listen_address as is (to
> InetAddress.getLocalHost()) and just start the new instance up as you
> pointed out in your original answer Hannu.
>
> Thanks.
>
> On Tue, Jun 13, 2017 at 12:35 PM Rutvij Bhatt  > wrote:
>
> Hannu/Nitan,
>
> Thanks for your help so far! From what you said in your first
> response, I can get away with just attaching the EBS volume to
> Cassandra and starting it with the old node's private IP as my
> listen_address because it will take over the token assignment from
> the old node using the data files? With regards to "Cassandra
> automatically realizes that have just effectively changed IP
> address.", it says in the first link to change this manually to
> the desired address - does this not apply in my case if I'm
> replacing the old node?
>
> As for the plan I outlined earlier, is this more for DR scenarios
> where I have lost a node due to hardware failure and I need to
> recover the data in a safe manner by requesting a stream from the
> other replicas?  Am I understanding this right?
>
>
> On Tue, Jun 13, 2017 at 11:59 AM Hannu Kröger  > wrote:
>
> Hello,
>
> So the local information about tokens is stored in the system
> keyspace. Also the host id and all that.
>
> Also documented here:
> 
> https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE
>
> If for any reason that causes issues, you can also check this:
> https://issues.apache.org/jira/browse/CASSANDRA-8382
>
> If you copy all cassandra data, you are on the safe side. Good
> point in the links is that if you have IP addresses in topolgy
> or other files, then update those as well.
>
> Hannu
>
> On 13 June 2017 at 11:53:13, Nitan Kainth (ni...@bamlabs.com
> ) wrote:
>
>> Hannu, 
>>
>> "Cassandra automatically realizes that have just effectively
>> changed IP address” —> are you sure C* will take care of IP
>> change as is? How will it know which token range to be
>> assigned to this new IP address?
>>
>>> On Jun 13, 2017, at 10:51 AM, Hannu Kröger
>>> mailto:hkro...@gmail.com>> wrote:
>>>
>>> Cassandra automatically realizes that have just effectively
>>> changed IP address
>>

Re: Node replacement strategy with AWS EBS

[Rutvij Bhatt]

Nevermind, I misunderstood the first link. In this case, the replacement
would just be leaving the listen_address as is (to
InetAddress.getLocalHost()) and just start the new instance up as you
pointed out in your original answer Hannu.

Thanks.

On Tue, Jun 13, 2017 at 12:35 PM Rutvij Bhatt  wrote:

> Hannu/Nitan,
>
> Thanks for your help so far! From what you said in your first response, I
> can get away with just attaching the EBS volume to Cassandra and starting
> it with the old node's private IP as my listen_address because it will take
> over the token assignment from the old node using the data files? With
> regards to "Cassandra automatically realizes that have just effectively
> changed IP address.", it says in the first link to change this manually to
> the desired address - does this not apply in my case if I'm replacing the
> old node?
>
> As for the plan I outlined earlier, is this more for DR scenarios where I
> have lost a node due to hardware failure and I need to recover the data in
> a safe manner by requesting a stream from the other replicas?  Am I
> understanding this right?
>
>
> On Tue, Jun 13, 2017 at 11:59 AM Hannu Kröger  wrote:
>
>> Hello,
>>
>> So the local information about tokens is stored in the system keyspace.
>> Also the host id and all that.
>>
>> Also documented here:
>>
>> https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE
>>
>> If for any reason that causes issues, you can also check this:
>> https://issues.apache.org/jira/browse/CASSANDRA-8382
>>
>> If you copy all cassandra data, you are on the safe side. Good point in
>> the links is that if you have IP addresses in topolgy or other files, then
>> update those as well.
>>
>> Hannu
>>
>> On 13 June 2017 at 11:53:13, Nitan Kainth (ni...@bamlabs.com) wrote:
>>
>> Hannu,
>>
>> "Cassandra automatically realizes that have just effectively changed IP
>> address” —> are you sure C* will take care of IP change as is? How will it
>> know which token range to be assigned to this new IP address?
>>
>> On Jun 13, 2017, at 10:51 AM, Hannu Kröger  wrote:
>>
>> Cassandra automatically realizes that have just effectively changed IP
>> address
>>
>>
>>

Re: Node replacement strategy with AWS EBS

[Rutvij Bhatt]

Hannu/Nitan,

Thanks for your help so far! From what you said in your first response, I
can get away with just attaching the EBS volume to Cassandra and starting
it with the old node's private IP as my listen_address because it will take
over the token assignment from the old node using the data files? With
regards to "Cassandra automatically realizes that have just effectively
changed IP address.", it says in the first link to change this manually to
the desired address - does this not apply in my case if I'm replacing the
old node?

As for the plan I outlined earlier, is this more for DR scenarios where I
have lost a node due to hardware failure and I need to recover the data in
a safe manner by requesting a stream from the other replicas?  Am I
understanding this right?


On Tue, Jun 13, 2017 at 11:59 AM Hannu Kröger  wrote:

> Hello,
>
> So the local information about tokens is stored in the system keyspace.
> Also the host id and all that.
>
> Also documented here:
>
> https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE
>
> If for any reason that causes issues, you can also check this:
> https://issues.apache.org/jira/browse/CASSANDRA-8382
>
> If you copy all cassandra data, you are on the safe side. Good point in
> the links is that if you have IP addresses in topolgy or other files, then
> update those as well.
>
> Hannu
>
> On 13 June 2017 at 11:53:13, Nitan Kainth (ni...@bamlabs.com) wrote:
>
> Hannu,
>
> "Cassandra automatically realizes that have just effectively changed IP
> address” —> are you sure C* will take care of IP change as is? How will it
> know which token range to be assigned to this new IP address?
>
> On Jun 13, 2017, at 10:51 AM, Hannu Kröger  wrote:
>
> Cassandra automatically realizes that have just effectively changed IP
> address
>
>
>

Re: Node replacement strategy with AWS EBS

[Nitan Kainth]

Thank you Hannu.


> On Jun 13, 2017, at 10:59 AM, Hannu Kröger  wrote:
> 
> Hello,
> 
> So the local information about tokens is stored in the system keyspace. Also 
> the host id and all that.
> 
> Also documented here:
> https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE
>  
> 
> 
> If for any reason that causes issues, you can also check this:
> https://issues.apache.org/jira/browse/CASSANDRA-8382 
> 
> 
> If you copy all cassandra data, you are on the safe side. Good point in the 
> links is that if you have IP addresses in topolgy or other files, then update 
> those as well.
> 
> Hannu
> 
> On 13 June 2017 at 11:53:13, Nitan Kainth (ni...@bamlabs.com 
> ) wrote:
> 
>> Hannu, 
>> 
>> "Cassandra automatically realizes that have just effectively changed IP 
>> address” —> are you sure C* will take care of IP change as is? How will it 
>> know which token range to be assigned to this new IP address?
>> 
>>> On Jun 13, 2017, at 10:51 AM, Hannu Kröger >> > wrote:
>>> 
>>> Cassandra automatically realizes that have just effectively changed IP 
>>> address

Re: Node replacement strategy with AWS EBS

[Hannu Kröger]

Hello,

So the local information about tokens is stored in the system keyspace.
Also the host id and all that.

Also documented here:
https://support.datastax.com/hc/en-us/articles/204289959-Changing-IP-addresses-in-DSE

If for any reason that causes issues, you can also check this:
https://issues.apache.org/jira/browse/CASSANDRA-8382

If you copy all cassandra data, you are on the safe side. Good point in the
links is that if you have IP addresses in topolgy or other files, then
update those as well.

Hannu

On 13 June 2017 at 11:53:13, Nitan Kainth (ni...@bamlabs.com) wrote:

Hannu,

"Cassandra automatically realizes that have just effectively changed IP
address” —> are you sure C* will take care of IP change as is? How will it
know which token range to be assigned to this new IP address?

On Jun 13, 2017, at 10:51 AM, Hannu Kröger  wrote:

Cassandra automatically realizes that have just effectively changed IP
address

Re: Node replacement strategy with AWS EBS

[Nitan Kainth]

Hannu, 

"Cassandra automatically realizes that have just effectively changed IP 
address” —> are you sure C* will take care of IP change as is? How will it know 
which token range to be assigned to this new IP address?

> On Jun 13, 2017, at 10:51 AM, Hannu Kröger  wrote:
> 
> Cassandra automatically realizes that have just effectively changed IP address

Re: Node replacement strategy with AWS EBS

[Hannu Kröger]

Hello,

I think that’s not the optimal way to handle it.

If you are just attaching the same EBS volume to a new node you can do like
this:
1) nodetool drain on old
2) stop cassandra on old
3) Attach EBS to new node
4) Start Cassandra on new node

Cassandra automatically realizes that have just effectively changed IP
address.

replace_address will also stream all the data, so that’s inefficient way to
do it if you already have all the data.

Hannu

On 13 June 2017 at 11:23:56, Rutvij Bhatt (rut...@sense.com) wrote:

Hi!

We're running a Cassandra cluster on AWS. I want to replace an old node
with EBS storage with a new one. The steps I'm following are as follows and
I want to get a second opinion on whether this is the right thing to do:

1. Remove old node from gossip.
2. Run nodetool drain
3. Stop cassandra
4. Create new new node and update JVM_OPTS in cassandra-env.sh with
cassandra.replace_address= as instructed
here -
http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsReplaceNode.html
5. Attach the EBS volume from the old node at the same mount point.
6. Start cassandra on the new node.
7. Run nodetool repair to catch the replacing node up on whatever it has
missed.

Thanks!

Re: Node replacement strategy with AWS EBS

[Rutvij Bhatt]

Nitan,

Yes, that is what I've done. I snapshotted the volume after step 3 and will
create a new volume from that snapshot and attach it to the new instance.
Curious if I am indeed replacing a node completely, is there any logical
difference between snapshot->create->attach vs detach from old->attach to
new besides a margin of safety?

Thanks for your reply!

On Tue, Jun 13, 2017 at 11:37 AM Nitan Kainth  wrote:

> Steps are good Rutvij. Step 1 is not mandatory.
>
> We snapshot EBS volume and then restored on new node. How are you
> re-attaching EBS volume without snapshot?
>
>
> I
>
> On Jun 13, 2017, at 10:21 AM, Rutvij Bhatt  wrote:
>
> Hi!
>
> We're running a Cassandra cluster on AWS. I want to replace an old node
> with EBS storage with a new one. The steps I'm following are as follows and
> I want to get a second opinion on whether this is the right thing to do:
>
> 1. Remove old node from gossip.
> 2. Run nodetool drain
> 3. Stop cassandra
> 4. Create new new node and update JVM_OPTS in cassandra-env.sh with
> cassandra.replace_address= as instructed
> here -
> http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsReplaceNode.html
> 5. Attach the EBS volume from the old node at the same mount point.
> 6. Start cassandra on the new node.
> 7. Run nodetool repair to catch the replacing node up on whatever it has
> missed.
>
> Thanks!
>
>
>

Re: Node replacement strategy with AWS EBS

[Nitan Kainth]

Steps are good Rutvij. Step 1 is not mandatory. 

We snapshot EBS volume and then restored on new node. How are you re-attaching 
EBS volume without snapshot?


I
> On Jun 13, 2017, at 10:21 AM, Rutvij Bhatt  wrote:
> 
> Hi!
> 
> We're running a Cassandra cluster on AWS. I want to replace an old node with 
> EBS storage with a new one. The steps I'm following are as follows and I want 
> to get a second opinion on whether this is the right thing to do:
> 
> 1. Remove old node from gossip.
> 2. Run nodetool drain
> 3. Stop cassandra
> 4. Create new new node and update JVM_OPTS in cassandra-env.sh with 
> cassandra.replace_address= as instructed here 
> - 
> http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsReplaceNode.html
>  
> 
> 5. Attach the EBS volume from the old node at the same mount point.
> 6. Start cassandra on the new node.
> 7. Run nodetool repair to catch the replacing node up on whatever it has 
> missed.
> 
> Thanks!

Node replacement strategy with AWS EBS

[Rutvij Bhatt]

Hi!

We're running a Cassandra cluster on AWS. I want to replace an old node
with EBS storage with a new one. The steps I'm following are as follows and
I want to get a second opinion on whether this is the right thing to do:

1. Remove old node from gossip.
2. Run nodetool drain
3. Stop cassandra
4. Create new new node and update JVM_OPTS in cassandra-env.sh with
cassandra.replace_address= as instructed
here -
http://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsReplaceNode.html
5. Attach the EBS volume from the old node at the same mount point.
6. Start cassandra on the new node.
7. Run nodetool repair to catch the replacing node up on whatever it has
missed.

Thanks!