Re: C* in multiple AWS AZ's

[Randy Lynn]

Thanks Alain,

Wanted to just circle back on all the above..
Thanks everyone for your help, and input.

I'm glad to hear someone else did a site-to-site tunnel with Cassandra
between regions. When originally setting up all the docs and information
all preached public IP's. I can totally understand that it's probably a
throughput / latency problem that the tunnel introduces, but so far it
seems to be working well.  The tunnel is stable too, which originally I was
concerned with hiccups and maintenance on the tunnel.

I did go with GPFS. The move went great. The new i3 nodes are pretty
awesome - especially the NVMe!
I did go with ubuntu 16.04 LTS versus the AWS Linux AMI. So far no
stability problems with the NVMe or the i3 in general with the Ubuntu 16.04.

On Tue, Jul 3, 2018 at 7:12 AM, Alain RODRIGUEZ  wrote:

> Hi,
>
> Nice that you solved the issue. I had some thoughts while reading:
>
>
>> My original thought was a new DC parallel to the current, and then
>> decommission the other DC.
>>
>
> I also think this is the best way to go when possible. It can be reverted
> any time in the process, respect distributions, you can switch app per app,
> and some other advantages.
>
> Also, I am a bit late on this topic, but using this:
>
> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel..
>> I'm wanting to move the current US-EAST from AZ 1a to 1e.
>
>
> You could have kept Ec2Snitch and use the 'dc_suffix' option in 
> 'cassandra-rackdc.properties'
> file to create a non-conflicting logical datacenter in the same region. For
> example with a 'dc_suffix' set to '-WHATEVER' the new data center created
> would have been called 'US-EAST-WHATEVER' :-).
>
> I know all docs say use ec2multiregion for multi-DC.
>>
>
> Using EC2RegionSnitch brought nothing but troubles to me in the past. You
> have to open security groups for both public and private IPs, the number of
> rules is limited (or was for me when I did it) and after some headaches, we
> did the same, a tunnel between VPCs and used ec2_snitch.
>
> Yet, now I would keep GPFS if that works fully for you now. It will even
> allow you to hybrid the cluster with non AWS hardware or transition out of
> AWS if you need at some point in the future.
>
> As a side note, using i3, you might have to use a recent operating system
> (such as Ubuntu 16.04) to have the latest drivers for NVMe. NVMe support in
> Ubuntu 14.04 AMI is not reliable. It might be absent or lead to data
> corruption. Make sure the OS in use works well with this hardware.
>
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-
> instance-store.html
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
> 2018-06-29 16:35 GMT+01:00 Pradeep Chhetri :
>
>> Ohh i see now. It makes sense. Thanks a lot.
>>
>> On Fri, Jun 29, 2018 at 9:17 PM, Randy Lynn  wrote:
>>
>>> data is only lost if you stop the node. between restarts the storage is
>>> fine.
>>>
>>> On Fri, Jun 29, 2018 at 10:39 AM, Pradeep Chhetri >> > wrote:
>>>
 Isnt NVMe storage an instance storage ie. the data will be lost in case
 the instance restarts. How are you going to make sure that there is no data
 loss in case instance gets rebooted?

 On Fri, 29 Jun 2018 at 7:00 PM, Randy Lynn  wrote:

> GPFS - Rahul FTW! Thank you for your help!
>
> Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did
> not have the benefit of doing benchmarks.. but we're moving from 1,500 
> IOPS
> so I intrinsically know we'll get better throughput.
>
> On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh <
> rahul.xavier.si...@gmail.com> wrote:
>
>> Totally agree. GPFS for the win. EC2 multi region snitch is an
>> automation tool like Ansible or Puppet. Unless you have two orders of
>> magnitude more servers than you do now, you don’t need it.
>>
>> Rahul
>> On Jun 29, 2018, 6:18 AM -0400, kurt greaves ,
>> wrote:
>>
>> Yes. You would just end up with a rack named differently to the AZ.
>> This is not a problem as racks are just logical. I would recommend
>> migrating all your DCs to GPFS though for consistency.
>>
>> On Fri., 29 Jun. 2018, 09:04 Randy Lynn,  wrote:
>>
>>> So we have two data centers already running..
>>>
>>> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site
>>> tunnel.. I'm wanting to move the current US-EAST from AZ 1a to 1e..
>>> I know all docs say use ec2multiregion for multi-DC.
>>>
>>> I like the GPFS idea. would that work with the multi-DC too?
>>> What's the downside? status would report rack of 1a, even though in
>>> 1e?
>>>
>>> Thanks in advance for the help/thoughts!!
>>>
>>>
>>> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves 
>>> wrote:
>>>
 There is a need 

Re: C* in multiple AWS AZ's

[Alain RODRIGUEZ]

Hi,

Nice that you solved the issue. I had some thoughts while reading:


> My original thought was a new DC parallel to the current, and then
> decommission the other DC.
>

I also think this is the best way to go when possible. It can be reverted
any time in the process, respect distributions, you can switch app per app,
and some other advantages.

Also, I am a bit late on this topic, but using this:

AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel..
> I'm wanting to move the current US-EAST from AZ 1a to 1e.


You could have kept Ec2Snitch and use the 'dc_suffix' option in
'cassandra-rackdc.properties'
file to create a non-conflicting logical datacenter in the same region. For
example with a 'dc_suffix' set to '-WHATEVER' the new data center created
would have been called 'US-EAST-WHATEVER' :-).

I know all docs say use ec2multiregion for multi-DC.
>

Using EC2RegionSnitch brought nothing but troubles to me in the past. You
have to open security groups for both public and private IPs, the number of
rules is limited (or was for me when I did it) and after some headaches, we
did the same, a tunnel between VPCs and used ec2_snitch.

Yet, now I would keep GPFS if that works fully for you now. It will even
allow you to hybrid the cluster with non AWS hardware or transition out of
AWS if you need at some point in the future.

As a side note, using i3, you might have to use a recent operating system
(such as Ubuntu 16.04) to have the latest drivers for NVMe. NVMe support in
Ubuntu 14.04 AMI is not reliable. It might be absent or lead to data
corruption. Make sure the OS in use works well with this hardware.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ssd-instance-store.html

C*heers,
---
Alain Rodriguez - @arodream - al...@thelastpickle.com
France / Spain

The Last Pickle - Apache Cassandra Consulting
http://www.thelastpickle.com

2018-06-29 16:35 GMT+01:00 Pradeep Chhetri :

> Ohh i see now. It makes sense. Thanks a lot.
>
> On Fri, Jun 29, 2018 at 9:17 PM, Randy Lynn  wrote:
>
>> data is only lost if you stop the node. between restarts the storage is
>> fine.
>>
>> On Fri, Jun 29, 2018 at 10:39 AM, Pradeep Chhetri 
>> wrote:
>>
>>> Isnt NVMe storage an instance storage ie. the data will be lost in case
>>> the instance restarts. How are you going to make sure that there is no data
>>> loss in case instance gets rebooted?
>>>
>>> On Fri, 29 Jun 2018 at 7:00 PM, Randy Lynn  wrote:
>>>
 GPFS - Rahul FTW! Thank you for your help!

 Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did
 not have the benefit of doing benchmarks.. but we're moving from 1,500 IOPS
 so I intrinsically know we'll get better throughput.

 On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh <
 rahul.xavier.si...@gmail.com> wrote:

> Totally agree. GPFS for the win. EC2 multi region snitch is an
> automation tool like Ansible or Puppet. Unless you have two orders of
> magnitude more servers than you do now, you don’t need it.
>
> Rahul
> On Jun 29, 2018, 6:18 AM -0400, kurt greaves ,
> wrote:
>
> Yes. You would just end up with a rack named differently to the AZ.
> This is not a problem as racks are just logical. I would recommend
> migrating all your DCs to GPFS though for consistency.
>
> On Fri., 29 Jun. 2018, 09:04 Randy Lynn,  wrote:
>
>> So we have two data centers already running..
>>
>> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site
>> tunnel.. I'm wanting to move the current US-EAST from AZ 1a to 1e..
>> I know all docs say use ec2multiregion for multi-DC.
>>
>> I like the GPFS idea. would that work with the multi-DC too?
>> What's the downside? status would report rack of 1a, even though in
>> 1e?
>>
>> Thanks in advance for the help/thoughts!!
>>
>>
>> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves 
>> wrote:
>>
>>> There is a need for a repair with both DCs as rebuild will not
>>> stream all replicas, so unless you can guarantee you were perfectly
>>> consistent at time of rebuild you'll want to do a repair after rebuild.
>>>
>>> On another note you could just replace the nodes but use GPFS
>>> instead of EC2 snitch, using the same rack name.
>>>
>>> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, <
>>> rahul.xavier.si...@gmail.com> wrote:
>>>
 Parallel load is the best approach and then switch your Data access
 code to only access the new hardware. After you verify that there are 
 no
 local read / writes on the OLD dc and that the updates are only via 
 Gossip,
 then go ahead and change the replication factor on the key space to 
 have
 zero replicas in the old DC. Then you can decommissioned.

 This way you are hundred percent sure that you aren’t missing any
 new data. 

Re: C* in multiple AWS AZ's

[Pradeep Chhetri]

Ohh i see now. It makes sense. Thanks a lot.

On Fri, Jun 29, 2018 at 9:17 PM, Randy Lynn  wrote:

> data is only lost if you stop the node. between restarts the storage is
> fine.
>
> On Fri, Jun 29, 2018 at 10:39 AM, Pradeep Chhetri 
> wrote:
>
>> Isnt NVMe storage an instance storage ie. the data will be lost in case
>> the instance restarts. How are you going to make sure that there is no data
>> loss in case instance gets rebooted?
>>
>> On Fri, 29 Jun 2018 at 7:00 PM, Randy Lynn  wrote:
>>
>>> GPFS - Rahul FTW! Thank you for your help!
>>>
>>> Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did
>>> not have the benefit of doing benchmarks.. but we're moving from 1,500 IOPS
>>> so I intrinsically know we'll get better throughput.
>>>
>>> On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh <
>>> rahul.xavier.si...@gmail.com> wrote:
>>>
 Totally agree. GPFS for the win. EC2 multi region snitch is an
 automation tool like Ansible or Puppet. Unless you have two orders of
 magnitude more servers than you do now, you don’t need it.

 Rahul
 On Jun 29, 2018, 6:18 AM -0400, kurt greaves ,
 wrote:

 Yes. You would just end up with a rack named differently to the AZ.
 This is not a problem as racks are just logical. I would recommend
 migrating all your DCs to GPFS though for consistency.

 On Fri., 29 Jun. 2018, 09:04 Randy Lynn,  wrote:

> So we have two data centers already running..
>
> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site
> tunnel.. I'm wanting to move the current US-EAST from AZ 1a to 1e..
> I know all docs say use ec2multiregion for multi-DC.
>
> I like the GPFS idea. would that work with the multi-DC too?
> What's the downside? status would report rack of 1a, even though in 1e?
>
> Thanks in advance for the help/thoughts!!
>
>
> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves 
> wrote:
>
>> There is a need for a repair with both DCs as rebuild will not stream
>> all replicas, so unless you can guarantee you were perfectly consistent 
>> at
>> time of rebuild you'll want to do a repair after rebuild.
>>
>> On another note you could just replace the nodes but use GPFS instead
>> of EC2 snitch, using the same rack name.
>>
>> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, <
>> rahul.xavier.si...@gmail.com> wrote:
>>
>>> Parallel load is the best approach and then switch your Data access
>>> code to only access the new hardware. After you verify that there are no
>>> local read / writes on the OLD dc and that the updates are only via 
>>> Gossip,
>>> then go ahead and change the replication factor on the key space to have
>>> zero replicas in the old DC. Then you can decommissioned.
>>>
>>> This way you are hundred percent sure that you aren’t missing any
>>> new data. No need for a DC to DC repair but a repair is always healthy.
>>>
>>> Rahul
>>> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn ,
>>> wrote:
>>>
>>> Already running with Ec2.
>>>
>>> My original thought was a new DC parallel to the current, and then
>>> decommission the other DC.
>>>
>>> Also my data load is small right now.. I know small is relative
>>> term.. each node is carrying about 6GB..
>>>
>>> So given the data size, would you go with parallel DC or let the new
>>> AZ carry a heavy load until the others are migrated over?
>>> and then I think "repair" to cleanup the replications?
>>>
>>>
>>> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh <
>>> rahul.xavier.si...@gmail.com> wrote:
>>>
 You don’t have to use EC2 snitch on AWS but if you have already
 started with it , it may put a node in a different DC.

 If your data density won’t be ridiculous You could add 3 to
 different DC/ Region and then sync up. After the new DC is operational 
 you
 can remove one at a time on the old DC and at the same time add to the 
 new
 one.

 Rahul
 On Jun 28, 2018, 9:03 AM -0500, Randy Lynn ,
 wrote:

 I have a 6-node cluster I'm migrating to the new i3 types.
 But at the same time I want to migrate to a different AZ.

 What happens if I do the "running node replace method" with 1 node
 at a time moving to the new AZ. Meaning, I'll have temporarily;

 5 nodes in AZ 1c
 1 new node in AZ 1e.

 I'll wash-rinse-repeat till all 6 are on the new machine type and
 in the new AZ.

 Any thoughts about whether this gets weird with the Ec2Snitch and a
 RF 3?

 --
 Randy Lynn
 rl...@getavail.com

 office:
 859.963.1616 <+1-859-963-1616> ext 202
 163 East Main Street - Lexington, 

Re: C* in multiple AWS AZ's

[Randy Lynn]

data is only lost if you stop the node. between restarts the storage is
fine.

On Fri, Jun 29, 2018 at 10:39 AM, Pradeep Chhetri 
wrote:

> Isnt NVMe storage an instance storage ie. the data will be lost in case
> the instance restarts. How are you going to make sure that there is no data
> loss in case instance gets rebooted?
>
> On Fri, 29 Jun 2018 at 7:00 PM, Randy Lynn  wrote:
>
>> GPFS - Rahul FTW! Thank you for your help!
>>
>> Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did
>> not have the benefit of doing benchmarks.. but we're moving from 1,500 IOPS
>> so I intrinsically know we'll get better throughput.
>>
>> On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh <
>> rahul.xavier.si...@gmail.com> wrote:
>>
>>> Totally agree. GPFS for the win. EC2 multi region snitch is an
>>> automation tool like Ansible or Puppet. Unless you have two orders of
>>> magnitude more servers than you do now, you don’t need it.
>>>
>>> Rahul
>>> On Jun 29, 2018, 6:18 AM -0400, kurt greaves ,
>>> wrote:
>>>
>>> Yes. You would just end up with a rack named differently to the AZ. This
>>> is not a problem as racks are just logical. I would recommend migrating all
>>> your DCs to GPFS though for consistency.
>>>
>>> On Fri., 29 Jun. 2018, 09:04 Randy Lynn,  wrote:
>>>
 So we have two data centers already running..

 AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site
 tunnel.. I'm wanting to move the current US-EAST from AZ 1a to 1e..
 I know all docs say use ec2multiregion for multi-DC.

 I like the GPFS idea. would that work with the multi-DC too?
 What's the downside? status would report rack of 1a, even though in 1e?

 Thanks in advance for the help/thoughts!!


 On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves 
 wrote:

> There is a need for a repair with both DCs as rebuild will not stream
> all replicas, so unless you can guarantee you were perfectly consistent at
> time of rebuild you'll want to do a repair after rebuild.
>
> On another note you could just replace the nodes but use GPFS instead
> of EC2 snitch, using the same rack name.
>
> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, <
> rahul.xavier.si...@gmail.com> wrote:
>
>> Parallel load is the best approach and then switch your Data access
>> code to only access the new hardware. After you verify that there are no
>> local read / writes on the OLD dc and that the updates are only via 
>> Gossip,
>> then go ahead and change the replication factor on the key space to have
>> zero replicas in the old DC. Then you can decommissioned.
>>
>> This way you are hundred percent sure that you aren’t missing any new
>> data. No need for a DC to DC repair but a repair is always healthy.
>>
>> Rahul
>> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn ,
>> wrote:
>>
>> Already running with Ec2.
>>
>> My original thought was a new DC parallel to the current, and then
>> decommission the other DC.
>>
>> Also my data load is small right now.. I know small is relative
>> term.. each node is carrying about 6GB..
>>
>> So given the data size, would you go with parallel DC or let the new
>> AZ carry a heavy load until the others are migrated over?
>> and then I think "repair" to cleanup the replications?
>>
>>
>> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh <
>> rahul.xavier.si...@gmail.com> wrote:
>>
>>> You don’t have to use EC2 snitch on AWS but if you have already
>>> started with it , it may put a node in a different DC.
>>>
>>> If your data density won’t be ridiculous You could add 3 to
>>> different DC/ Region and then sync up. After the new DC is operational 
>>> you
>>> can remove one at a time on the old DC and at the same time add to the 
>>> new
>>> one.
>>>
>>> Rahul
>>> On Jun 28, 2018, 9:03 AM -0500, Randy Lynn ,
>>> wrote:
>>>
>>> I have a 6-node cluster I'm migrating to the new i3 types.
>>> But at the same time I want to migrate to a different AZ.
>>>
>>> What happens if I do the "running node replace method" with 1 node
>>> at a time moving to the new AZ. Meaning, I'll have temporarily;
>>>
>>> 5 nodes in AZ 1c
>>> 1 new node in AZ 1e.
>>>
>>> I'll wash-rinse-repeat till all 6 are on the new machine type and in
>>> the new AZ.
>>>
>>> Any thoughts about whether this gets weird with the Ec2Snitch and a
>>> RF 3?
>>>
>>> --
>>> Randy Lynn
>>> rl...@getavail.com
>>>
>>> office:
>>> 859.963.1616 <+1-859-963-1616> ext 202
>>> 163 East Main Street - Lexington, KY 40507 - USA
>>> 
>>>
>>>  getavail.com 
>>>
>>>
>>
>>
>> --

Re: C* in multiple AWS AZ's

[Pradeep Chhetri]

Isnt NVMe storage an instance storage ie. the data will be lost in case the
instance restarts. How are you going to make sure that there is no data
loss in case instance gets rebooted?

On Fri, 29 Jun 2018 at 7:00 PM, Randy Lynn  wrote:

> GPFS - Rahul FTW! Thank you for your help!
>
> Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did not
> have the benefit of doing benchmarks.. but we're moving from 1,500 IOPS so
> I intrinsically know we'll get better throughput.
>
> On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh  > wrote:
>
>> Totally agree. GPFS for the win. EC2 multi region snitch is an automation
>> tool like Ansible or Puppet. Unless you have two orders of magnitude more
>> servers than you do now, you don’t need it.
>>
>> Rahul
>> On Jun 29, 2018, 6:18 AM -0400, kurt greaves ,
>> wrote:
>>
>> Yes. You would just end up with a rack named differently to the AZ. This
>> is not a problem as racks are just logical. I would recommend migrating all
>> your DCs to GPFS though for consistency.
>>
>> On Fri., 29 Jun. 2018, 09:04 Randy Lynn,  wrote:
>>
>>> So we have two data centers already running..
>>>
>>> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site
>>> tunnel.. I'm wanting to move the current US-EAST from AZ 1a to 1e..
>>> I know all docs say use ec2multiregion for multi-DC.
>>>
>>> I like the GPFS idea. would that work with the multi-DC too?
>>> What's the downside? status would report rack of 1a, even though in 1e?
>>>
>>> Thanks in advance for the help/thoughts!!
>>>
>>>
>>> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves 
>>> wrote:
>>>
 There is a need for a repair with both DCs as rebuild will not stream
 all replicas, so unless you can guarantee you were perfectly consistent at
 time of rebuild you'll want to do a repair after rebuild.

 On another note you could just replace the nodes but use GPFS instead
 of EC2 snitch, using the same rack name.

 On Fri., 29 Jun. 2018, 00:19 Rahul Singh, 
 wrote:

> Parallel load is the best approach and then switch your Data access
> code to only access the new hardware. After you verify that there are no
> local read / writes on the OLD dc and that the updates are only via 
> Gossip,
> then go ahead and change the replication factor on the key space to have
> zero replicas in the old DC. Then you can decommissioned.
>
> This way you are hundred percent sure that you aren’t missing any new
> data. No need for a DC to DC repair but a repair is always healthy.
>
> Rahul
> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn ,
> wrote:
>
> Already running with Ec2.
>
> My original thought was a new DC parallel to the current, and then
> decommission the other DC.
>
> Also my data load is small right now.. I know small is relative term..
> each node is carrying about 6GB..
>
> So given the data size, would you go with parallel DC or let the new
> AZ carry a heavy load until the others are migrated over?
> and then I think "repair" to cleanup the replications?
>
>
> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh <
> rahul.xavier.si...@gmail.com> wrote:
>
>> You don’t have to use EC2 snitch on AWS but if you have already
>> started with it , it may put a node in a different DC.
>>
>> If your data density won’t be ridiculous You could add 3 to different
>> DC/ Region and then sync up. After the new DC is operational you can 
>> remove
>> one at a time on the old DC and at the same time add to the new one.
>>
>> Rahul
>> On Jun 28, 2018, 9:03 AM -0500, Randy Lynn ,
>> wrote:
>>
>> I have a 6-node cluster I'm migrating to the new i3 types.
>> But at the same time I want to migrate to a different AZ.
>>
>> What happens if I do the "running node replace method" with 1 node at
>> a time moving to the new AZ. Meaning, I'll have temporarily;
>>
>> 5 nodes in AZ 1c
>> 1 new node in AZ 1e.
>>
>> I'll wash-rinse-repeat till all 6 are on the new machine type and in
>> the new AZ.
>>
>> Any thoughts about whether this gets weird with the Ec2Snitch and a
>> RF 3?
>>
>> --
>> Randy Lynn
>> rl...@getavail.com
>>
>> office:
>> 859.963.1616 <+1-859-963-1616> ext 202
>> 163 East Main Street - Lexington, KY 40507 - USA
>> 
>>
>>  getavail.com 
>>
>>
>
>
> --
> Randy Lynn
> rl...@getavail.com
>
> office:
> 859.963.1616 <+1-859-963-1616> ext 202
> 163 East Main Street - Lexington, KY 40507 - USA
> 
>
>  getavail.com 
>
>

Re: C* in multiple AWS AZ's

[Randy Lynn]

GPFS - Rahul FTW! Thank you for your help!

Yes, Pradeep - migrating to i3 from r3. moving for NVMe storage, I did not
have the benefit of doing benchmarks.. but we're moving from 1,500 IOPS so
I intrinsically know we'll get better throughput.

On Fri, Jun 29, 2018 at 7:21 AM, Rahul Singh 
wrote:

> Totally agree. GPFS for the win. EC2 multi region snitch is an automation
> tool like Ansible or Puppet. Unless you have two orders of magnitude more
> servers than you do now, you don’t need it.
>
> Rahul
> On Jun 29, 2018, 6:18 AM -0400, kurt greaves ,
> wrote:
>
> Yes. You would just end up with a rack named differently to the AZ. This
> is not a problem as racks are just logical. I would recommend migrating all
> your DCs to GPFS though for consistency.
>
> On Fri., 29 Jun. 2018, 09:04 Randy Lynn,  wrote:
>
>> So we have two data centers already running..
>>
>> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel..
>> I'm wanting to move the current US-EAST from AZ 1a to 1e..
>> I know all docs say use ec2multiregion for multi-DC.
>>
>> I like the GPFS idea. would that work with the multi-DC too?
>> What's the downside? status would report rack of 1a, even though in 1e?
>>
>> Thanks in advance for the help/thoughts!!
>>
>>
>> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves 
>> wrote:
>>
>>> There is a need for a repair with both DCs as rebuild will not stream
>>> all replicas, so unless you can guarantee you were perfectly consistent at
>>> time of rebuild you'll want to do a repair after rebuild.
>>>
>>> On another note you could just replace the nodes but use GPFS instead of
>>> EC2 snitch, using the same rack name.
>>>
>>> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, 
>>> wrote:
>>>
 Parallel load is the best approach and then switch your Data access
 code to only access the new hardware. After you verify that there are no
 local read / writes on the OLD dc and that the updates are only via Gossip,
 then go ahead and change the replication factor on the key space to have
 zero replicas in the old DC. Then you can decommissioned.

 This way you are hundred percent sure that you aren’t missing any new
 data. No need for a DC to DC repair but a repair is always healthy.

 Rahul
 On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , wrote:

 Already running with Ec2.

 My original thought was a new DC parallel to the current, and then
 decommission the other DC.

 Also my data load is small right now.. I know small is relative term..
 each node is carrying about 6GB..

 So given the data size, would you go with parallel DC or let the new AZ
 carry a heavy load until the others are migrated over?
 and then I think "repair" to cleanup the replications?


 On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh <
 rahul.xavier.si...@gmail.com> wrote:

> You don’t have to use EC2 snitch on AWS but if you have already
> started with it , it may put a node in a different DC.
>
> If your data density won’t be ridiculous You could add 3 to different
> DC/ Region and then sync up. After the new DC is operational you can 
> remove
> one at a time on the old DC and at the same time add to the new one.
>
> Rahul
> On Jun 28, 2018, 9:03 AM -0500, Randy Lynn ,
> wrote:
>
> I have a 6-node cluster I'm migrating to the new i3 types.
> But at the same time I want to migrate to a different AZ.
>
> What happens if I do the "running node replace method" with 1 node at
> a time moving to the new AZ. Meaning, I'll have temporarily;
>
> 5 nodes in AZ 1c
> 1 new node in AZ 1e.
>
> I'll wash-rinse-repeat till all 6 are on the new machine type and in
> the new AZ.
>
> Any thoughts about whether this gets weird with the Ec2Snitch and a RF
> 3?
>
> --
> Randy Lynn
> rl...@getavail.com
>
> office:
> 859.963.1616 <+1-859-963-1616> ext 202
> 163 East Main Street - Lexington, KY 40507 - USA
> 
>
>  getavail.com 
>
>


 --
 Randy Lynn
 rl...@getavail.com

 office:
 859.963.1616 <+1-859-963-1616> ext 202
 163 East Main Street - Lexington, KY 40507 - USA
 

  getavail.com 


>>
>>
>> --
>> Randy Lynn
>> rl...@getavail.com
>>
>> office:
>> 859.963.1616 <+1-859-963-1616> ext 202
>> 163 East Main Street - Lexington, KY 40507 - USA
>> 
>>
>>  getavail.com 
>>
>


-- 
Randy Lynn
rl...@getavail.com

office:
859.963.1616 

Re: C* in multiple AWS AZ's

[Rahul Singh]

Totally agree. GPFS for the win. EC2 multi region snitch is an automation tool 
like Ansible or Puppet. Unless you have two orders of magnitude more servers 
than you do now, you don’t need it.

Rahul
On Jun 29, 2018, 6:18 AM -0400, kurt greaves , wrote:
> Yes. You would just end up with a rack named differently to the AZ. This is 
> not a problem as racks are just logical. I would recommend migrating all your 
> DCs to GPFS though for consistency.
>
> > On Fri., 29 Jun. 2018, 09:04 Randy Lynn,  wrote:
> > > So we have two data centers already running..
> > >
> > > AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel.. 
> > > I'm wanting to move the current US-EAST from AZ 1a to 1e..
> > > I know all docs say use ec2multiregion for multi-DC.
> > >
> > > I like the GPFS idea. would that work with the multi-DC too?
> > > What's the downside? status would report rack of 1a, even though in 1e?
> > >
> > > Thanks in advance for the help/thoughts!!
> > >
> > >
> > > > On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves  
> > > > wrote:
> > > > > There is a need for a repair with both DCs as rebuild will not stream 
> > > > > all replicas, so unless you can guarantee you were perfectly 
> > > > > consistent at time of rebuild you'll want to do a repair after 
> > > > > rebuild.
> > > > >
> > > > > On another note you could just replace the nodes but use GPFS instead 
> > > > > of EC2 snitch, using the same rack name.
> > > > >
> > > > > > On Fri., 29 Jun. 2018, 00:19 Rahul Singh, 
> > > > > >  wrote:
> > > > > > > Parallel load is the best approach and then switch your Data 
> > > > > > > access code to only access the new hardware. After you verify 
> > > > > > > that there are no local read / writes on the OLD dc and that the 
> > > > > > > updates are only via Gossip, then go ahead and change the 
> > > > > > > replication factor on the key space to have zero replicas in the 
> > > > > > > old DC. Then you can decommissioned.
> > > > > > >
> > > > > > > This way you are hundred percent sure that you aren’t missing any 
> > > > > > > new data. No need for a DC to DC repair but a repair is always 
> > > > > > > healthy.
> > > > > > >
> > > > > > > Rahul
> > > > > > > On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , 
> > > > > > > wrote:
> > > > > > > > Already running with Ec2.
> > > > > > > >
> > > > > > > > My original thought was a new DC parallel to the current, and 
> > > > > > > > then decommission the other DC.
> > > > > > > >
> > > > > > > > Also my data load is small right now.. I know small is relative 
> > > > > > > > term.. each node is carrying about 6GB..
> > > > > > > >
> > > > > > > > So given the data size, would you go with parallel DC or let 
> > > > > > > > the new AZ carry a heavy load until the others are migrated 
> > > > > > > > over?
> > > > > > > > and then I think "repair" to cleanup the replications?
> > > > > > > >
> > > > > > > >
> > > > > > > > > On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh 
> > > > > > > > >  wrote:
> > > > > > > > > > You don’t have to use EC2 snitch on AWS but if you have 
> > > > > > > > > > already started with it , it may put a node in a different 
> > > > > > > > > > DC.
> > > > > > > > > >
> > > > > > > > > > If your data density won’t be ridiculous You could add 3 to 
> > > > > > > > > > different DC/ Region and then sync up. After the new DC is 
> > > > > > > > > > operational you can remove one at a time on the old DC and 
> > > > > > > > > > at the same time add to the new one.
> > > > > > > > > >
> > > > > > > > > > Rahul
> > > > > > > > > > On Jun 28, 2018, 9:03 AM -0500, Randy Lynn 
> > > > > > > > > > , wrote:
> > > > > > > > > > > I have a 6-node cluster I'm migrating to the new i3 types.
> > > > > > > > > > > But at the same time I want to migrate to a different AZ.
> > > > > > > > > > >
> > > > > > > > > > > What happens if I do the "running node replace method" 
> > > > > > > > > > > with 1 node at a time moving to the new AZ. Meaning, I'll 
> > > > > > > > > > > have temporarily;
> > > > > > > > > > >
> > > > > > > > > > > 5 nodes in AZ 1c
> > > > > > > > > > > 1 new node in AZ 1e.
> > > > > > > > > > >
> > > > > > > > > > > I'll wash-rinse-repeat till all 6 are on the new machine 
> > > > > > > > > > > type and in the new AZ.
> > > > > > > > > > >
> > > > > > > > > > > Any thoughts about whether this gets weird with the 
> > > > > > > > > > > Ec2Snitch and a RF 3?
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > > Randy Lynn
> > > > > > > > > > > rl...@getavail.com
> > > > > > > > > > >
> > > > > > > > > > > office:
> > > > > > > > > > > 859.963.1616 ext 202
> > > > > > > > > > > 163 East Main Street - Lexington, KY 40507 - USA
> > > > > > > > > > >
> > > > > > > > > > > getavail.com
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Randy Lynn
> > > > > > > > rl...@getavail.com
> > > > > > > >
> > > > > > > > office:
> > > > > > > > 859.963.1616 ext 202
> > > > > > > > 163 East Main 

Re: C* in multiple AWS AZ's

[kurt greaves]

Yes. You would just end up with a rack named differently to the AZ. This is
not a problem as racks are just logical. I would recommend migrating all
your DCs to GPFS though for consistency.

On Fri., 29 Jun. 2018, 09:04 Randy Lynn,  wrote:

> So we have two data centers already running..
>
> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel..
> I'm wanting to move the current US-EAST from AZ 1a to 1e..
> I know all docs say use ec2multiregion for multi-DC.
>
> I like the GPFS idea. would that work with the multi-DC too?
> What's the downside? status would report rack of 1a, even though in 1e?
>
> Thanks in advance for the help/thoughts!!
>
>
> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves 
> wrote:
>
>> There is a need for a repair with both DCs as rebuild will not stream all
>> replicas, so unless you can guarantee you were perfectly consistent at time
>> of rebuild you'll want to do a repair after rebuild.
>>
>> On another note you could just replace the nodes but use GPFS instead of
>> EC2 snitch, using the same rack name.
>>
>> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, 
>> wrote:
>>
>>> Parallel load is the best approach and then switch your Data access code
>>> to only access the new hardware. After you verify that there are no local
>>> read / writes on the OLD dc and that the updates are only via Gossip, then
>>> go ahead and change the replication factor on the key space to have zero
>>> replicas in the old DC. Then you can decommissioned.
>>>
>>> This way you are hundred percent sure that you aren’t missing any new
>>> data. No need for a DC to DC repair but a repair is always healthy.
>>>
>>> Rahul
>>> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , wrote:
>>>
>>> Already running with Ec2.
>>>
>>> My original thought was a new DC parallel to the current, and then
>>> decommission the other DC.
>>>
>>> Also my data load is small right now.. I know small is relative term..
>>> each node is carrying about 6GB..
>>>
>>> So given the data size, would you go with parallel DC or let the new AZ
>>> carry a heavy load until the others are migrated over?
>>> and then I think "repair" to cleanup the replications?
>>>
>>>
>>> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh <
>>> rahul.xavier.si...@gmail.com> wrote:
>>>
 You don’t have to use EC2 snitch on AWS but if you have already started
 with it , it may put a node in a different DC.

 If your data density won’t be ridiculous You could add 3 to different
 DC/ Region and then sync up. After the new DC is operational you can remove
 one at a time on the old DC and at the same time add to the new one.

 Rahul
 On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , wrote:

 I have a 6-node cluster I'm migrating to the new i3 types.
 But at the same time I want to migrate to a different AZ.

 What happens if I do the "running node replace method" with 1 node at a
 time moving to the new AZ. Meaning, I'll have temporarily;

 5 nodes in AZ 1c
 1 new node in AZ 1e.

 I'll wash-rinse-repeat till all 6 are on the new machine type and in
 the new AZ.

 Any thoughts about whether this gets weird with the Ec2Snitch and a RF
 3?

 --
 Randy Lynn
 rl...@getavail.com

 office:
 859.963.1616 <+1-859-963-1616> ext 202
 163 East Main Street - Lexington, KY 40507 - USA
 

  getavail.com 


>>>
>>>
>>> --
>>> Randy Lynn
>>> rl...@getavail.com
>>>
>>> office:
>>> 859.963.1616 <+1-859-963-1616> ext 202
>>> 163 East Main Street - Lexington, KY 40507 - USA
>>> 
>>>
>>>  getavail.com 
>>>
>>>
>
>
> --
> Randy Lynn
> rl...@getavail.com
>
> office:
> 859.963.1616 <+1-859-963-1616> ext 202
> 163 East Main Street - Lexington, KY 40507 - USA
>
>  getavail.com 
>

Re: C* in multiple AWS AZ's

[Pradeep Chhetri]

Just curious -

>From which instance type are you migrating to i3 type and what are the
reasons to move to i3 type ?

Are you going to take benefit from NVMe instance storage - if yes, how ?

Since we are also migrating our cluster on AWS - but we are currently using
r4 instance, so i was interested to know if you did a comparison between r4
and i3 types.

Regards,
Pradeep

On Fri, Jun 29, 2018 at 4:49 AM, Randy Lynn  wrote:

> So we have two data centers already running..
>
> AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel..
> I'm wanting to move the current US-EAST from AZ 1a to 1e..
> I know all docs say use ec2multiregion for multi-DC.
>
> I like the GPFS idea. would that work with the multi-DC too?
> What's the downside? status would report rack of 1a, even though in 1e?
>
> Thanks in advance for the help/thoughts!!
>
>
> On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves 
> wrote:
>
>> There is a need for a repair with both DCs as rebuild will not stream all
>> replicas, so unless you can guarantee you were perfectly consistent at time
>> of rebuild you'll want to do a repair after rebuild.
>>
>> On another note you could just replace the nodes but use GPFS instead of
>> EC2 snitch, using the same rack name.
>>
>> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, 
>> wrote:
>>
>>> Parallel load is the best approach and then switch your Data access code
>>> to only access the new hardware. After you verify that there are no local
>>> read / writes on the OLD dc and that the updates are only via Gossip, then
>>> go ahead and change the replication factor on the key space to have zero
>>> replicas in the old DC. Then you can decommissioned.
>>>
>>> This way you are hundred percent sure that you aren’t missing any new
>>> data. No need for a DC to DC repair but a repair is always healthy.
>>>
>>> Rahul
>>> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , wrote:
>>>
>>> Already running with Ec2.
>>>
>>> My original thought was a new DC parallel to the current, and then
>>> decommission the other DC.
>>>
>>> Also my data load is small right now.. I know small is relative term..
>>> each node is carrying about 6GB..
>>>
>>> So given the data size, would you go with parallel DC or let the new AZ
>>> carry a heavy load until the others are migrated over?
>>> and then I think "repair" to cleanup the replications?
>>>
>>>
>>> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh <
>>> rahul.xavier.si...@gmail.com> wrote:
>>>
 You don’t have to use EC2 snitch on AWS but if you have already started
 with it , it may put a node in a different DC.

 If your data density won’t be ridiculous You could add 3 to different
 DC/ Region and then sync up. After the new DC is operational you can remove
 one at a time on the old DC and at the same time add to the new one.

 Rahul
 On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , wrote:

 I have a 6-node cluster I'm migrating to the new i3 types.
 But at the same time I want to migrate to a different AZ.

 What happens if I do the "running node replace method" with 1 node at a
 time moving to the new AZ. Meaning, I'll have temporarily;

 5 nodes in AZ 1c
 1 new node in AZ 1e.

 I'll wash-rinse-repeat till all 6 are on the new machine type and in
 the new AZ.

 Any thoughts about whether this gets weird with the Ec2Snitch and a RF
 3?

 --
 Randy Lynn
 rl...@getavail.com

 office:
 859.963.1616 <+1-859-963-1616> ext 202
 163 East Main Street - Lexington, KY 40507 - USA
 

  getavail.com 


>>>
>>>
>>> --
>>> Randy Lynn
>>> rl...@getavail.com
>>>
>>> office:
>>> 859.963.1616 <+1-859-963-1616> ext 202
>>> 163 East Main Street - Lexington, KY 40507 - USA
>>> 
>>>
>>>  getavail.com 
>>>
>>>
>
>
> --
> Randy Lynn
> rl...@getavail.com
>
> office:
> 859.963.1616 <+1-859-963-1616> ext 202
> 163 East Main Street - Lexington, KY 40507 - USA
> 
>
>  getavail.com 
>

Re: C* in multiple AWS AZ's

[Randy Lynn]

So we have two data centers already running..

AP-SYDNEY, and US-EAST.. I'm using Ec2Snitch over a site-to-site tunnel..
I'm wanting to move the current US-EAST from AZ 1a to 1e..
I know all docs say use ec2multiregion for multi-DC.

I like the GPFS idea. would that work with the multi-DC too?
What's the downside? status would report rack of 1a, even though in 1e?

Thanks in advance for the help/thoughts!!


On Thu, Jun 28, 2018 at 6:20 PM, kurt greaves  wrote:

> There is a need for a repair with both DCs as rebuild will not stream all
> replicas, so unless you can guarantee you were perfectly consistent at time
> of rebuild you'll want to do a repair after rebuild.
>
> On another note you could just replace the nodes but use GPFS instead of
> EC2 snitch, using the same rack name.
>
> On Fri., 29 Jun. 2018, 00:19 Rahul Singh, 
> wrote:
>
>> Parallel load is the best approach and then switch your Data access code
>> to only access the new hardware. After you verify that there are no local
>> read / writes on the OLD dc and that the updates are only via Gossip, then
>> go ahead and change the replication factor on the key space to have zero
>> replicas in the old DC. Then you can decommissioned.
>>
>> This way you are hundred percent sure that you aren’t missing any new
>> data. No need for a DC to DC repair but a repair is always healthy.
>>
>> Rahul
>> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , wrote:
>>
>> Already running with Ec2.
>>
>> My original thought was a new DC parallel to the current, and then
>> decommission the other DC.
>>
>> Also my data load is small right now.. I know small is relative term..
>> each node is carrying about 6GB..
>>
>> So given the data size, would you go with parallel DC or let the new AZ
>> carry a heavy load until the others are migrated over?
>> and then I think "repair" to cleanup the replications?
>>
>>
>> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh <
>> rahul.xavier.si...@gmail.com> wrote:
>>
>>> You don’t have to use EC2 snitch on AWS but if you have already started
>>> with it , it may put a node in a different DC.
>>>
>>> If your data density won’t be ridiculous You could add 3 to different
>>> DC/ Region and then sync up. After the new DC is operational you can remove
>>> one at a time on the old DC and at the same time add to the new one.
>>>
>>> Rahul
>>> On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , wrote:
>>>
>>> I have a 6-node cluster I'm migrating to the new i3 types.
>>> But at the same time I want to migrate to a different AZ.
>>>
>>> What happens if I do the "running node replace method" with 1 node at a
>>> time moving to the new AZ. Meaning, I'll have temporarily;
>>>
>>> 5 nodes in AZ 1c
>>> 1 new node in AZ 1e.
>>>
>>> I'll wash-rinse-repeat till all 6 are on the new machine type and in the
>>> new AZ.
>>>
>>> Any thoughts about whether this gets weird with the Ec2Snitch and a RF 3?
>>>
>>> --
>>> Randy Lynn
>>> rl...@getavail.com
>>>
>>> office:
>>> 859.963.1616 <+1-859-963-1616> ext 202
>>> 163 East Main Street - Lexington, KY 40507 - USA
>>> 
>>>
>>>  getavail.com 
>>>
>>>
>>
>>
>> --
>> Randy Lynn
>> rl...@getavail.com
>>
>> office:
>> 859.963.1616 <+1-859-963-1616> ext 202
>> 163 East Main Street - Lexington, KY 40507 - USA
>> 
>>
>>  getavail.com 
>>
>>


-- 
Randy Lynn
rl...@getavail.com

office:
859.963.1616 <+1-859-963-1616> ext 202
163 East Main Street - Lexington, KY 40507 - USA

 getavail.com 

Re: C* in multiple AWS AZ's

[kurt greaves]

There is a need for a repair with both DCs as rebuild will not stream all
replicas, so unless you can guarantee you were perfectly consistent at time
of rebuild you'll want to do a repair after rebuild.

On another note you could just replace the nodes but use GPFS instead of
EC2 snitch, using the same rack name.

On Fri., 29 Jun. 2018, 00:19 Rahul Singh, 
wrote:

> Parallel load is the best approach and then switch your Data access code
> to only access the new hardware. After you verify that there are no local
> read / writes on the OLD dc and that the updates are only via Gossip, then
> go ahead and change the replication factor on the key space to have zero
> replicas in the old DC. Then you can decommissioned.
>
> This way you are hundred percent sure that you aren’t missing any new
> data. No need for a DC to DC repair but a repair is always healthy.
>
> Rahul
> On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , wrote:
>
> Already running with Ec2.
>
> My original thought was a new DC parallel to the current, and then
> decommission the other DC.
>
> Also my data load is small right now.. I know small is relative term..
> each node is carrying about 6GB..
>
> So given the data size, would you go with parallel DC or let the new AZ
> carry a heavy load until the others are migrated over?
> and then I think "repair" to cleanup the replications?
>
>
> On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh <
> rahul.xavier.si...@gmail.com> wrote:
>
>> You don’t have to use EC2 snitch on AWS but if you have already started
>> with it , it may put a node in a different DC.
>>
>> If your data density won’t be ridiculous You could add 3 to different DC/
>> Region and then sync up. After the new DC is operational you can remove one
>> at a time on the old DC and at the same time add to the new one.
>>
>> Rahul
>> On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , wrote:
>>
>> I have a 6-node cluster I'm migrating to the new i3 types.
>> But at the same time I want to migrate to a different AZ.
>>
>> What happens if I do the "running node replace method" with 1 node at a
>> time moving to the new AZ. Meaning, I'll have temporarily;
>>
>> 5 nodes in AZ 1c
>> 1 new node in AZ 1e.
>>
>> I'll wash-rinse-repeat till all 6 are on the new machine type and in the
>> new AZ.
>>
>> Any thoughts about whether this gets weird with the Ec2Snitch and a RF 3?
>>
>> --
>> Randy Lynn
>> rl...@getavail.com
>>
>> office:
>> 859.963.1616 <+1-859-963-1616> ext 202
>> 163 East Main Street - Lexington, KY 40507 - USA
>> 
>>
>>  getavail.com 
>>
>>
>
>
> --
> Randy Lynn
> rl...@getavail.com
>
> office:
> 859.963.1616 <+1-859-963-1616> ext 202
> 163 East Main Street - Lexington, KY 40507 - USA
>
>  getavail.com 
>
>

Re: C* in multiple AWS AZ's

[Rahul Singh]

Parallel load is the best approach and then switch your Data access code to 
only access the new hardware. After you verify that there are no local read / 
writes on the OLD dc and that the updates are only via Gossip, then go ahead 
and change the replication factor on the key space to have zero replicas in the 
old DC. Then you can decommissioned.

This way you are hundred percent sure that you aren’t missing any new data. No 
need for a DC to DC repair but a repair is always healthy.

Rahul
On Jun 28, 2018, 9:15 AM -0500, Randy Lynn , wrote:
> Already running with Ec2.
>
> My original thought was a new DC parallel to the current, and then 
> decommission the other DC.
>
> Also my data load is small right now.. I know small is relative term.. each 
> node is carrying about 6GB..
>
> So given the data size, would you go with parallel DC or let the new AZ carry 
> a heavy load until the others are migrated over?
> and then I think "repair" to cleanup the replications?
>
>
> > On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh 
> >  wrote:
> > > You don’t have to use EC2 snitch on AWS but if you have already started 
> > > with it , it may put a node in a different DC.
> > >
> > > If your data density won’t be ridiculous You could add 3 to different DC/ 
> > > Region and then sync up. After the new DC is operational you can remove 
> > > one at a time on the old DC and at the same time add to the new one.
> > >
> > > Rahul
> > > On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , wrote:
> > > > I have a 6-node cluster I'm migrating to the new i3 types.
> > > > But at the same time I want to migrate to a different AZ.
> > > >
> > > > What happens if I do the "running node replace method" with 1 node at a 
> > > > time moving to the new AZ. Meaning, I'll have temporarily;
> > > >
> > > > 5 nodes in AZ 1c
> > > > 1 new node in AZ 1e.
> > > >
> > > > I'll wash-rinse-repeat till all 6 are on the new machine type and in 
> > > > the new AZ.
> > > >
> > > > Any thoughts about whether this gets weird with the Ec2Snitch and a RF 
> > > > 3?
> > > >
> > > > --
> > > > Randy Lynn
> > > > rl...@getavail.com
> > > >
> > > > office:
> > > > 859.963.1616 ext 202
> > > > 163 East Main Street - Lexington, KY 40507 - USA
> > > >
> > > > getavail.com
>
>
>
> --
> Randy Lynn
> rl...@getavail.com
>
> office:
> 859.963.1616 ext 202
> 163 East Main Street - Lexington, KY 40507 - USA
>
> getavail.com

Re: C* in multiple AWS AZ's

[Randy Lynn]

Already running with Ec2.

My original thought was a new DC parallel to the current, and then
decommission the other DC.

Also my data load is small right now.. I know small is relative term.. each
node is carrying about 6GB..

So given the data size, would you go with parallel DC or let the new AZ
carry a heavy load until the others are migrated over?
and then I think "repair" to cleanup the replications?


On Thu, Jun 28, 2018 at 10:09 AM, Rahul Singh 
wrote:

> You don’t have to use EC2 snitch on AWS but if you have already started
> with it , it may put a node in a different DC.
>
> If your data density won’t be ridiculous You could add 3 to different DC/
> Region and then sync up. After the new DC is operational you can remove one
> at a time on the old DC and at the same time add to the new one.
>
> Rahul
> On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , wrote:
>
> I have a 6-node cluster I'm migrating to the new i3 types.
> But at the same time I want to migrate to a different AZ.
>
> What happens if I do the "running node replace method" with 1 node at a
> time moving to the new AZ. Meaning, I'll have temporarily;
>
> 5 nodes in AZ 1c
> 1 new node in AZ 1e.
>
> I'll wash-rinse-repeat till all 6 are on the new machine type and in the
> new AZ.
>
> Any thoughts about whether this gets weird with the Ec2Snitch and a RF 3?
>
> --
> Randy Lynn
> rl...@getavail.com
>
> office:
> 859.963.1616 <+1-859-963-1616> ext 202
> 163 East Main Street - Lexington, KY 40507 - USA
> 
>
>  getavail.com 
>
>


-- 
Randy Lynn
rl...@getavail.com

office:
859.963.1616 <+1-859-963-1616> ext 202
163 East Main Street - Lexington, KY 40507 - USA

 getavail.com 

Re: C* in multiple AWS AZ's

[Rahul Singh]

You don’t have to use EC2 snitch on AWS but if you have already started with it 
, it may put a node in a different DC.

If your data density won’t be ridiculous You could add 3 to different DC/ 
Region and then sync up. After the new DC is operational you can remove one at 
a time on the old DC and at the same time add to the new one.

Rahul
On Jun 28, 2018, 9:03 AM -0500, Randy Lynn , wrote:
> I have a 6-node cluster I'm migrating to the new i3 types.
> But at the same time I want to migrate to a different AZ.
>
> What happens if I do the "running node replace method" with 1 node at a time 
> moving to the new AZ. Meaning, I'll have temporarily;
>
> 5 nodes in AZ 1c
> 1 new node in AZ 1e.
>
> I'll wash-rinse-repeat till all 6 are on the new machine type and in the new 
> AZ.
>
> Any thoughts about whether this gets weird with the Ec2Snitch and a RF 3?
>
> --
> Randy Lynn
> rl...@getavail.com
>
> office:
> 859.963.1616 ext 202
> 163 East Main Street - Lexington, KY 40507 - USA
>
> getavail.com

Re: C* in multiple AWS AZ's

[Jeff Jirsa]

The single node in 1e will be a replica for every range (and you won’t be able 
to tolerate an outage in 1c), potentially putting it under significant load

-- 
Jeff Jirsa


> On Jun 28, 2018, at 7:02 AM, Randy Lynn  wrote:
> 
> I have a 6-node cluster I'm migrating to the new i3 types.
> But at the same time I want to migrate to a different AZ.
> 
> What happens if I do the "running node replace method" with 1 node at a time 
> moving to the new AZ. Meaning, I'll have temporarily;
> 
> 5 nodes in AZ 1c
> 1 new node in AZ 1e.
> 
> I'll wash-rinse-repeat till all 6 are on the new machine type and in the new 
> AZ.
> 
> Any thoughts about whether this gets weird with the Ec2Snitch and a RF 3?
> 
> -- 
> Randy Lynn 
> rl...@getavail.com 
> 
> office: 
> 859.963.1616 ext 202 
> 163 East Main Street - Lexington, KY 40507 - USA 
> 
>   getavail.com

C* in multiple AWS AZ's

[Randy Lynn]

I have a 6-node cluster I'm migrating to the new i3 types.
But at the same time I want to migrate to a different AZ.

What happens if I do the "running node replace method" with 1 node at a
time moving to the new AZ. Meaning, I'll have temporarily;

5 nodes in AZ 1c
1 new node in AZ 1e.

I'll wash-rinse-repeat till all 6 are on the new machine type and in the
new AZ.

Any thoughts about whether this gets weird with the Ec2Snitch and a RF 3?

-- 
Randy Lynn
rl...@getavail.com

office:
859.963.1616 <+1-859-963-1616> ext 202
163 East Main Street - Lexington, KY 40507 - USA

 getavail.com