Re: Elastic IP for Cassandra in AWS

[Justin Cameron]

+1

Scaling up & down can also take some time (multiple days even) if the nodes
in the cluster hold a lot of data. The process has a performance impact on
the cluster, as the rest of the cluster must stream data to the new nodes,
or stream data away from the decommissioning nodes, also usually leading to
increased compaction activity. When adding/removing multiple nodes in
parallel this performance impact is increased. If the cluster is already
under very heavy load, or has other performance-related issues, then an ASG
automatically adding multiple nodes could be enough to bring your
application down or make it unusable. It's usually also a good idea to run
a repair afterwards.

You'd also need to handle the case where the ASG decided to remove nodes
that were still joining the cluster.

Definitely not impossible, but not an easy task.

On Tue, 17 Oct 2017 at 16:40 Jeff Jirsa  wrote:

> Just to expand on this:
>
> In an asg environment, you could have an autoscaling event to expand or
> shrink the cluster and multiple nodes terminate at the same time. Your
> userdata or your AMI would have to know how to find the cluster, know how
> many instances were down and what the target size was, check to see which
> of those were down from app level issues versus down from instance level
> issues, vs not joined the cluster yet, figure out which of the down
> instances it is meant to replace, and then rejoin the cluster before
> another replica fails. And even without down instances, simultaneous
> bootstrap and decom has serious implications for consistency guarantees.
>
> It’s not impossible, but it’s far from trivial, and figuring out how to
> grab an appropriate EIP is about the easiest part of it. If you remove the
> scale in/our component it gets a lot easier, but then you likely don’t
> really need an ASG at all.
>
>
>
> --
> Jeff Jirsa
>
>
> On Oct 16, 2017, at 10:06 PM, Justin Cameron 
> wrote:
>
> I wouldn't recommend putting your Cassandra nodes in an ASG. Scaling
> production Cassandra clusters can be a tricky process that should be done
> manually. There are a lot of things that can go wrong if you aren't
> watching/controlling the process.
>
> Terraform is a good option for automating complex AWS deployments. It also
> allows you to scale your cluster incrementally once it's been deployed.
> CloudFormation is another option. There are also fully managed services
> that will take care of it all for you, such as Instaclustr (full disclosure
> - I work for Instaclustr).
>
>
> On Tue, 17 Oct 2017 at 15:45 cass savy  wrote:
>
>> How can we solve EIP allocation using APIs for  multiple nodes that are
>> in one ASG? What are the config management tools  that you are referring to?
>>
>> On Mon, Oct 16, 2017 at 9:23 PM, kurt greaves 
>> wrote:
>>
>>> AWS API's provide the functionality to allocate and associate elastic
>>> IPs to instances. Generally the API's aren't pretty but they work. What
>>> issues are you having? If it's a configuration problem there are a variety
>>> of config management tools that you can use to populate the yaml/env files
>>> with the correct IP addresses.​
>>>
>>
>> --
>
>
> *Justin Cameron*Senior Software Engineer
>
>
> 
>
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>
> --


*Justin Cameron*Senior Software Engineer





This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Re: Elastic IP for Cassandra in AWS

[Oleksandr Shulgin]

On Tue, Oct 17, 2017 at 7:40 AM, Jeff Jirsa  wrote:

> Just to expand on this:
>
> In an asg environment, you could have an autoscaling event to expand or
> shrink the cluster and multiple nodes terminate at the same time. Your
> userdata or your AMI would have to know how to find the cluster, know how
> many instances were down and what the target size was, check to see which
> of those were down from app level issues versus down from instance level
> issues, vs not joined the cluster yet, figure out which of the down
> instances it is meant to replace, and then rejoin the cluster before
> another replica fails. And even without down instances, simultaneous
> bootstrap and decom has serious implications for consistency guarantees.
>
> It’s not impossible, but it’s far from trivial, and figuring out how to
> grab an appropriate EIP is about the easiest part of it. If you remove the
> scale in/our component it gets a lot easier, but then you likely don’t
> really need an ASG at all.
>

We've actually tried both approaches[1,2] and I can say that AutoScaling
isn't suitable for Cassandra deployment.  It might be all OK for a while,
but as soon as a node fails, it's pretty tricky to automate (and in
general: impossible w/o data loss) to bring up a replacement node in a sane
way.

Why put Cassandra nodes in AutoScaling Group at all?  If that's for
recovering lost nodes, then you'd better have a look on auto-recovery
features of AWS.  That one requires use of EBS for data, so you won't be
able to run the locally-attached SSD storage, but not every application
needs that speed.  Remember, most of IO in Cassandra is sequential by
design anyway.

To answer original question, allocating an EIP is more or less a single
line of Python code:
https://github.com/zalando/planb-cassandra/blob/7f1dfcb99a77eda72bce97da8f9b15412eed3c1a/planb/create_cluster.py#L253

Associating an EIP with a running instance is another line:
https://github.com/zalando/planb-cassandra/blob/7f1dfcb99a77eda72bce97da8f9b15412eed3c1a/planb/create_cluster.py#L477

[1] https://github.com/a1exsh/stups-cassandra
[2] https://github.com/zalando/planb-cassandra/

Cheers,
--
Alex

Re: Elastic IP for Cassandra in AWS

[Jeff Jirsa]

Just to expand on this:

In an asg environment, you could have an autoscaling event to expand or shrink 
the cluster and multiple nodes terminate at the same time. Your userdata or 
your AMI would have to know how to find the cluster, know how many instances 
were down and what the target size was, check to see which of those were down 
from app level issues versus down from instance level issues, vs not joined the 
cluster yet, figure out which of the down instances it is meant to replace, and 
then rejoin the cluster before another replica fails. And even without down 
instances, simultaneous bootstrap and decom has serious implications for 
consistency guarantees.

It’s not impossible, but it’s far from trivial, and figuring out how to grab an 
appropriate EIP is about the easiest part of it. If you remove the scale in/our 
component it gets a lot easier, but then you likely don’t really need an ASG at 
all. 


-- 
Jeff Jirsa


> On Oct 16, 2017, at 10:06 PM, Justin Cameron  wrote:
> 
> I wouldn't recommend putting your Cassandra nodes in an ASG. Scaling 
> production Cassandra clusters can be a tricky process that should be done 
> manually. There are a lot of things that can go wrong if you aren't 
> watching/controlling the process.
> 
> Terraform is a good option for automating complex AWS deployments. It also 
> allows you to scale your cluster incrementally once it's been deployed. 
> CloudFormation is another option. There are also fully managed services that 
> will take care of it all for you, such as Instaclustr (full disclosure - I 
> work for Instaclustr).
> 
> 
>> On Tue, 17 Oct 2017 at 15:45 cass savy  wrote:
>> How can we solve EIP allocation using APIs for  multiple nodes that are in 
>> one ASG? What are the config management tools  that you are referring to?
>> 
>>> On Mon, Oct 16, 2017 at 9:23 PM, kurt greaves  wrote:
>>> AWS API's provide the functionality to allocate and associate elastic IPs 
>>> to instances. Generally the API's aren't pretty but they work. What issues 
>>> are you having? If it's a configuration problem there are a variety of 
>>> config management tools that you can use to populate the yaml/env files 
>>> with the correct IP addresses.​
>> 
> 
> -- 
> Justin Cameron
> Senior Software Engineer
> 
> 
> 
> 
> 
> 
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia) 
> and Instaclustr Inc (USA).
> 
> This email and any attachments may contain confidential and legally 
> privileged information.  If you are not the intended recipient, do not copy 
> or disclose its content, but please reply to this email immediately and 
> highlight the error to the sender and then immediately delete the message.

Re: Elastic IP for Cassandra in AWS

[Justin Cameron]

I wouldn't recommend putting your Cassandra nodes in an ASG. Scaling
production Cassandra clusters can be a tricky process that should be done
manually. There are a lot of things that can go wrong if you aren't
watching/controlling the process.

Terraform is a good option for automating complex AWS deployments. It also
allows you to scale your cluster incrementally once it's been deployed.
CloudFormation is another option. There are also fully managed services
that will take care of it all for you, such as Instaclustr (full disclosure
- I work for Instaclustr).


On Tue, 17 Oct 2017 at 15:45 cass savy  wrote:

> How can we solve EIP allocation using APIs for  multiple nodes that are in
> one ASG? What are the config management tools  that you are referring to?
>
> On Mon, Oct 16, 2017 at 9:23 PM, kurt greaves 
> wrote:
>
>> AWS API's provide the functionality to allocate and associate elastic IPs
>> to instances. Generally the API's aren't pretty but they work. What issues
>> are you having? If it's a configuration problem there are a variety of
>> config management tools that you can use to populate the yaml/env files
>> with the correct IP addresses.​
>>
>
> --


*Justin Cameron*Senior Software Engineer





This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Re: Elastic IP for Cassandra in AWS

[cass savy]

How can we solve EIP allocation using APIs for  multiple nodes that are in
one ASG? What are the config management tools  that you are referring to?

On Mon, Oct 16, 2017 at 9:23 PM, kurt greaves  wrote:

> AWS API's provide the functionality to allocate and associate elastic IPs
> to instances. Generally the API's aren't pretty but they work. What issues
> are you having? If it's a configuration problem there are a variety of
> config management tools that you can use to populate the yaml/env files
> with the correct IP addresses.​
>

Re: Elastic IP for Cassandra in AWS

[kurt greaves]

AWS API's provide the functionality to allocate and associate elastic IPs
to instances. Generally the API's aren't pretty but they work. What issues
are you having? If it's a configuration problem there are a variety of
config management tools that you can use to populate the yaml/env files
with the correct IP addresses.​

Elastic IP for Cassandra in AWS

[cass savy]

We are working on automation framework to build new C* ring in AWS and
looking for suggestions to determine ways to automatically allocate
EIP/public IP during new cluster build and scale up/replace node situations.

We are using  elastic IP/public ip for broadcast address and private IP for
listen address for AWS Cassandra clusters.

 Has anybody done this for their PROD cluster? If yes, can you suggest
means to dynamically/automatically allocate/associate   EIP/public IP to C*
node using any automation framework/services or tools?