On Tue, Oct 17, 2017 at 7:40 AM, Jeff Jirsa <jji...@gmail.com> wrote:
> Just to expand on this: > > In an asg environment, you could have an autoscaling event to expand or > shrink the cluster and multiple nodes terminate at the same time. Your > userdata or your AMI would have to know how to find the cluster, know how > many instances were down and what the target size was, check to see which > of those were down from app level issues versus down from instance level > issues, vs not joined the cluster yet, figure out which of the down > instances it is meant to replace, and then rejoin the cluster before > another replica fails. And even without down instances, simultaneous > bootstrap and decom has serious implications for consistency guarantees. > > It’s not impossible, but it’s far from trivial, and figuring out how to > grab an appropriate EIP is about the easiest part of it. If you remove the > scale in/our component it gets a lot easier, but then you likely don’t > really need an ASG at all. > We've actually tried both approaches[1,2] and I can say that AutoScaling isn't suitable for Cassandra deployment. It might be all OK for a while, but as soon as a node fails, it's pretty tricky to automate (and in general: impossible w/o data loss) to bring up a replacement node in a sane way. Why put Cassandra nodes in AutoScaling Group at all? If that's for recovering lost nodes, then you'd better have a look on auto-recovery features of AWS. That one requires use of EBS for data, so you won't be able to run the locally-attached SSD storage, but not every application needs that speed. Remember, most of IO in Cassandra is sequential by design anyway. To answer original question, allocating an EIP is more or less a single line of Python code: https://github.com/zalando/planb-cassandra/blob/7f1dfcb99a77eda72bce97da8f9b15412eed3c1a/planb/create_cluster.py#L253 Associating an EIP with a running instance is another line: https://github.com/zalando/planb-cassandra/blob/7f1dfcb99a77eda72bce97da8f9b15412eed3c1a/planb/create_cluster.py#L477 [1] https://github.com/a1exsh/stups-cassandra [2] https://github.com/zalando/planb-cassandra/ Cheers, -- Alex
