My approach is the obvious taking a big outage window especially at work we are using 1.2 and is using token range. I am generally a believer that (1) patches should be applied, but (2) routinely we replace each host with a new EC2, so (1) I know my infrastructure code (puppet/chef/ansible/salt stack/whatever) doesn't just work on a 12-year old working EC2 (of course, I am joking about EC2 been up for 12 years).
The other approach I have been meaning to try is simply taking one node out at a time, and let the new node stream over (this has an issue with the IP changed, but be careful). But I think with the new versions of Cassandra, you can use vnode. I have been meaning to also try just adding new vnode and gradually retiring the older one. Does this approach sound like what people would do? EBS is great, and generally performance is great (but please pick a reasonable instance type appropriate for your workload, running on m3.large is definitely not the way to go for production usually, just an example). John On Sun, May 28, 2017 at 1:59 PM, daemeon reiydelle <daeme...@gmail.com> wrote: > This is in fact an interesting security practice that makes sense. It > assumes the existing ami had security holes that WERE ALREADY exploited. > See if you can negotiate moving the hdfs volumes to persistent storage. Fyi > two major banks I have worked with did much the same, but as the storage > was SAN (with VMware ) I was able to make adjustments to the ansible > scripts (client was providing mobile banking solutions to the bank) > > I had another client using AWS, Chef, Terraform. I WAS NOT able to make > this work in Chef. I can do it with Ansible, Terraform, AWS however. > > “All men dream, but not equally. Those who dream by night in the dusty > recesses of their minds wake up in the day to find it was vanity, but the > dreamers of the day are dangerous men, for they may act their dreams with > open eyes, to make it possible.” — T.E. Lawrence > > sent from my mobile > Daemeon Reiydelle > skype daemeon.c.m.reiydelle > USA 415.501.0198 <(415)%20501-0198> > > On May 28, 2017 1:25 AM, "Anthony Grasso" <anthony.gra...@gmail.com> > wrote: > >> Hi Surbhi, >> >> Please see my comment inline below. >> >> On 28 May 2017 at 12:11, Jeff Jirsa <jji...@apache.org> wrote: >> >>> >>> >>> On 2017-05-27 18:04 (-0700), Surbhi Gupta <surbhi.gupt...@gmail.com> >>> wrote: >>> > Thanks a lot for all of your reply. >>> > Our requirement is : >>> > Our company releases AMI almost every month where they have some or the >>> > other security packages. >>> > So as per our security team we need to move our cassandra cluster to >>> the >>> > new AMI . >>> > As this process happens every month, we would like to automate the >>> process . >>> > Few points to consider here: >>> > >>> > 1. We are using ephemeral drives to store cassandra data >>> > 2. We are on dse 4.8.x >>> > >>> > So currently to do the process, we pinup a new nodes with new DC name >>> and >>> > join that DC, alter the keyspace, do rebuild and later alter the >>> keyspace >>> > again to remove the old DC . >>> > >>> > But all of this process is manually done as of now. >>> > >>> > So i wanted to understand , on AWS, how do you do above kind of task >>> > automatically ? >>> >>> >>> At a previous employer, they used M4 class instances with data on a >>> dedicated EBS volumes, so we could swap AMIs / stop / start / adjust >>> instances without having to deal with this. This worked reasonably well for >>> their scale (which was petabytes of data). >>> >> >> This is a really good option as it avoids streaming data to replace a >> node which could potentially be quicker if dealing with large amounts of >> data on each node. >> >> >>> >>> Other companies using ephemeral tend to be more willing to just >>> terminate instances and replace them (-Dcassandra.replace_address). If you >>> stop cassandra, then boot a replacement with 'replace_address' set, it'll >>> take over for the stopped instance, including re-streaming all data (as >>> best it can, subject to consistency level and repair status). This may be >>> easier for you to script than switching your fleet to EBS, but it's not >>> without risk. >>> >> >> A quick note if you do decide to go down this path. If you are using >> Cassandra version 2.x.x and above, the cassandra.replace_address_firs >> t_boot can also be used. This option works once when Cassandra is first >> started and the replacement node inserted into the cluster. After that, the >> option is ignored for all subsequent restarts, where as >> cassandra.replace_address needs to be removed from the *cassandra-env.sh* >> file in order to restart the node. Restart behaviour aside, both options >> operate in the same way to replace a node in the cluster. >> >> >>> >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >>> For additional commands, e-mail: user-h...@cassandra.apache.org >>> >>> >>
