Re: AWS ephemeral instances + backup

2019-12-09 Thread Carl Mueller
iday, December 6, 2019 at 10:16 AM > *To: *"user@cassandra.apache.org" > *Subject: *Re: AWS ephemeral instances + backup > > > > *Message from External Sender* > > If you’re only going to have a small storage footprint per node like > 100gb, another option c

Re: AWS ephemeral instances + backup

2019-12-06 Thread Reid Pinchback
Correction: “most of your database will be in chunk cache, or buffer cache anyways. From: Reid Pinchback Reply-To: "user@cassandra.apache.org" Date: Friday, December 6, 2019 at 10:16 AM To: "user@cassandra.apache.org" Subject: Re: AWS ephemeral instances + backup M

Re: AWS ephemeral instances + backup

2019-12-06 Thread Reid Pinchback
the temporary slowpoke. From: Carl Mueller Reply-To: "user@cassandra.apache.org" Date: Thursday, December 5, 2019 at 3:21 PM To: "user@cassandra.apache.org" Subject: AWS ephemeral instances + backup Message from External Sender Does anyone have experience tooling written to support

Re: AWS ephemeral instances + backup

2019-12-05 Thread daemeon reiydelle
If you can handle the slower IO of S3 this can work, but you will have a window of out of date images. YOu don't have a concept of persistent snapshots. <==> Life lived is not about the size of the dog in the fight: It is about the size of the fight in the dog. *Daemeon Reiydelle* *email:

Re: AWS ephemeral instances + backup

2019-12-05 Thread Jon Haddad
You can easily do this with bcache or LVM http://rustyrazorblade.com/post/2018/2018-04-24-intro-to-lvm/. Medusa might be a good route to go down if you want to do backups instead: https://thelastpickle.com/blog/2019/11/05/cassandra-medusa-backup-tool-is-open-source.html On Thu, Dec 5, 2019 at

Re: AWS ephemeral instances + backup

2019-12-05 Thread Ben Slater
We have some tooling that does that kind of thing using S3 rather than attached EBS but a similar principle. There is a bit of an overview here: https://www.instaclustr.com/advanced-node-replace/ It's become a pretty core part of our ops toolbox since we introduced it. Cheers Ben --- *Ben

Re: AWS ephemeral instances + backup

2019-12-05 Thread Jeff Jirsa
No experience doing it that way personally, but I'm curious: Are you backing up in case of ephemeral instance dying, or backing up in case of data problems / errors / etc? On instance dying, you're probably fine with just straight normal replacements, not restoring from backup. For the rest, is

AWS ephemeral instances + backup

2019-12-05 Thread Carl Mueller
Does anyone have experience tooling written to support this strategy: Use case: run cassandra on i3 instances on ephemerals but synchronize the sstables and commitlog files to the cheapest EBS volume type (those have bad IOPS but decent enough throughput) On node replace, the startup script for