Re: number of replicas per data center?
On Sun, Jan 18, 2015 at 8:50 PM, Kevin Burton bur...@spinn3r.com wrote: Ah.. six replicas. At least its super inexpensive that way (sarcasm!) People with larger numbers of data centers do tend to reduce their replication factor per DC. It's all about how much consistency you want to risk, rebuild over the WAN, etc.. =Rob
Re: number of replicas per data center?
Since our workload is spread globally, we spread our nodes across AWS regions as well: 2 nodes per zone, 6 nodes per region (datacenter) (RF 3), 12 nodes total (except during upgrade migrations). We autodeploy into VPCs. If a region goes bad we can route all traffic to another and bring up a third. Super inexpensive in that is very reliable, available and easy to manage. ml On Sun, Jan 18, 2015 at 11:50 PM, Kevin Burton bur...@spinn3r.com wrote: Ah.. six replicas. At least its super inexpensive that way (sarcasm!) On Sun, Jan 18, 2015 at 8:14 PM, Jonathan Haddad j...@jonhaddad.com wrote: Sorry, I left out RF. Yes, I prefer 3 replicas in each datacenter, and that's pretty common. On Sun Jan 18 2015 at 8:02:12 PM Kevin Burton bur...@spinn3r.com wrote: 3 what? :-P replicas per datacenter or 3 data centers? So if you have 2 data centers you would have 6 total replicas with 3 local replicas per datacenter? On Sun, Jan 18, 2015 at 7:53 PM, Jonathan Haddad j...@jonhaddad.com wrote: Personally I wouldn't go 3 unless you have a good reason. On Sun Jan 18 2015 at 7:52:10 PM Kevin Burton bur...@spinn3r.com wrote: How do people normally setup multiple data center replication in terms of number of *local* replicas? So say you have two data centers, do you have 2 local replicas, for a total of 4 replicas? Or do you have 2 in one datacenter, and 1 in another? If you only have one in a local datacenter then when it fails you have to transfer all that data over the WAN. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: number of replicas per data center?
Ah.. six replicas. At least its super inexpensive that way (sarcasm!) Well it's up to you to decide what your data locality and fault tolerance requirements are. If you want to run two DC's, costs are going to increase since each DC has a full set of replicas within itself. But you get the benefit of being more or less immune to losing an entire data center with no practical application impact. But it's true that multi-DC redundancy has a cost, and sometimes the increased cost of running a multi-DC RF=3 cluster costs more than the increased risk of downtime from a lower RF merits. If you use RF=2 and write your application to prefer QUORUM but degrade to ONE when QUORUM can't be achieved, you can still get many of the advantages of RF=3 and reduce your costs, particularly once you're in multiple DC's (i.e. you can tolerate one node being offline without a significant application impact). I know there are people who run production clusters that way, as Michael Laing mentioned. It's all tunable, you get to decide what's right for your organization. You also get to tune RF per datacenter per keyspace, so maybe you want to have a remote DC with RF=1 as a fallback in case your primary DC goes offline, but don't want to pay for a fully redundant location, and are ok with inconsistent reads if you lose your entire primary DC, and any node in your fallback DC. But practically speaking, at RF=2, you can't even take a node offline for maintenance without reducing your effective replication to 1, and at that point any spontaneous failure takes your application offline (and you also can't achieve quorum, so CAS operations will always fail). Maybe you're ok with that, I can think of usages where that would be fine. But you have to decide for yourself, and you're given all the tools you need to implement that decision. On Sun, Jan 18, 2015 at 9:50 PM, Kevin Burton bur...@spinn3r.com wrote: Ah.. six replicas. At least its super inexpensive that way (sarcasm!) On Sun, Jan 18, 2015 at 8:14 PM, Jonathan Haddad j...@jonhaddad.com wrote: Sorry, I left out RF. Yes, I prefer 3 replicas in each datacenter, and that's pretty common. On Sun Jan 18 2015 at 8:02:12 PM Kevin Burton bur...@spinn3r.com wrote: 3 what? :-P replicas per datacenter or 3 data centers? So if you have 2 data centers you would have 6 total replicas with 3 local replicas per datacenter? On Sun, Jan 18, 2015 at 7:53 PM, Jonathan Haddad j...@jonhaddad.com wrote: Personally I wouldn't go 3 unless you have a good reason. On Sun Jan 18 2015 at 7:52:10 PM Kevin Burton bur...@spinn3r.com wrote: How do people normally setup multiple data center replication in terms of number of *local* replicas? So say you have two data centers, do you have 2 local replicas, for a total of 4 replicas? Or do you have 2 in one datacenter, and 1 in another? If you only have one in a local datacenter then when it fails you have to transfer all that data over the WAN. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: number of replicas per data center?
Ah.. six replicas. At least its super inexpensive that way (sarcasm!) On Sun, Jan 18, 2015 at 8:14 PM, Jonathan Haddad j...@jonhaddad.com wrote: Sorry, I left out RF. Yes, I prefer 3 replicas in each datacenter, and that's pretty common. On Sun Jan 18 2015 at 8:02:12 PM Kevin Burton bur...@spinn3r.com wrote: 3 what? :-P replicas per datacenter or 3 data centers? So if you have 2 data centers you would have 6 total replicas with 3 local replicas per datacenter? On Sun, Jan 18, 2015 at 7:53 PM, Jonathan Haddad j...@jonhaddad.com wrote: Personally I wouldn't go 3 unless you have a good reason. On Sun Jan 18 2015 at 7:52:10 PM Kevin Burton bur...@spinn3r.com wrote: How do people normally setup multiple data center replication in terms of number of *local* replicas? So say you have two data centers, do you have 2 local replicas, for a total of 4 replicas? Or do you have 2 in one datacenter, and 1 in another? If you only have one in a local datacenter then when it fails you have to transfer all that data over the WAN. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: number of replicas per data center?
3 what? :-P replicas per datacenter or 3 data centers? So if you have 2 data centers you would have 6 total replicas with 3 local replicas per datacenter? On Sun, Jan 18, 2015 at 7:53 PM, Jonathan Haddad j...@jonhaddad.com wrote: Personally I wouldn't go 3 unless you have a good reason. On Sun Jan 18 2015 at 7:52:10 PM Kevin Burton bur...@spinn3r.com wrote: How do people normally setup multiple data center replication in terms of number of *local* replicas? So say you have two data centers, do you have 2 local replicas, for a total of 4 replicas? Or do you have 2 in one datacenter, and 1 in another? If you only have one in a local datacenter then when it fails you have to transfer all that data over the WAN. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: number of replicas per data center?
Sorry, I left out RF. Yes, I prefer 3 replicas in each datacenter, and that's pretty common. On Sun Jan 18 2015 at 8:02:12 PM Kevin Burton bur...@spinn3r.com wrote: 3 what? :-P replicas per datacenter or 3 data centers? So if you have 2 data centers you would have 6 total replicas with 3 local replicas per datacenter? On Sun, Jan 18, 2015 at 7:53 PM, Jonathan Haddad j...@jonhaddad.com wrote: Personally I wouldn't go 3 unless you have a good reason. On Sun Jan 18 2015 at 7:52:10 PM Kevin Burton bur...@spinn3r.com wrote: How do people normally setup multiple data center replication in terms of number of *local* replicas? So say you have two data centers, do you have 2 local replicas, for a total of 4 replicas? Or do you have 2 in one datacenter, and 1 in another? If you only have one in a local datacenter then when it fails you have to transfer all that data over the WAN. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: number of replicas per data center?
Personally I wouldn't go 3 unless you have a good reason. On Sun Jan 18 2015 at 7:52:10 PM Kevin Burton bur...@spinn3r.com wrote: How do people normally setup multiple data center replication in terms of number of *local* replicas? So say you have two data centers, do you have 2 local replicas, for a total of 4 replicas? Or do you have 2 in one datacenter, and 1 in another? If you only have one in a local datacenter then when it fails you have to transfer all that data over the WAN. -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile https://plus.google.com/102718274791889610666/posts http://spinn3r.com
Re: number of replicas per data center?
I like to have 3 replicas across 3 racks in each datacenter as a rue of thumb. You can vary that, but it depends upon the use case, and the SLA's for latency. This can get a little complicated if you're using the cloud and automated deployment strategies as I like to use the same abstractions externally as internally. -- Colin Clark +1-320-221-9531 On Jan 18, 2015, at 9:49 PM, Kevin Burton bur...@spinn3r.com wrote: How do people normally setup multiple data center replication in terms of number of *local* replicas? So say you have two data centers, do you have 2 local replicas, for a total of 4 replicas? Or do you have 2 in one datacenter, and 1 in another? If you only have one in a local datacenter then when it fails you have to transfer all that data over the WAN. -- Founder/CEO Spinn3r.com Location: San Francisco, CA blog: http://burtonator.wordpress.com … or check out my Google+ profile
