Re: Expanding Cassandra on EC2 with consistency
Hi Dan, We run RF 2 on RAID0 EBS drives. The reason we use EBS over on-instance storage is two fold; Firstly we have a relatively small cluster ( 4 nodes ), so we're quite sensitive to any AWS issues (at the region level). If we had a larger cluster then we would definitely use ephemeral storage as ephemeral storage will provide much more consistent (*slightly higher*) throughput than EBS. I'm sure you've read a lot about how bad EBS performance is, but genuinely we see very little difference between EBS / ephemeral storage in terms of performance when in a RAID0 setup. Some numbers which are similar to our tests can be found here: http://stu.mp/2009/12/disk-io-and-throughput-benchmarks-on-amazons-ec2.html. The only way you'll know for yourself whether EBS is acceptable is running your own tests, but definitely take anything you read on blog posts etc about EBS performance with a pinch of salt. The only real disadvantage of EBS in our experience is that it's an additional moving part that will fail. You can find yourself in a position where EC2 is running normally but EBS is down, thus your nodes are down. The second reason, and the most important for us was that we didn't have time to build a good automated backup service for the ephemeral storage. The advantage of EBS is that the instance can fail and we can start a new one using the same drive, where-as data on ephemeral will get lost if the node is lost/shutdown. Using ephemeral storage would put us in a difficult position of loosing some data should a couple of instances fail for whatever reason (we don't loose instances often, however when we do we tend to loose several). If you run a larger cluster (imo 10+) then definitely use ephemeral as you shouldn't be very sensitive to loosing a node or two. The second point however doesn't hold as strong today as it did when we made our decision last year. Netflix recently open-sourced a really good tool ( https://github.com/Netflix/Priam/wiki/Backups ) which will automate the back-up of Cassandra data to S3. I'd definitely recommend checking it out to see if it will help you with AWS backups / restores, we're currently looking at rolling it out. Hope that helps, Alex. On Wed, Jul 4, 2012 at 2:56 AM, Dan Foody dan.fo...@gmail.com wrote: Hi Alex, Can you share what replication factor you're running? And, are you using ephemeral disks or EBS volumes? Thanks! - Dan On Jul 3, 2012, at 5:52 PM, Alex Major wrote: Hi Mike, We've run a small (4 node) cluster in the EU region since September last year. We run across all 3 availability zones in the EU region, with 2 nodes in one AZ and then a further node in each AZ. The latency difference between running inside of and between AZ's has been minimal in our experience. It's only when we've gone cross-region that there's been latency problem. We temporarily ran a 9 node cluster across 3 regions, however even then using local quoram the latency was better than the standard datacenter - datacenter latency we're used to. EC2Snitch is definitely the way to go in favour of NTS in my opinion. NTS was a pain to get setup with the internal (private) IP address setup, so much so that we never got it safely replicating the data as we wanted. Alex. On Tue, Jul 3, 2012 at 2:16 PM, Michael Theroux mthero...@yahoo.comwrote: Hello, We are currently running a web application utilizing Cassandra on EC2. Given the recent outages experienced with Amazon, we want to consider expanding Cassandra across availability zones sooner rather than later. We are trying to determine the optimal way to deploy Cassandra in this deployment. We are researching the NetworkTopologyStrategy, and the EC2Snitch. We are also interested in providing a high level of read or write consistency, My understanding is that the EC2Snitch recognizes availability zones as racks, and regions as data-centers. This seems to be a common configuration. However, if we were to want to utilize queries with a READ or WRITE consistency of QUORUM, would there be a high possibility that the communication necessary to establish a quorum, across availability zones? My understanding is that the NetworkTopologyStrategy attempts to prefer replicas be stored on other racks within the datacenter, which would equate to other availability zones in EC2. This implies to me that in order to have the quorum of nodes necessary to achieve consistency, that Cassandra will communicate with nodes across availability zones. First, is my understanding correct? Second, given the high latency that can sometimes exists between availability zones, is this a problem, and instead we should treat availability zones as data centers? Ideally, we would be able to setup a situation where we could store replicas across availability zones in case of failure, but establish a high level of read or write consistency within a single availability zone. I appreciate your responses, Thanks,
Re: Expanding Cassandra on EC2 with consistency
Hi Mike, I'm not sure about all your questions, however you should take a look at LOCAL_QUORUM for your question about consistency level reads/writes. 2012/7/3 Michael Theroux mthero...@yahoo.com Hello, We are currently running a web application utilizing Cassandra on EC2. Given the recent outages experienced with Amazon, we want to consider expanding Cassandra across availability zones sooner rather than later. We are trying to determine the optimal way to deploy Cassandra in this deployment. We are researching the NetworkTopologyStrategy, and the EC2Snitch. We are also interested in providing a high level of read or write consistency, My understanding is that the EC2Snitch recognizes availability zones as racks, and regions as data-centers. This seems to be a common configuration. However, if we were to want to utilize queries with a READ or WRITE consistency of QUORUM, would there be a high possibility that the communication necessary to establish a quorum, across availability zones? My understanding is that the NetworkTopologyStrategy attempts to prefer replicas be stored on other racks within the datacenter, which would equate to other availability zones in EC2. This implies to me that in order to have the quorum of nodes necessary to achieve consistency, that Cassandra will communicate with nodes across availability zones. First, is my understanding correct? Second, given the high latency that can sometimes exists between availability zones, is this a problem, and instead we should treat availability zones as data centers? Ideally, we would be able to setup a situation where we could store replicas across availability zones in case of failure, but establish a high level of read or write consistency within a single availability zone. I appreciate your responses, Thanks, -Mike -- With kind regards, Robin Verlangen *Software engineer* * * W http://www.robinverlangen.nl E ro...@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Re: Expanding Cassandra on EC2 with consistency
Yes, I saw LOCAL_QUORUM. The definition I saw was: Ensure that the write has been written to ReplicationFactor / 2 + 1 nodes, within the local datacenter (requiresNetworkTopologyStrategy) This will allow a quorum within a datacenter. However, I think this means that if availability zones are racks, that the quorum would still be across availability zones. -Mike On Jul 3, 2012, at 9:22 AM, Robin Verlangen wrote: Hi Mike, I'm not sure about all your questions, however you should take a look at LOCAL_QUORUM for your question about consistency level reads/writes. 2012/7/3 Michael Theroux mthero...@yahoo.com Hello, We are currently running a web application utilizing Cassandra on EC2. Given the recent outages experienced with Amazon, we want to consider expanding Cassandra across availability zones sooner rather than later. We are trying to determine the optimal way to deploy Cassandra in this deployment. We are researching the NetworkTopologyStrategy, and the EC2Snitch. We are also interested in providing a high level of read or write consistency, My understanding is that the EC2Snitch recognizes availability zones as racks, and regions as data-centers. This seems to be a common configuration. However, if we were to want to utilize queries with a READ or WRITE consistency of QUORUM, would there be a high possibility that the communication necessary to establish a quorum, across availability zones? My understanding is that the NetworkTopologyStrategy attempts to prefer replicas be stored on other racks within the datacenter, which would equate to other availability zones in EC2. This implies to me that in order to have the quorum of nodes necessary to achieve consistency, that Cassandra will communicate with nodes across availability zones. First, is my understanding correct? Second, given the high latency that can sometimes exists between availability zones, is this a problem, and instead we should treat availability zones as data centers? Ideally, we would be able to setup a situation where we could store replicas across availability zones in case of failure, but establish a high level of read or write consistency within a single availability zone. I appreciate your responses, Thanks, -Mike -- With kind regards, Robin Verlangen Software engineer W http://www.robinverlangen.nl E ro...@us2.nl Disclaimer: The information contained in this message and attachments is intended solely for the attention and use of the named addressee and may be confidential. If you are not the intended recipient, you are reminded that the information remains the property of the sender. You must not use, disclose, distribute, copy, print or rely on this e-mail. If you have received this message in error, please contact the sender immediately and irrevocably delete this message and any copies.
Re: Expanding Cassandra on EC2 with consistency
Hi Mike, We've run a small (4 node) cluster in the EU region since September last year. We run across all 3 availability zones in the EU region, with 2 nodes in one AZ and then a further node in each AZ. The latency difference between running inside of and between AZ's has been minimal in our experience. It's only when we've gone cross-region that there's been latency problem. We temporarily ran a 9 node cluster across 3 regions, however even then using local quoram the latency was better than the standard datacenter - datacenter latency we're used to. EC2Snitch is definitely the way to go in favour of NTS in my opinion. NTS was a pain to get setup with the internal (private) IP address setup, so much so that we never got it safely replicating the data as we wanted. Alex. On Tue, Jul 3, 2012 at 2:16 PM, Michael Theroux mthero...@yahoo.com wrote: Hello, We are currently running a web application utilizing Cassandra on EC2. Given the recent outages experienced with Amazon, we want to consider expanding Cassandra across availability zones sooner rather than later. We are trying to determine the optimal way to deploy Cassandra in this deployment. We are researching the NetworkTopologyStrategy, and the EC2Snitch. We are also interested in providing a high level of read or write consistency, My understanding is that the EC2Snitch recognizes availability zones as racks, and regions as data-centers. This seems to be a common configuration. However, if we were to want to utilize queries with a READ or WRITE consistency of QUORUM, would there be a high possibility that the communication necessary to establish a quorum, across availability zones? My understanding is that the NetworkTopologyStrategy attempts to prefer replicas be stored on other racks within the datacenter, which would equate to other availability zones in EC2. This implies to me that in order to have the quorum of nodes necessary to achieve consistency, that Cassandra will communicate with nodes across availability zones. First, is my understanding correct? Second, given the high latency that can sometimes exists between availability zones, is this a problem, and instead we should treat availability zones as data centers? Ideally, we would be able to setup a situation where we could store replicas across availability zones in case of failure, but establish a high level of read or write consistency within a single availability zone. I appreciate your responses, Thanks, -Mike
