Running across multiple EC2 regions...
Hello, Me and some of my colleagues are about to start some experiments running Cassandra across EC2 regions using virtual networks and have some questions about how this is going to work. I've read these threads about patching the .yaml file to bind to the Listen address to the public IP... http://mail-archives.apache.org/mod_mbox/cassandra-user/201104.mbox/%3CBANLkTikWGOtWkBOBAs+ibq5voSmjLm=gQQ@mail.gmail.http://mail-archives.apache.org/mod_mbox/cassandra-user/201104.mbox/%3CBANLkTikWGOtWkBOBAs+ibq5voSmjLm=g...@mail.gmail.com%3E com%3Ehttp://mail-archives.apache.org/mod_mbox/cassandra-user/201104.mbox/%3CBANLkTikWGOtWkBOBAs+ibq5voSmjLm=g...@mail.gmail.com%3E http://mail-archives.apache.org/mod_mbox/cassandra-user/201103.mbox/%3CAANLkTikWsuUOjEU228niWi0iDTAO5J=5wO=i=hg33...@mail.gmail.com%3E And this EC2 Snitch patch that lets it work across regions... https://issues.apache.org/jira/browse/CASSANDRA-2452 I'm pretty familiar with EC2 networking, but not very familiar how Cassandra will use the RPC and Listen ports with this Snitch. So, my question is: If we use the EC2 Snitch patch to set up across regions, will the instance's private IP/interface *ever* be used? Or will all traffic always go in and out of the public interface?? Using the public interface is slower and more expensive that the private interface. What I'm trying to do is set up a virtual network that lets all the nodes use private IPs, but can still communicate across regions. We're going to try this with a virtual network as well as with the EC2 Snitch to see how things compare. Being able to use the EC2 private interface is going to make big difference. Thanks in advance. CM
Re: Running across multiple EC2 regions...
Thanks Vijay, that helps a lot. FYI, I did read the comments but didn't understand what I was reading since I don't know what IECSC is?? Googled it but still came up empty. What is this?? Sorry, if this is obvious, but I'm pretty new to all this Thanks CM On Fri, Jul 29, 2011 at 3:18 PM, Vijay vijay2...@gmail.com wrote: So, my question is: If we use the EC2 Snitch patch to set up across regions, will the instance's private IP/interface *ever* be used? Or will all traffic always go in and out of the public interface?? Using the public interface is slower and more expensive that the private interface. If you use 2452, communication within a region is via private ip and communication between the regions are public (Handshaking or intial communication will still be via public ip). In EC2 they dont have 2 interface but they nat the public IP even then this patch will do the right thing for you. There is comments in the patch + * 1) Snitch will automatically set the public IP by querying the AWS API + * + * 2) Snitch will set the private IP as a Gossip application state. + * + * 3) Snitch implements IESCS and will reset the connection if it is within the + * same region to communicate via private IP. + * + * Implements Ec2Snitch to inherit its functionality and extend it for + * Multi-Region. + * + * Operational: All the nodes in this cluster needs to be able to (modify the + * Security group settings in AWS) communicate via Public IP's. Regards, /VJ On Fri, Jul 29, 2011 at 2:11 PM, Chris Marino ch...@vsider.com wrote: Hello, Me and some of my colleagues are about to start some experiments running Cassandra across EC2 regions using virtual networks and have some questions about how this is going to work. I've read these threads about patching the .yaml file to bind to the Listen address to the public IP... http://mail-archives.apache.org/mod_mbox/cassandra-user/201104.mbox/%3CBANLkTikWGOtWkBOBAs+ibq5voSmjLm=gQQ@mail.gmail.http://mail-archives.apache.org/mod_mbox/cassandra-user/201104.mbox/%3CBANLkTikWGOtWkBOBAs+ibq5voSmjLm=g...@mail.gmail.com%3E com%3Ehttp://mail-archives.apache.org/mod_mbox/cassandra-user/201104.mbox/%3CBANLkTikWGOtWkBOBAs+ibq5voSmjLm=g...@mail.gmail.com%3E http://mail-archives.apache.org/mod_mbox/cassandra-user/201103.mbox/%3CAANLkTikWsuUOjEU228niWi0iDTAO5J=5wO=i=hg33...@mail.gmail.com%3E And this EC2 Snitch patch that lets it work across regions... https://issues.apache.org/jira/browse/CASSANDRA-2452 I'm pretty familiar with EC2 networking, but not very familiar how Cassandra will use the RPC and Listen ports with this Snitch. So, my question is: If we use the EC2 Snitch patch to set up across regions, will the instance's private IP/interface *ever* be used? Or will all traffic always go in and out of the public interface?? Using the public interface is slower and more expensive that the private interface. What I'm trying to do is set up a virtual network that lets all the nodes use private IPs, but can still communicate across regions. We're going to try this with a virtual network as well as with the EC2 Snitch to see how things compare. Being able to use the EC2 private interface is going to make big difference. Thanks in advance. CM
Different Load values after stress test runs....
Hi, we're running some performance tests against some clusters and I'm curious about some of the numbers I see. I'm running the stress test against two identically configured clusters, but after I run at stress test, I get different Load values across the clusters? The difference between the two clusters is that one uses standard EC2 interfaces, but the other runs on a virtual network. Are these differences indicating something that I should be aware of?? Here is a sample of the kinds of results I'm seeing. Address DC RackStatus State LoadOwns Token 12760588759xxx 10.0.0.17 DC1 RAC1Up Normal 94 MB 25.00% 0 10.0.0.18 DC1 RAC1Up Normal 104.52 MB 25.00% 42535295865xxx 10.0.0.19 DC1 RAC1Up Normal 78.58 MB 25.00% 85070591730xxx 10.0.0.20 DC1 RAC1Up Normal 78.58 MB 25.00% 12760588759xxx Address DC RackStatus State LoadOwns Token 12760588759xxx 10.120.35.52DC1 RAC1Up Normal 103.74 MB 25.00% 0 10.120.6.124DC1 RAC1Up Normal 118.99 MB 25.00% 42535295865xxx 10.127.90.142 DC1 RAC1Up Normal 104.26 MB 25.00% 85070591730xxx 10.94.69.237DC1 RAC1Up Normal 75.74 MB 25.00% 12760588759xxx The first cluster with the vNet (10.0.0.0/28 addresses) consistently show smaller Load values. The total Load of 355MB vs. 402MB with native EC2 interfaces?? Is a total Load value even meaningful?? The stress test is the very first thing that's run against the clusters. [I'm also a little puzzled that these numbers are not uniform within the clusters, but I suspect that's because the stress test is using a key distribution that is Gaussian. I'm not 100% sure of this either since I've seen conflicting documentation. Haven't tried 'random' keys, but I presume that would change them to be uniform] Except for these curious Load numbers, things seem to be running just fine. Getting good fast results. Over 10 iterations I'm getting more than 10-12K inserts per sec. (default values for the stress test). Should I expect the Load to be the same across different clusters?? What might explain the differences I'm seeing??? Thanks in advance. CM
Cassandra performance on a virtual network....
Hello everyone, I wanted to tell you about some performance benchmarking we have done with Cassandra running in EC2 on a virtual network. The purpose of the experiment was to see how running Cassandra on a virtual network could simplify operational complexity and to determine the performance impact, relative to native interfaces. The summary results for running a 4 node cluster are: Cassandra Performance on vCider Virtual Network Replication Factor 1 32 64 128 192 256 byte cols. v. Unencrypted: -8.2% 0.8% -2.3%-2.3% -6.7% v. Encrypted: 63.8% 55.4% 60.0% 53.9% 61.7% v. Node Only Encryption: -0.7% -5.0%1.9%5.4%4.7% Replication Factor 3 32 64128 192 256 byte cols v. Unencrypted: -4.5% -4.7% -5.8% -4.5%-1.5% v. Encrypted: 31.5% 29.6% 31.4% 27.3% 29.9% v. Node Only Encryption: 3.8% 3.9% 6.1%8.3% 4.0% There is tremendous EC2 performance variability and our experiments tried to adjust for that by running 10 trials for each column size and averaging them. Averaged across all column widths, the performance was: Replication Factor 1 v. Unencrypted: -3.7% v. Encrypted: +59% v. Node Only Encryption: +1.3% Replication Factor 3 v. Unencrypted: -4.2% v. Encrypted: +30% v. Node Only Encryption: +5.2% As you might expect, the performance while running on a virtual network was slower than running on the native interfaces. However, when you encrypt communications (both node and client) the performance of the virtual network was faster by nearly 60% (30% with R3). Since this measurement is primarily an indication of the client encryption performance, we also measured performance of the somewhat unrealistic configuration when only node communications were encrypted. Here the virtual network performed better as well. The overall decrease performance loss -3.7% to -4.2% for un-encrypted R1 v. R3 is understandable since R3 is more network intensive than R1. However, since the virtual network performs encryption in the kernel (which seems to be faster than what Cassandra can do natively) when encryption is turned on, the performance gains are greater with R3 since more data needs to be encrypted. We ran the tests using the Cassandra stress test tool across a range of column widths, replication strategies and consistency levels (One, Quourm). We used OpenVPN for client encryption. The complete test results are attached. I’m going to write up a more complete analysis of these results, but wanted to share them with you to see if there was anything obvious that we overlooked. We are currently running experiments against clusters running in multiple EC2 regions. We expect similar performance characteristics across regions, but with the added benefit of not needing to fuss with the EC2 snitch. The virtual network lets you assign your own private IPs for all Cassandra interfaces so the standard Snitch can be used everywhere. If you're running Cassandra in EC2 (or any other public cloud) and want encrypted communications, running on virtual network is a clear winner. Here, not only is it 30-60% faster, but you don’t have to bother with the point-to-point configurations of setting up a third party encryption technique. Since these run in user space, its not surprising that dramatic performance gains can be achieved with the kernel based approach of the virtual network. When we’re done will put everything in a public repo that includes all Puppet configuration modules as well as collection of scripts that automate nearly all of the testing. I hope to have that in the next week or so, but wanted to get some of these single region results out there in advance. If you are interested, you can learn more about the vCider virtual network at www.vcider.com Let me know if you have any questions. CM vCider.Cassandra.benchmarks.pdf Description: Adobe PDF document
Local Quorum Performance...
Hi, I have a question about what to expect when running a cluster across datacenters with Local Quorum consistency. My simplistic assumption is that the performance of an 8 node cluster split across 2 data centers and running with local quorum would perform roughly the same as a 4 node cluster in one data center. I'm 95% certain we've set up the keyspace so that the entire range is in one datacenter and the client is local. I see the keyspace split across all the local nodes, with remote nodes owning 0%. Yet when I run the stress tests against this configuration with local quorum, I see dramatically different results from when I ran the same tests against a 4 node cluster. I'm still 5% unsure of this because the documentation on how to configure this is pretty thin. My understanding of Local Quorum was that once the data was written to a local quorum, the commit would complete. I also believed that this would eliminate any WAN latency required for replication to the other DC. It not just that the split cluster runs slower, its also that there is enormous variability in identical tests. Sometimes by a factor of 2 or more. It seems as though the WAN latency is not only impacting performance, but that it's also introducing a wide variation on overally performance. Should WAN latency be completely hidden with local quorum? Or are there second order issues involved that will impact performance?? I'm running in EC2 across us-east/west regions. I already know how unpredictable EC2 performance can be, but what I'm seeing with here is far beyond normal.performance variability for EC2 Is there something obvious that I'm missing that would explain why the results are so different?? Here's the config when we run a 2x2 cluster: Address DC RackStatus State Load OwnsToken 85070591730234615865843651857942052865 192.168.2.1 us-east 1b Up Normal 25.26 MB 50.00% 0 192.168.2.6 us-west 1c Up Normal 12.68 MB 0.00% 1 192.168.2.2 us-east 1b Up Normal 12.56 MB 50.00% 85070591730234615865843651857942052864 192.168.2.7 us-west 1c Up Normal 25.48 MB 0.00% 85070591730234615865843651857942052865 Thanks in advance. CM
Re: Local Quorum Performance...
Anthony, We used the Ec2Snitch for one sets of runs, but for another set we're using PropertyFileSnitch. With the PropertyFileSnitch we see: Address DC RackStatus State Load OwnsToken 85070591730234615865843651857942052865 192.168.2.1 us-east 1b Up Normal 60.59 MB 50.00% 0 192.168.2.6 us-west 1c Up Normal 26.5 MB 0.00% 1 192.168.2.2 us-east 1b Up Normal 29.86 MB 50.00% 85070591730234615865843651857942052864 192.168.2.7 us-west 1c Up Normal 60.63 MB 0.00% 85070591730234615865843651857942052865 While with the EC2Snitch wwe see: Address DC RackStatus State Load OwnsToken 85070591730234615865843651857942052865 107.20.68.176 us-east 1b Up Normal 59.95 MB 50.00% 0 204.236.179.193 us-west 1c Up Normal 53.67 MB 0.00% 1 184.73.133.171 us-east 1b Up Normal 60.65 MB 50.00% 85070591730234615865843651857942052864 204.236.166.4 us-west 1c Up Normal 26.33 MB 0.00% 85070591730234615865843651857942052865 What also strange is that the Load on the nodes changes as well. For example, node 204.236.166.4 sometimes is very low (~26KB), other times its closer to 30MB. We see the same kind of variability in both clusters. For both clusters, we're running stress tests with the following options: --consistency-level=LOCAL_QUORUM --threads=4 --replication-strategy=NetworkTopologyStrategy --strategy-properties=us-east:2,us-west:2 --column-size=128 --keep-going --num-keys=10 -r Any clues to what is going on here are greatly appreciated. Thanks CM On Sat, Sep 17, 2011 at 12:15 PM, Ikeda Anthony anthony.ikeda@gmail.com wrote: What snitch do you have configured? We typically see a proper spread of data across all our nodes equally. Anthony On 17/09/2011, at 10:06 AM, Chris Marino wrote: Hi, I have a question about what to expect when running a cluster across datacenters with Local Quorum consistency. My simplistic assumption is that the performance of an 8 node cluster split across 2 data centers and running with local quorum would perform roughly the same as a 4 node cluster in one data center. I'm 95% certain we've set up the keyspace so that the entire range is in one datacenter and the client is local. I see the keyspace split across all the local nodes, with remote nodes owning 0%. Yet when I run the stress tests against this configuration with local quorum, I see dramatically different results from when I ran the same tests against a 4 node cluster. I'm still 5% unsure of this because the documentation on how to configure this is pretty thin. My understanding of Local Quorum was that once the data was written to a local quorum, the commit would complete. I also believed that this would eliminate any WAN latency required for replication to the other DC. It not just that the split cluster runs slower, its also that there is enormous variability in identical tests. Sometimes by a factor of 2 or more. It seems as though the WAN latency is not only impacting performance, but that it's also introducing a wide variation on overally performance. Should WAN latency be completely hidden with local quorum? Or are there second order issues involved that will impact performance?? I'm running in EC2 across us-east/west regions. I already know how unpredictable EC2 performance can be, but what I'm seeing with here is far beyond normal.performance variability for EC2 Is there something obvious that I'm missing that would explain why the results are so different?? Here's the config when we run a 2x2 cluster: Address DC RackStatus State LoadOwns Token 85070591730234615865843651857942052865 192.168.2.1 us-east 1b Up Normal 25.26 MB50.00% 0 192.168.2.6 us-west 1c Up Normal 12.68 MB0.00% 1 192.168.2.2 us-east 1b Up Normal 12.56 MB50.00% 85070591730234615865843651857942052864 192.168.2.7 us-west 1c Up Normal 25.48 MB0.00% 85070591730234615865843651857942052865 Thanks in advance. CM
Cassandra performance benchmark on a virtual network....
Hi, I posted this message last month and I promised to put up a public repository with all of our configuration details. You can find it at https://github.com/vCider/BenchmarksCassandra We've built an completely automated system with Puppet that configures EC2 instances with Cassandra as well as the virtual network and then runs the benchmarks. Feel free to give it a try. I'm very interested to see if anyone can reproduce our results. We spent some time documenting what we did, but there may be some gaps. If you have any problems send an email to cassan...@vcider.com Thanks CM -- Forwarded message -- From: Chris Marino ch...@vcider.com Date: Mon, Sep 12, 2011 at 4:23 PM Subject: Cassandra performance on a virtual network To: user@cassandra.apache.org Hello everyone, I wanted to tell you about some performance benchmarking we have done with Cassandra running in EC2 on a virtual network. The purpose of the experiment was to see how running Cassandra on a virtual network could simplify operational complexity and to determine the performance impact, relative to native interfaces. The summary results for running a 4 node cluster are: Cassandra Performance on vCider Virtual Network Replication Factor 1 32 64 128 192 256 byte cols. v. Unencrypted: -8.2% 0.8% -2.3%-2.3% -6.7% v. Encrypted: 63.8% 55.4% 60.0% 53.9% 61.7% v. Node Only Encryption: -0.7% -5.0%1.9%5.4%4.7% Replication Factor 3 32 64128 192 256 byte cols v. Unencrypted: -4.5% -4.7% -5.8% -4.5%-1.5% v. Encrypted: 31.5% 29.6% 31.4% 27.3% 29.9% v. Node Only Encryption: 3.8% 3.9% 6.1%8.3% 4.0% There is tremendous EC2 performance variability and our experiments tried to adjust for that by running 10 trials for each column size and averaging them. Averaged across all column widths, the performance was: Replication Factor 1 v. Unencrypted:-3.7% v. Encrypted: +59% v. Node Only Encryption: +1.3% Replication Factor 3 v. Unencrypted:-4.2% v. Encrypted:+30% v. Node Only Encryption: +5.2% As you might expect, the performance while running on a virtual network was slower than running on the native interfaces. However, when you encrypt communications (both node and client) the performance of the virtual network was faster by nearly 60% (30% with R3). Since this measurement is primarily an indication of the client encryption performance, we also measured performance of the somewhat unrealistic configuration when only node communications were encrypted. Here the virtual network performed better as well. The overall decrease performance loss -3.7% to -4.2% for un-encrypted R1 v. R3 is understandable since R3 is more network intensive than R1. However, since the virtual network performs encryption in the kernel (which seems to be faster than what Cassandra can do natively) when encryption is turned on, the performance gains are greater with R3 since more data needs to be encrypted. We ran the tests using the Cassandra stress test tool across a range of column widths, replication strategies and consistency levels (One, Quourm). We used OpenVPN for client encryption. The complete test results are attached. I’m going to write up a more complete analysis of these results, but wanted to share them with you to see if there was anything obvious that we overlooked. We are currently running experiments against clusters running in multiple EC2 regions. We expect similar performance characteristics across regions, but with the added benefit of not needing to fuss with the EC2 snitch. The virtual network lets you assign your own private IPs for all Cassandra interfaces so the standard Snitch can be used everywhere. If you're running Cassandra in EC2 (or any other public cloud) and want encrypted communications, running on virtual network is a clear winner. Here, not only is it 30-60% faster, but you don’t have to bother with the point-to-point configurations of setting up a third party encryption technique. Since these run in user space, its not surprising that dramatic performance gains can be achieved with the kernel based approach of the virtual network. When we’re done will put everything in a public repo that includes all Puppet configuration modules as well as collection of scripts that automate nearly all of the testing. I hope to have that in the next week or so, but wanted to get some of these single region results out there in advance. If you are interested, you can learn more about the vCider virtual network at www.vcider.com Let me know if you have any questions. CM
Re: Cassandra performance question
We did some benchmarking as well. http://blog.vcider.com/2011/09/virtual-networks-can-run-cassandra-up-to-60-faster/ Although we were primarily interested in the networking issues CM On Fri, Dec 30, 2011 at 12:08 PM, Jeremy Hanna jeremy.hanna1...@gmail.comwrote: This might be helpful: http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html On Dec 30, 2011, at 1:59 PM, Dom Wong wrote: Hi, could anyone tell me whether this is possible with Cassandra using an appropriately sized EC2 cluster. 100,000 clients writing 50k each to their own specific row at 5 second intervals?
Re: Cassandra performance question
Hi Jonathan, yes, when I say 'node encryption' I mean inter-Cassandra node encryption. When I say 'client encryption' I mean encrypted traffic from the Cassandra nodes to the clients. For these benchmarks we used the stress test client load generator. We ran test with no encryption, then with 'node only' and then again with both node and client encryption. Looking at the post again right now I realize that I never mention the client encryption technique. My mistake. We just used OpenVPN. We set it up on the client stress test machine and set up individual VPNs to each Cassandra node in the cluster. I tired to make it clear in my post where the performance gains were coming from. Didn't mean to confuse anyone. That said, if you're running in a public cloud and you want to encrypt, you've got to tackle the client traffic encryption problem as well so it seemed like performance gains that most could expect. CM On Mon, Jan 23, 2012 at 7:46 PM, Jonathan Ellis jbel...@gmail.com wrote: Can you elaborate on to what exactly you were testing on the Cassandra side? It sounds like what this post refers to as node encryption corresponds to enabling internode_encryption: all, but I couldn't guess what your client encryption is since Cassandra doesn't support that out of the box yet. (Which is highly relevant since that's where most of the slowdown you observed comes from.) On Fri, Dec 30, 2011 at 2:11 PM, Chris Marino ch...@vcider.com wrote: We did some benchmarking as well. http://blog.vcider.com/2011/09/virtual-networks-can-run-cassandra-up-to-60-faster/ Although we were primarily interested in the networking issues CM On Fri, Dec 30, 2011 at 12:08 PM, Jeremy Hanna jeremy.hanna1...@gmail.com wrote: This might be helpful: http://techblog.netflix.com/2011/11/benchmarking-cassandra-scalability-on.html On Dec 30, 2011, at 1:59 PM, Dom Wong wrote: Hi, could anyone tell me whether this is possible with Cassandra using an appropriately sized EC2 cluster. 100,000 clients writing 50k each to their own specific row at 5 second intervals? -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Mixing Ec2MultiregionSnitch with private network
Hi Patrick, I'm not sure if it's doable, but I can tell you for sure that there are lots differences in the way the networks will need to be set up. If you've got to secure client traffic, it's going to get even more complicated with encrypted traffic, etc. We did some performance testing and configuration testing with Cassandra across regions using a virtual network (my company's product). Have a look at what we did. I think when you add in your own datacenter, things are going to get even more complicated. One of the nice things about using a virtual network in EC2 is that you can set up multiple network interfaces so you don't have to use the the multi-region snitch. These interfaces are also clever about using the real and the NAT'ed EC2 interfaces for cluster traffic (better performance and $0 EC2 data bandwidth costs), so things can be set up just like in your own datacetner without worrying about EC2's public/private IPs, NATing, etc. You can read about what we did on our blog. http://blog.vcider.com/2011/09/running-cassandra-on-a-virtual-network-in-ec2/ and http://blog.vcider.com/2011/09/virtual-networks-can-run-cassandra-up-to-60-faster/ Let me know if you have any questions. CM On Mon, Jun 4, 2012 at 3:27 PM, Patrick Lu kuma...@hotmail.com wrote: Hi All, Does anyone have experience on Cassandra deployment mixing with EC2 and own data center? We plan to use ec2multiregionsnitch to build a Cassandra cluster across EC2 regions, and the same time to have a couple nodes (in the cluster) sitting in our own data center. Any comment whether it’s doable? Thanks. Patrick.
