EC2 Snitch load imbalance
I have a setup with 6 cassandra nodes (1.2.18), using RandomPartition, not using vnodes -- this is a legacy cluster. We went from 3 nodes to 6 in the last few days to add capacity. However, there appears to be an imbalance: Datacenter: us-east == Replicas: 2 Address RackStatus State LoadOwns Token 113427455640312821154458202477256070484 x.x.x.73 1d Up Normal 154.64 GB 33.33% 85070591730234615865843651857942052863 x.x.x.2511a Up Normal 62.26 GB16.67% 28356863910078205288614550619314017621 x.x.x.238 1b Up Normal 243.7 GB50.00% 56713727820156410577229101238628035242 x.x.x.25 1a Up Normal 169.3 GB33.33% 210 x.x.x.162 1b Up Normal 118.24 GB 50.00% 141784319550391026443072753096570088105 x.x.x.208 1d Up Normal 226.85 GB 16.67% 113427455640312821154458202477256070484 What is the cause of this imbalance ? How can I rectify it ? Regards, Oleg
Re: EC2 Snitch load imbalance
Oleg, If you are running nodetool status, be sure to specify the keyspace also. If you don't specify the keyspace the results will be nonsense. https://issues.apache.org/jira/browse/CASSANDRA-7173 Regards, Mark On 28 October 2014 10:35, Oleg Dulin oleg.du...@gmail.com wrote: I have a setup with 6 cassandra nodes (1.2.18), using RandomPartition, not using vnodes -- this is a legacy cluster. We went from 3 nodes to 6 in the last few days to add capacity. However, there appears to be an imbalance: Datacenter: us-east == Replicas: 2 Address RackStatus State LoadOwns Token 113427455640312821154458202477256070484 x.x.x.73 1d Up Normal 154.64 GB 33.33% 85070591730234615865843651857942052863 x.x.x.2511a Up Normal 62.26 GB16.67% 28356863910078205288614550619314017621 x.x.x.238 1b Up Normal 243.7 GB50.00% 56713727820156410577229101238628035242 x.x.x.25 1a Up Normal 169.3 GB33.33% 210 x.x.x.162 1b Up Normal 118.24 GB 50.00% 141784319550391026443072753096570088105 x.x.x.208 1d Up Normal 226.85 GB 16.67% 113427455640312821154458202477256070484 What is the cause of this imbalance ? How can I rectify it ? Regards, Oleg
Re: EC2 Snitch load imbalance
Thanks Mark. The output in my original post is with keyspace specified. On 2014-10-28 12:00:15 +, Mark Reddy said: Oleg, If you are running nodetool status, be sure to specify the keyspace also. If you don't specify the keyspace the results will be nonsense. https://issues.apache.org/jira/browse/CASSANDRA-7173 Regards, Mark On 28 October 2014 10:35, Oleg Dulin oleg.du...@gmail.com wrote: I have a setup with 6 cassandra nodes (1.2.18), using RandomPartition, not using vnodes -- this is a legacy cluster. We went from 3 nodes to 6 in the last few days to add capacity. However, there appears to be an imbalance: Datacenter: us-east == Replicas: 2 Address Rack Status State Load Owns Token 113427455640312821154458202477256070484 x.x.x.73 1d Up Normal 154.64 GB 33.33% 85070591730234615865843651857942052863 x.x.x.251 1a Up Normal 62.26 GB 16.67% 28356863910078205288614550619314017621 x.x.x.238 1b Up Normal 243.7 GB 50.00% 56713727820156410577229101238628035242 x.x.x.25 1a Up Normal 169.3 GB 33.33% 210 x.x.x.162 1b Up Normal 118.24 GB 50.00% 141784319550391026443072753096570088105 x.x.x.208 1d Up Normal 226.85 GB 16.67% 113427455640312821154458202477256070484 What is the cause of this imbalance ? How can I rectify it ? Regards, Oleg
Re: EC2 Snitch load imbalance
On Tue, Oct 28, 2014 at 3:35 AM, Oleg Dulin oleg.du...@gmail.com wrote: I have a setup with 6 cassandra nodes (1.2.18), using RandomPartition, not using vnodes -- this is a legacy cluster. We went from 3 nodes to 6 in the last few days to add capacity. However, there appears to be an imbalance: The first step to check is if the ranges are even. Did you pick initial_token for these new nodes? =Rob
Ec2 snitch with network topology strategy
I was thinking of leveraging ec2 snitch. But my question is then how do I give replica placement options? Or can I give snitch as ec2snitch and write the nodes cassandra-topology.prop and in give locator strategy at time of creating keyspace as network topology strategy. But will it work? And those who are struggling to deploy cassandra with across ec2 regions. 1. approach is to use milind's patch, it works but has some limitation. https://issues.apache.org/jira/browse/CASSANDRA-2362 2. openvpn is a good option but neverthless is futile with encryption available in 0.8.0 cassandra 3. Vijay has come up with a patch and so far tested I have not seen any jerks. https://issues.apache.org/jira/browse/CASSANDRA-2452 - its marked to be there in 0.8.2 release. -pankaj -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Ec2-snitch-with-network-topology-strategy-tp6528188p6528188.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: Ec2 snitch with network topology strategy
Hmm... Just tested the config. It works, got confused with the options, my bad. On Wed, Jun 29, 2011 at 2:26 PM, pankajsoni0126 pankajsoni0...@gmail.comwrote: I was thinking of leveraging ec2 snitch. But my question is then how do I give replica placement options? Or can I give snitch as ec2snitch and write the nodes cassandra-topology.prop and in give locator strategy at time of creating keyspace as network topology strategy. But will it work? And those who are struggling to deploy cassandra with across ec2 regions. 1. approach is to use milind's patch, it works but has some limitation. https://issues.apache.org/jira/browse/CASSANDRA-2362 2. openvpn is a good option but neverthless is futile with encryption available in 0.8.0 cassandra 3. Vijay has come up with a patch and so far tested I have not seen any jerks. https://issues.apache.org/jira/browse/CASSANDRA-2452 - its marked to be there in 0.8.2 release. -pankaj -- View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Ec2-snitch-with-network-topology-strategy-tp6528188p6528188.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.
Re: EC2 Snitch
We are using this patch in our multi-region testing... yes this approach is going to be integrated into https://issues.apache.org/jira/browse/CASSANDRA-2491 once it is committed (you might want to wait for that). Yes this fix the Amazon infrastructure problems and it will automatically detect the DC and RAC only thing is that we have to allow IP's in the security groups to talk to each other's region. Regards, /VJ On Tue, May 10, 2011 at 6:19 PM, Sameer Farooqui cassandral...@gmail.comwrote: Has anybody successfully used EC2 Snitch for cross-region deployments on EC2? Brandon Williams has not recommended using this just yet, but I was curious if anybody is using it with 0.8.0. Also, the snitch just let's the cluster automatically discover what the different regions (aka data centers) and racks (aka availability zones) are, right? So, does this fix the Amazon NATed infrastructure problem where we can't use the Amazon external IP as the Cassandra listen address? Cassandra can only bind to addresses that are attached to the server and the Amazon external IP is NATed so Cassandra can't see it. The EC2-Snitch patch was kinda unclear on this: https://issues.apache.org/jira/browse/CASSANDRA-2452 Also, the nightly builds link to Cassandra seems to be down, has it been relocated? (click on Latest Builds/Hudson link on right): http://cassandra.apache.org/download/ This link is dead: http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/
EC2 Snitch
Has anybody successfully used EC2 Snitch for cross-region deployments on EC2? Brandon Williams has not recommended using this just yet, but I was curious if anybody is using it with 0.8.0. Also, the snitch just let's the cluster automatically discover what the different regions (aka data centers) and racks (aka availability zones) are, right? So, does this fix the Amazon NATed infrastructure problem where we can't use the Amazon external IP as the Cassandra listen address? Cassandra can only bind to addresses that are attached to the server and the Amazon external IP is NATed so Cassandra can't see it. The EC2-Snitch patch was kinda unclear on this: https://issues.apache.org/jira/browse/CASSANDRA-2452 Also, the nightly builds link to Cassandra seems to be down, has it been relocated? (click on Latest Builds/Hudson link on right): http://cassandra.apache.org/download/ This link is dead: http://hudson.zones.apache.org/hudson/job/Cassandra/lastSuccessfulBuild/artifact/cassandra/build/
Re: Cassandra Chef recipe and EC2 snitch
the link mentions an ec2snitch, but i don't. the cookbook can just generate property files for the propertysnitch (which i much prefer to the use of a dedicated snitch). the algorithm for decided locality is at least as dumb as the one in the rack-aware snitch. on the plus side, it's just a little ruby, so easily fixed! On Thu, Jul 22, 2010 at 10:08 AM, Allan Carroll alla...@gmail.com wrote: Hi all, I'm setting up a new cluster on EC2 for the first time and looking at the wiki cloud setup page (http://wiki.apache.org/cassandra/CloudConfig). There's a chef recipe linked there that mentions an ec2snitch. The link doesn't seem to go where it says it does. Does anyone know where those resources have gone or are they no longer available? Thanks -Allan
Cassandra Chef recipe and EC2 snitch
Hi all, I'm setting up a new cluster on EC2 for the first time and looking at the wiki cloud setup page (http://wiki.apache.org/cassandra/CloudConfig). There's a chef recipe linked there that mentions an ec2snitch. The link doesn't seem to go where it says it does. Does anyone know where those resources have gone or are they no longer available? Thanks -Allan
Re: Cassandra Chef recipe and EC2 snitch
You don't need the ec2snitch necessarily. AFAIK, It's meant to be a better way of detecting where your ec2 instances are. But, unless you're popping instances all the time, I don't think it's worth it. Check out the step-by-step guide on that same page. Pure EC2 api calls to setup your cluster. You can also use rackaware-ness in EC2. Just add in the PropertyFile endpoint and put your rack file in /etc/cassandra/rack.properties. Dave Viner On Thu, Jul 22, 2010 at 10:08 AM, Allan Carroll alla...@gmail.com wrote: Hi all, I'm setting up a new cluster on EC2 for the first time and looking at the wiki cloud setup page (http://wiki.apache.org/cassandra/CloudConfig). There's a chef recipe linked there that mentions an ec2snitch. The link doesn't seem to go where it says it does. Does anyone know where those resources have gone or are they no longer available? Thanks -Allan
