Re: EC2 node adding trouble

2011-05-26 Thread aaron morton
This is the *most* useful page on the wiki http://wiki.apache.org/cassandra/Operations Hope that helps. - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 27 May 2011, at 02:06, Marcus Bointon wrote: > On 26 May 2011, at 15:21, Sasha Do

Re: EC2 node adding trouble

2011-05-26 Thread aaron morton
This ticket may be just the ticket :) https://issues.apache.org/jira/browse/CASSANDRA-2452 Cheers - Aaron Morton Freelance Cassandra Developer @aaronmorton http://www.thelastpickle.com On 27 May 2011, at 01:16, Sasha Dolgy wrote: > As an aside, you can also use that command to

Re: EC2 node adding trouble

2011-05-26 Thread Marcus Bointon
On 26 May 2011, at 15:21, Sasha Dolgy wrote: > Turn the node off, remove the node from the ring using nodetool and > removetoken i've found this to be the best problem-free way. > Maybe it's better now ... > http://blog.sasha.dolgy.com/2011/03/apache-cassandra-nodetool.html So I'd need to ha

Re: EC2 node adding trouble

2011-05-26 Thread Sasha Dolgy
On Thu, May 26, 2011 at 3:12 PM, Marcus Bointon wrote: > I'd like to make sure I've got the right sequence of operations for adding a > node without downtime. If I'm going from 2 to 3 nodes: > > 1 Calculate new initial_token values using the python script > 2 Change token values in existing nodes

Re: EC2 node adding trouble

2011-05-26 Thread Sasha Dolgy
As an aside, you can also use that command to pull meta-data about instances in AWS. I have implemented this to maintain a list of seed nodes. This way, when a new instance is brought online, the default cassandra.yaml is `enhanced` to contain a dynamic list of valid seeds, proper hostname and a

Re: EC2 node adding trouble

2011-05-26 Thread Marcus Bointon
On 24 May 2011, at 23:58, Sameer Farooqui wrote: > So, once you know what token each of the 3 nodes should have, shut down the > first two nodes, change their tokens and add the correct token to the 3rd > node (in the YAML file). I'd like to make sure I've got the right sequence of operations f

Re: EC2 node adding trouble

2011-05-26 Thread Marcus Bointon
Thanks for all your helpful suggestions - I've now got it working. It was down to a combination of things. 1. A missing rule in a security group 2. A missing DNS name for the new node, so its default name was defaulting to localhost 3. Google DNS caching the failed DNS lookup for the full durati

Re: EC2 node adding trouble

2011-05-26 Thread Marcus Bointon
On 26 May 2011, at 00:17, aaron morton wrote: > I've seen discussion of using the EIP but I do not have direct experience. The idea is not to use the external IP, but the external DNS name because of this very useful trick (please excuse me if you already know this!): Say the DNS name of an el

Re: EC2 node adding trouble

2011-05-25 Thread Sasha Dolgy
All of the nodes are 0.8.x and no longer 0.7.2 ? It is peculiar. As another person noted, maybe reviewing the results of netstat -an on each node could help. I've had issues in the past with JMX -- thinking I had configured it for IP x.x.x.x and went crazy only to see that it was never configure

Re: EC2 node adding trouble

2011-05-25 Thread aaron morton
I've seen discussion of using the EIP but I do not have direct experience. Others may be able to provide more help. Previous discussion http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Does-anybody-have-experience-with-running-Cassandra-in-Amazon-s-Virtual-Private-Cloud-VPC-td6

Re: EC2 node adding trouble

2011-05-25 Thread mcasandra
ttp://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/EC2-node-adding-trouble-tp6399102p6403602.html Sent from the cassandra-u...@incubator.apache.org mailing list archive at Nabble.com.

Re: EC2 node adding trouble

2011-05-24 Thread Marcus Bointon
On 24 May 2011, at 23:58, Sameer Farooqui wrote: > Even with AutoBootstrap it is recommended that you always specify the > InitialToken on the new node because the picking of an initial token will > almost certainly result in an unbalanced ring. > > Right now, I'm afraid that if you simply copi

Re: EC2 node adding trouble

2011-05-24 Thread Marcus Bointon
On 25 May 2011, at 02:10, aaron morton wrote: > Check the listen_address and rpc_address in the yaml file for each node. I > think they are normally set to the private and public respectively. On all modes, listen_address and rpc_address are blank and 0.0.0.0 respectively, the seed node addres

Re: EC2 node adding trouble

2011-05-24 Thread aaron morton
Check the listen_address and rpc_address in the yaml file for each node. I think they are normally set to the private and public respectively. This may make your live easier http://www.datastax.com/dev/blog/setting-up-a-cassandra-cluster-with-the-datastax-ami Cheers - Aaron Mor

Re: EC2 node adding trouble

2011-05-24 Thread Sameer Farooqui
Even with AutoBootstrap it is recommended that you always specify the InitialToken on the new node because the picking of an initial token will almost certainly result in an unbalanced ring. Right now, I'm afraid that if you simply copied the YAML file from one of the two nodes to the 3rd node, th

Re: EC2 node adding trouble

2011-05-24 Thread Marcus Bointon
On 24 May 2011, at 19:33, Sameer Farooqui wrote: > What region and availability zones are the different nodes in? Are you using > EC2 Snitch? Did you set up the cluster using the Datastax AMI? The two existing ones are in us-east-1c and us-east-1d, the new one is in us-east-1c, so all same regi

Re: EC2 node adding trouble

2011-05-24 Thread Sameer Farooqui
What region and availability zones are the different nodes in? Are you using EC2 Snitch? Did you set up the cluster using the Datastax AMI? - Sameer On Tue, May 24, 2011 at 8:35 AM, Marcus Bointon wrote: > Hi, > > First time here. I'm having trouble adding a third node to an existing > 2-node r

EC2 node adding trouble

2011-05-24 Thread Marcus Bointon
Hi, First time here. I'm having trouble adding a third node to an existing 2-node ring (successfully upgraded from 0.72) running cassandra 0.8rc1 (successfully upgraded from 0.72) on ubuntu on EC2. Evidently the seed node is working as the second node is already talking to it, nodetool lists b