Re: GossipingPropertyFileSnitch vs Ec2Snitch
On Fri, Aug 21, 2015 at 2:37 AM, Alain RODRIGUEZ arodr...@gmail.com wrote: So what would be the procedure to go from EC2Snitch to GPFS ? I see 2 possibilities and I am not sure of the result of each of these: - Node by node, on a rolling restart way, and making sure that you use the former DC names and racks. Copying the EC2 network. Would this be doable or would that need a full cluster restart at once ? - Add the new DC using directly GPFS on all the node without modifying the old one (hybride: 1 DC EC2Snitch, 1 DC GPFS), again copying the EC2 DCs and racks. Would this work ? I believe both of these would work. If you have the extra nodes I probably would do the additional-DC method. I also recommend : 1) get a list of one or more partition keys for all ranges in your cluster 2) test them with nodetool getendpoints before changing snitch 3) test them with nodetool getendpoints after changing snitch Also, Is this need of mirroring the exact DC and racks name mandatory in those 2 procedures as I think it is ? In the spoof-ec2snitch method, yes. If you don't, Very Bad Things will happen. In the additional-DC method, it doesn't matter what you call the new DC or racks, just that they are configured properly. =Rob
Re: GossipingPropertyFileSnitch vs Ec2Snitch
Hi guys, I am also wondering about this kind of stuff, since we are going out of AWS and were using EC2Snitch. I felt free to write it here, since it looks linked to previous comments. @John, we have been using EC2Snitch for many years, and it just works fine ! Plus, we don't have to keep the GPFS file up to date every time we add a node or remove one. EC2Snitch is quite strait-forward and easy to use. The first issue we have today, is indeed to add a new DC from outside AWS. I am trying to find out how to do that right now. If we figure out that this is easy, you can perfectly go with EC2Snitch imho. So what would be the procedure to go from EC2Snitch to GPFS ? I see 2 possibilities and I am not sure of the result of each of these: - Node by node, on a rolling restart way, and making sure that you use the former DC names and racks. Copying the EC2 network. Would this be doable or would that need a full cluster restart at once ? - Add the new DC using directly GPFS on all the node without modifying the old one (hybride: 1 DC EC2Snitch, 1 DC GPFS), again copying the EC2 DCs and racks. Would this work ? Also, Is this need of mirroring the exact DC and racks name mandatory in those 2 procedures as I think it is ? I think this worth a good testing anyway. C*heers, Alain 2015-08-03 22:04 GMT+02:00 John Wong gokoproj...@gmail.com: Thanks Rob. But even if we do move out of EC2 in X years, unlike partitioner, I don't think there is much to do to switch snitch. Yes, I am evaluating if there is actually good benefit to use EC2. John On Mon, Aug 3, 2015 at 1:54 PM, Robert Coli rc...@eventbrite.com wrote: On Sun, Aug 2, 2015 at 6:22 PM, John Wong gokoproj...@gmail.com wrote: What other benefits can Ec2Snitch provide in a single-region, multi-az AWS deployment besides automatically setting dc and rack for you as the snitch reads from EC2 metadata. Obviously there is a concern with what if there is something wrong with the metadata server... GPFS is broadly similar to EC2Snitch/EC2MRSnitch, but is usable if you for example have a DR site outside of EC2. In general I'd probably suggest using GPFS unless you're really sure you'll always be in EC2, forever and always. =Rob
Re: GossipingPropertyFileSnitch vs Ec2Snitch
On Sun, Aug 2, 2015 at 6:22 PM, John Wong gokoproj...@gmail.com wrote: What other benefits can Ec2Snitch provide in a single-region, multi-az AWS deployment besides automatically setting dc and rack for you as the snitch reads from EC2 metadata. Obviously there is a concern with what if there is something wrong with the metadata server... GPFS is broadly similar to EC2Snitch/EC2MRSnitch, but is usable if you for example have a DR site outside of EC2. In general I'd probably suggest using GPFS unless you're really sure you'll always be in EC2, forever and always. =Rob
Re: GossipingPropertyFileSnitch vs Ec2Snitch
Thanks Rob. But even if we do move out of EC2 in X years, unlike partitioner, I don't think there is much to do to switch snitch. Yes, I am evaluating if there is actually good benefit to use EC2. John On Mon, Aug 3, 2015 at 1:54 PM, Robert Coli rc...@eventbrite.com wrote: On Sun, Aug 2, 2015 at 6:22 PM, John Wong gokoproj...@gmail.com wrote: What other benefits can Ec2Snitch provide in a single-region, multi-az AWS deployment besides automatically setting dc and rack for you as the snitch reads from EC2 metadata. Obviously there is a concern with what if there is something wrong with the metadata server... GPFS is broadly similar to EC2Snitch/EC2MRSnitch, but is usable if you for example have a DR site outside of EC2. In general I'd probably suggest using GPFS unless you're really sure you'll always be in EC2, forever and always. =Rob
GossipingPropertyFileSnitch vs Ec2Snitch
Hi Based on * http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html * http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchGossipPF_c.html What other benefits can Ec2Snitch provide in a single-region, multi-az AWS deployment besides automatically setting dc and rack for you as the snitch reads from EC2 metadata. Obviously there is a concern with what if there is something wrong with the metadata server... Thanks. John
Re: EC2snitch in AWS
By the way, if you need multiple DC on the same region, see dc-suffix in cassandra-rackdc.properties. C*heers, Alain 2015-05-29 11:34 GMT+02:00 Kaushal Shriyan kaushalshri...@gmail.com: Thanks Everyone. I will go through all the links and suggestions and ask here if I have further questions. Regards, Kaushal On 27 May 2015 22:44, Ali Akhtar ali.rac...@gmail.com wrote: What details specifically do you mean? I wrote this bash script which is what I've been using for installing cassandra 2.0.xx on AWS: https://gist.github.com/aliakhtar/3649e412787034156cbb On Wed, May 27, 2015 at 9:31 PM, Kaushal Shriyan kaushalshri...@gmail.com wrote: Hi, Can somebody please share me details about setting up of EC2snitch in AWS single region which has availability zone 1a and 1b? I am using Cassandra version 1.2.19 in the setup. I would appreciate your help. Regards, Kaushal
Re: EC2snitch in AWS
Thanks Everyone. I will go through all the links and suggestions and ask here if I have further questions. Regards, Kaushal On 27 May 2015 22:44, Ali Akhtar ali.rac...@gmail.com wrote: What details specifically do you mean? I wrote this bash script which is what I've been using for installing cassandra 2.0.xx on AWS: https://gist.github.com/aliakhtar/3649e412787034156cbb On Wed, May 27, 2015 at 9:31 PM, Kaushal Shriyan kaushalshri...@gmail.com wrote: Hi, Can somebody please share me details about setting up of EC2snitch in AWS single region which has availability zone 1a and 1b? I am using Cassandra version 1.2.19 in the setup. I would appreciate your help. Regards, Kaushal
Re: EC2snitch in AWS
It's pretty straightforward - there is documentation here
http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html.
Simply set endpoint_snitch: Ec2Snitch in cassandra.yaml when creating the
cluster and create keyspaces with the necessary NetworkTopologyStrategy.
The only real gotcha that I've run into is that the datacenter
(cassandra) and regions (AWS) match up 1:1 except for us-east-1 (it's
just us-east in the Ec2Snitch.)
Below are some examples that specify that data for that keyspace will be
present on 3 nodes. See consistency documentation
http://docs.datastax.com/en/cassandra/2.0/cassandra/dml/dml_config_consistency_c.html
for
how to tune your clients to meet certain consistency guarantees.
CREATE KEYSPACE ExampleKeyspace WITH REPLICATION = { 'class' :
'NetworkTopologyStrategy', 'us-east' : 3 };
CREATE KEYSPACE ExampleKeyspace WITH REPLICATION = { 'class' :
'NetworkTopologyStrategy', 'us-west-2' : 3 };
Jared Biel | Bolder Thinking | 701-205-3153 | jared.b...@bolderthinking.com
On 27 May 2015 at 16:31, Kaushal Shriyan kaushalshri...@gmail.com wrote:
Hi,
Can somebody please share me details about setting up of EC2snitch in AWS
single region which has availability zone 1a and 1b? I am using Cassandra
version 1.2.19 in the setup.
I would appreciate your help.
Regards,
Kaushal
Re: EC2snitch in AWS
What details specifically do you mean? I wrote this bash script which is what I've been using for installing cassandra 2.0.xx on AWS: https://gist.github.com/aliakhtar/3649e412787034156cbb On Wed, May 27, 2015 at 9:31 PM, Kaushal Shriyan kaushalshri...@gmail.com wrote: Hi, Can somebody please share me details about setting up of EC2snitch in AWS single region which has availability zone 1a and 1b? I am using Cassandra version 1.2.19 in the setup. I would appreciate your help. Regards, Kaushal
Re: EC2snitch in AWS
Hi Kaushal, Here is the reference, http://docs.datastax.com/en/cassandra/2.0/cassandra/architecture/architectureSnitchEC2_t.html On Wed, May 27, 2015 at 9:31 AM, Kaushal Shriyan kaushalshri...@gmail.com wrote: Hi, Can somebody please share me details about setting up of EC2snitch in AWS single region which has availability zone 1a and 1b? I am using Cassandra version 1.2.19 in the setup. I would appreciate your help. Regards, Kaushal -- Arun Senior Hadoop/Cassandra Engineer Cloudwick 2014 Data Impact Award Winner (Cloudera) http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html
EC2snitch in AWS
Hi, Can somebody please share me details about setting up of EC2snitch in AWS single region which has availability zone 1a and 1b? I am using Cassandra version 1.2.19 in the setup. I would appreciate your help. Regards, Kaushal
Re: making sure 1 copy per availability zone(rack) using EC2Snitch
On Mon, Sep 9, 2013 at 11:21 AM, rash aroskar rashmi.aros...@gmail.comwrote: Are you suggesting deploying 1.2.9 only if using Cassandra DC outside of EC2 or if I wish to use rack replication at all? 1) use 1.2.9 no matter what, instead of 1.2.5 2) if only *ever* will have clusters in EC2, EC2Snitch is fine, but read and understand CASSANDRA-3810, especially if not using vnodes 3) if *ever* may have clusters outside of EC2 + inside EC2, use GossipingPropertyFileSnitch 4) if using vnodes, just create a cluster out of hosts with 50% in each AZ and you should be all set. =Rob
making sure 1 copy per availability zone(rack) using EC2Snitch
Hello, I am planning my new cassandra 1.2.5 cluster with all nodes in single region but divided among 2 availablity zones equally. I want to make sure with replication factor 2 I get 1 copy in every availability zone. As per my knowledge using placement strategy EC2Snitch should take care of this. Is this correct? Do I need to specify any strategy options? Thanks. Rashmi
Re: making sure 1 copy per availability zone(rack) using EC2Snitch
Thanks for quick response Rob. Are you suggesting deploying 1.2.9 only if using Cassandra DC outside of EC2 or if I wish to use rack replication at all? On Mon, Sep 9, 2013 at 12:43 PM, Robert Coli rc...@eventbrite.com wrote: On Mon, Sep 9, 2013 at 8:56 AM, rash aroskar rashmi.aros...@gmail.comwrote: I am planning my new cassandra 1.2.5 cluster with all nodes in single region but divided among 2 availablity zones equally. I want to make sure with replication factor 2 I get 1 copy in every availability zone. As per my knowledge using placement strategy EC2Snitch should take care of this. Is this correct? Do I need to specify any strategy options? https://issues.apache.org/jira/browse/CASSANDRA-3810 Has background on how one must configure all rack aware snitches. If you may ever have a Cassandra DC outside of EC2, for example for Disaster Recovery, use GossipingPropertyFileSnitch. Also, deploy on 1.2.9, not 1.2.5. =Rob
Re: making sure 1 copy per availability zone(rack) using EC2Snitch
On Mon, Sep 9, 2013 at 8:56 AM, rash aroskar rashmi.aros...@gmail.comwrote: I am planning my new cassandra 1.2.5 cluster with all nodes in single region but divided among 2 availablity zones equally. I want to make sure with replication factor 2 I get 1 copy in every availability zone. As per my knowledge using placement strategy EC2Snitch should take care of this. Is this correct? Do I need to specify any strategy options? https://issues.apache.org/jira/browse/CASSANDRA-3810 Has background on how one must configure all rack aware snitches. If you may ever have a Cassandra DC outside of EC2, for example for Disaster Recovery, use GossipingPropertyFileSnitch. Also, deploy on 1.2.9, not 1.2.5. =Rob
Re: Ec2Snitch to Ec2MultiRegionSnitch
. Why is this the best method according to you ? About seeds = Yes. Have 3 from each. What is the point of this ? I didn't thought this change would be that tricky, thank you guys for these warnings and your help ;) Alain 2013/4/23 Dane Miller d...@optimalsocial.com On Thu, Apr 18, 2013 at 7:41 AM, Alain RODRIGUEZ arodr...@gmail.com wrote: I am wondering about the process to grow from one data center to a few of them. First thing is we use EC2Snitch for now. So I guess we have to switch to Ec2MultiRegionSnitch. c/ I am using the SimpleStrategy. Is it worth it/mandatory to change this strategy when using multiple DC ? I suggest you thoroughly read the datastax documentation on cassandra replication. The change you are planning is big - make sure to try it in a test environment first. Also, you might find you don't really need Cassandra's rack aware feature, and can operate using (Gossiping)PropertyFileSnitch. The rack feature is listed as an anti-pattern here: http://www.datastax.com/docs/1.2/cluster_architecture/anti_patterns Here are some recent discussions on this list: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/migrating-from-SimpleStrategy-to-NetworkTopologyStrategy-tp7586272.html http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/migrating-from-SimpleStrategy-to-NetworkTopologyStrategy-tp7481090.html Dane
Re: Ec2Snitch to Ec2MultiRegionSnitch
If you are only using one Available Zone per region then you have only one rack per DC and the NetworkTopologyStrategy will do the right thing. So you mean this part doesn't need more testing ? This will work for sure ? Did you already did it yourself ? Because you are going to replicate your data 3 times in each DC so that each DC can operate with a LOCAL_QUOURM Yet I don't get it. Tell me where I am wrong. LOCAL_QUORUM need to read/write 2 nodes (since RF = 3) per region. So if I use 6 eu-west and 3 us-east, C* will be able to reach the LOCAL_QUORUM everywhere, won't it ? So why should I use 6 + 6 servers ? nodetool rebuild is designed to handle pulling data from another dc, so you can use it when the local DC does not contain data. i.e. you do not want a node in the new DC bootstrapping from other nodes in the new DC, they have no data Good to know, thanks about it, as about all your pointers to the doc. Cause it's easier to understand than interleaving the nodes and works with 2+ DC's. Good point. If your are interested, I'll let you know how all the things go when we'll add the second DC. 2013/4/24 aaron morton aa...@thelastpickle.com You are advising me to test it, what would be a good way of testing it (I can use AWS EC2 instances if needed) ? If you are only using one Available Zone per region then you have only one rack per DC and the NetworkTopologyStrategy will do the right thing. Why ? I mean we have maybe only 5% of our customers on the us-east zone, what in C* require to have the same number of node on each DC ? Because you are going to replicate your data 3 times in each DC so that each DC can operate with a LOCAL_QUOURM. What is better on adding nodes with no data and then rebuild them compared to using the auto_bootstrap ? nodetool rebuild is designed to handle pulling data from another dc, so you can use it when the local DC does not contain data. i.e. you do not want a node in the new DC bootstrapping from other nodes in the new DC, they have no data. Any doc on this ? I am not aware of all the possibilities. Why is this the best method according to you ? http://wiki.apache.org/cassandra/Operations?highlight=%28token%29#Token_selection http://www.datastax.com/docs/1.2/initialize/token_generation Cause it's easier to understand than interleaving the nodes and works with 2+ DC's. What is the point of this ? http://wiki.apache.org/cassandra/FAQ#seed I didn't thought this change would be that tricky, thank you guys for these warnings and your help ;) Yup, this is a lot of work. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 23/04/2013, at 7:26 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi,these advice are very welcome. @Dane, about the rack awareness, we use only one rack per DC, so I guess using EC2MultiRegionSnitch will do just fine and it doesn't need any configuration. Does it seem right to you. If we are someday interested on multi racks I will make sure to use them properly. Thank you for this insight anyway. You are advising me to test it, what would be a good way of testing it (I can use AWS EC2 instances if needed) ? @Aaron I recommend using the same number of nodes in both DC's. Why ? I mean we have maybe only 5% of our customers on the us-east zone, what in C* require to have the same number of node on each DC ? Add the nodes (I recommend 6) with auto_bootstrap: false added to the yaml. update the keyspace replication strategy to add rf:3 for the new DC. Use nodetool rebuild on the new nodes to rebuild them from the us-west DC. What is better on adding nodes with no data and then rebuild them compared to using the auto_bootstrap ? I prefer to use the offset method. Take the 6 tokens from your us-west DC and add 100 to them for the new dc. Any doc on this ? I am not aware of all the possibilities. Why is this the best method according to you ? About seeds = Yes. Have 3 from each. What is the point of this ? I didn't thought this change would be that tricky, thank you guys for these warnings and your help ;) Alain 2013/4/23 Dane Miller d...@optimalsocial.com On Thu, Apr 18, 2013 at 7:41 AM, Alain RODRIGUEZ arodr...@gmail.com wrote: I am wondering about the process to grow from one data center to a few of them. First thing is we use EC2Snitch for now. So I guess we have to switch to Ec2MultiRegionSnitch. c/ I am using the SimpleStrategy. Is it worth it/mandatory to change this strategy when using multiple DC ? I suggest you thoroughly read the datastax documentation on cassandra replication. The change you are planning is big - make sure to try it in a test environment first. Also, you might find you don't really need Cassandra's rack aware feature, and can operate using (Gossiping)PropertyFileSnitch. The rack feature is listed
Re: Ec2Snitch to Ec2MultiRegionSnitch
Hi,these advice are very welcome. @Dane, about the rack awareness, we use only one rack per DC, so I guess using EC2MultiRegionSnitch will do just fine and it doesn't need any configuration. Does it seem right to you. If we are someday interested on multi racks I will make sure to use them properly. Thank you for this insight anyway. You are advising me to test it, what would be a good way of testing it (I can use AWS EC2 instances if needed) ? @Aaron I recommend using the same number of nodes in both DC's. Why ? I mean we have maybe only 5% of our customers on the us-east zone, what in C* require to have the same number of node on each DC ? Add the nodes (I recommend 6) with auto_bootstrap: false added to the yaml. update the keyspace replication strategy to add rf:3 for the new DC. Use nodetool rebuild on the new nodes to rebuild them from the us-west DC. What is better on adding nodes with no data and then rebuild them compared to using the auto_bootstrap ? I prefer to use the offset method. Take the 6 tokens from your us-west DC and add 100 to them for the new dc. Any doc on this ? I am not aware of all the possibilities. Why is this the best method according to you ? About seeds = Yes. Have 3 from each. What is the point of this ? I didn't thought this change would be that tricky, thank you guys for these warnings and your help ;) Alain 2013/4/23 Dane Miller d...@optimalsocial.com On Thu, Apr 18, 2013 at 7:41 AM, Alain RODRIGUEZ arodr...@gmail.com wrote: I am wondering about the process to grow from one data center to a few of them. First thing is we use EC2Snitch for now. So I guess we have to switch to Ec2MultiRegionSnitch. c/ I am using the SimpleStrategy. Is it worth it/mandatory to change this strategy when using multiple DC ? I suggest you thoroughly read the datastax documentation on cassandra replication. The change you are planning is big - make sure to try it in a test environment first. Also, you might find you don't really need Cassandra's rack aware feature, and can operate using (Gossiping)PropertyFileSnitch. The rack feature is listed as an anti-pattern here: http://www.datastax.com/docs/1.2/cluster_architecture/anti_patterns Here are some recent discussions on this list: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/migrating-from-SimpleStrategy-to-NetworkTopologyStrategy-tp7586272.html http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/migrating-from-SimpleStrategy-to-NetworkTopologyStrategy-tp7481090.html Dane
Re: Ec2Snitch to Ec2MultiRegionSnitch
You are advising me to test it, what would be a good way of testing it (I can use AWS EC2 instances if needed) ? If you are only using one Available Zone per region then you have only one rack per DC and the NetworkTopologyStrategy will do the right thing. Why ? I mean we have maybe only 5% of our customers on the us-east zone, what in C* require to have the same number of node on each DC ? Because you are going to replicate your data 3 times in each DC so that each DC can operate with a LOCAL_QUOURM. What is better on adding nodes with no data and then rebuild them compared to using the auto_bootstrap ? nodetool rebuild is designed to handle pulling data from another dc, so you can use it when the local DC does not contain data. i.e. you do not want a node in the new DC bootstrapping from other nodes in the new DC, they have no data. Any doc on this ? I am not aware of all the possibilities. Why is this the best method according to you ? http://wiki.apache.org/cassandra/Operations?highlight=%28token%29#Token_selection http://www.datastax.com/docs/1.2/initialize/token_generation Cause it's easier to understand than interleaving the nodes and works with 2+ DC's. What is the point of this ? http://wiki.apache.org/cassandra/FAQ#seed I didn't thought this change would be that tricky, thank you guys for these warnings and your help ;) Yup, this is a lot of work. Cheers - Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 23/04/2013, at 7:26 PM, Alain RODRIGUEZ arodr...@gmail.com wrote: Hi,these advice are very welcome. @Dane, about the rack awareness, we use only one rack per DC, so I guess using EC2MultiRegionSnitch will do just fine and it doesn't need any configuration. Does it seem right to you. If we are someday interested on multi racks I will make sure to use them properly. Thank you for this insight anyway. You are advising me to test it, what would be a good way of testing it (I can use AWS EC2 instances if needed) ? @Aaron I recommend using the same number of nodes in both DC's. Why ? I mean we have maybe only 5% of our customers on the us-east zone, what in C* require to have the same number of node on each DC ? Add the nodes (I recommend 6) with auto_bootstrap: false added to the yaml. update the keyspace replication strategy to add rf:3 for the new DC. Use nodetool rebuild on the new nodes to rebuild them from the us-west DC. What is better on adding nodes with no data and then rebuild them compared to using the auto_bootstrap ? I prefer to use the offset method. Take the 6 tokens from your us-west DC and add 100 to them for the new dc. Any doc on this ? I am not aware of all the possibilities. Why is this the best method according to you ? About seeds = Yes. Have 3 from each. What is the point of this ? I didn't thought this change would be that tricky, thank you guys for these warnings and your help ;) Alain 2013/4/23 Dane Miller d...@optimalsocial.com On Thu, Apr 18, 2013 at 7:41 AM, Alain RODRIGUEZ arodr...@gmail.com wrote: I am wondering about the process to grow from one data center to a few of them. First thing is we use EC2Snitch for now. So I guess we have to switch to Ec2MultiRegionSnitch. c/ I am using the SimpleStrategy. Is it worth it/mandatory to change this strategy when using multiple DC ? I suggest you thoroughly read the datastax documentation on cassandra replication. The change you are planning is big - make sure to try it in a test environment first. Also, you might find you don't really need Cassandra's rack aware feature, and can operate using (Gossiping)PropertyFileSnitch. The rack feature is listed as an anti-pattern here: http://www.datastax.com/docs/1.2/cluster_architecture/anti_patterns Here are some recent discussions on this list: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/migrating-from-SimpleStrategy-to-NetworkTopologyStrategy-tp7586272.html http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/migrating-from-SimpleStrategy-to-NetworkTopologyStrategy-tp7481090.html Dane
Re: Ec2Snitch to Ec2MultiRegionSnitch
On Thu, Apr 18, 2013 at 7:41 AM, Alain RODRIGUEZ arodr...@gmail.com wrote: I am wondering about the process to grow from one data center to a few of them. First thing is we use EC2Snitch for now. So I guess we have to switch to Ec2MultiRegionSnitch. c/ I am using the SimpleStrategy. Is it worth it/mandatory to change this strategy when using multiple DC ? I suggest you thoroughly read the datastax documentation on cassandra replication. The change you are planning is big - make sure to try it in a test environment first. Also, you might find you don't really need Cassandra's rack aware feature, and can operate using (Gossiping)PropertyFileSnitch. The rack feature is listed as an anti-pattern here: http://www.datastax.com/docs/1.2/cluster_architecture/anti_patterns Here are some recent discussions on this list: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/migrating-from-SimpleStrategy-to-NetworkTopologyStrategy-tp7586272.html http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/migrating-from-SimpleStrategy-to-NetworkTopologyStrategy-tp7481090.html Dane
Ec2Snitch to Ec2MultiRegionSnitch
Hi, The company I work for is having so much success that we are expanding worldwide :). We have to deploy our Cassandra servers worldwide too in order to improve the latency of our new abroad customers. I am wondering about the process to grow from one data center to a few of them. First thing is we use EC2Snitch for now. So I guess we have to switch to Ec2MultiRegionSnitch. Is that doable without any down-time ? Our C* cluster : C*1.2.2, 6 EC2 m1.xLarge in eu-west already running, wanting to add 3 m1.xLarge on us-east I was planning to do it this way: 1/ Change the yaml conf on each of the 6 eu-west existing nodes - Ec2Snitch to Ec2MultiRegionSnitch - uncomment the broadcast_address and set the public ip of the node - let the private ip as defined right now the listen_address - switch seeds from private to public IP 2/ Rolling restart - nodetool disablegossip - nodetool disablethrift - nodetool drain - rm /path/cassandra/commitlog/* ? (I used to do it since drain was broken to avoid replaying counters logs, leading to overcounts, not sure how pertinent this is nowadays) - service cassandra stop - service cassandra start 3/ - Make sure everything is still running smoothly in eu-west servers 4/ - Add 3 nodes one by one with auto_bootstrap set to true. 5/ - Repair nodes (one by one) - Cleanup nodes (one by one) Questions : a/ Do I have to move the tokens since I don't use vnodes yet ? How should I position all these nodes ? b/ Is it useful to add a seed from the new us-east data center in the yaml of all nodes ? c/ I am using the SimpleStrategy. Is it worth it/mandatory to change this strategy when using multiple DC ? d/ With my 2 DC will I have 3 RF per DC or cross DC ? e/ Should I configure my C* client to use the C* nodes from their region as coordinators (which seems to me the good way) or should I configure all the servers everywhere ? Any comment on the process described above would be appreciated, specially if you are arguing that something is wrong about it. If you find out I am missing something, I will be glad to hear about it. Alain
Ec2Snitch and Ec2MultiRegionSnitch - does us-west-2 cause a bug with this snitch?
Hi,
As of Nov. 9, 2011 Amazon added us-west-2 (US West Oregon) region:
http://aws.typepad.com/aws/2011/11/now-open-us-west-portland-region.html
In looking at the EC2Snitch code (in the 0.8.x and 1.0.x branches), I see
it determining which data center (which I think is supposed to be
equivalent to region) it is in by this:
public Ec2Snitch() throws IOException, ConfigurationException
{
// Split us-east-1a or asia-1a into us-east/1a and asia/1a.
String[] splits = awsApiCall(ZONE_NAME_QUERY_URL).split(-);
ec2zone = splits[splits.length - 1];
ec2region = splits.length 3 ? splits[0] : splits[0] + - + splits[1];
logger.info(EC2Snitch using region: + ec2region + , zone:
+ ec2zone + .);
}
The Ec2MultiRegionSnitch then has this reconnect logic - which I think the
existence of a us-west-1 and us-west-2 regions and the way the Ec2Snitch is
assigning data centers will cause this to break:
private void reConnect(InetAddress endpoint, VersionedValue versionedValue)
{
if (!getDatacenter(endpoint).equals(getDatacenter(public_ip)))
return; // do nothing return back...
try
{
InetAddress remoteIP = InetAddress.getByName(versionedValue.value);
MessagingService.instance().getConnectionPool(endpoint).reset(remoteIP);
logger.debug(String.format(Intiated reconnect to an
Internal IP %s for the %s, remoteIP, endpoint));
} catch (UnknownHostException e)
{
logger.error(Error in getting the IP address resolved: , e);
}
}
Am I correct or do I not understand what was intended by these snitches?
Allen
Ec2Snitch nodetool issue after upgrade to 0.8.5
Upgraded one ring that has four nodes from 0.8.0 to 0.8.5 with only one minor problem. It relates to Ec2Snitch when running a 'nodetool ring' from two of the four nodes. the rest are all working fine: Address DC Rack Status State Load Owns Token 148362247927262972740864614603570725035 172.16.12.11 ap-southeast1a Up Normal 1.58 MB 24.02% 1909554714494251628118265338228798 172.16.12.10 ap-southeast1a Up Normal 1.63 MB 22.11% 56713727820156410577229101238628035242 172.16.14.10 ap-southeast1b Up Normal 1.85 MB 33.33% 113427455640312821154458202477256070484 172.16.14.12 ap-southeast1b Up Normal 1.36 MB 20.53% 14836224792726297274086461460357072503 works ... on 2 nodes which happen to be on the 172.16.14.0/24 network. the nodes where the error appears are on the 172.16.12.0/24 network and this is what is shown when nodetool ring is run: Address DC Rack Status State Load Owns Token 148362247927262972740864614603570725035 172.16.12.11 ap-southeast1a Up Normal 1.58 MB 24.02% 1909554714494251628118265338228798 172.16.12.10 ap-southeast1a Up Normal 1.62 MB 22.11% 56713727820156410577229101238628035242 Exception in thread main java.lang.NullPointerException at org.apache.cassandra.locator.Ec2Snitch.getDatacenter(Ec2Snitch.java:93) at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122) at org.apache.cassandra.locator.EndpointSnitchInfo.getDatacenter(EndpointSnitchInfo.java:49) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) I've stopped the node and started it ... still doesn't make a difference. I've also shut down all nodes in the ring so that it was fully offline, and then brought them all back up ... issue still persists on two of the nodes. There are no firewall rules restricting traffic between these nodes. For example, on a node where the ring throws the exception, the two hosts that don't show up i can still get nestats for: nodetool -h 172.16.12.11 -p 9090 netstats 172.16.14.10 Mode: Normal Nothing streaming to /172.16.14.10 Nothing streaming from /172.16.14.10 Pool NameActive Pending Completed Commandsn/a 0 3 Responses n/a 1 1483 nodetool -h 172.16.12.11 -p 9090 netstats
Re: Ec2Snitch nodetool issue after upgrade to 0.8.5
maybe it's related to this ... https://issues.apache.org/jira/browse/CASSANDRA-3114 odd thing is, we haven't moved to Ec2Snitch ... been using it for quite a long time now ... On Sat, Sep 10, 2011 at 1:42 PM, Sasha Dolgy sdo...@gmail.com wrote: Upgraded one ring that has four nodes from 0.8.0 to 0.8.5 with only one minor problem. It relates to Ec2Snitch when running a 'nodetool ring' from two of the four nodes. the rest are all working fine: Address DC Rack Status State Load Owns Token 148362247927262972740864614603570725035 172.16.12.11 ap-southeast1a Up Normal 1.58 MB 24.02% 1909554714494251628118265338228798 172.16.12.10 ap-southeast1a Up Normal 1.63 MB 22.11% 56713727820156410577229101238628035242 172.16.14.10 ap-southeast1b Up Normal 1.85 MB 33.33% 113427455640312821154458202477256070484 172.16.14.12 ap-southeast1b Up Normal 1.36 MB 20.53% 14836224792726297274086461460357072503 works ... on 2 nodes which happen to be on the 172.16.14.0/24 network. the nodes where the error appears are on the 172.16.12.0/24 network and this is what is shown when nodetool ring is run: Address DC Rack Status State Load Owns Token 148362247927262972740864614603570725035 172.16.12.11 ap-southeast1a Up Normal 1.58 MB 24.02% 1909554714494251628118265338228798 172.16.12.10 ap-southeast1a Up Normal 1.62 MB 22.11% 56713727820156410577229101238628035242 Exception in thread main java.lang.NullPointerException at org.apache.cassandra.locator.Ec2Snitch.getDatacenter(Ec2Snitch.java:93) at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122) at org.apache.cassandra.locator.EndpointSnitchInfo.getDatacenter(EndpointSnitchInfo.java:49) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) I've stopped the node and started it ... still doesn't make a difference. I've also shut down all nodes in the ring so that it was fully offline, and then brought them all back up ... issue still persists on two of the nodes. There are no firewall rules restricting traffic between these nodes. For example, on a node where the ring throws the exception, the two hosts that don't show up i can still get nestats for: nodetool -h 172.16.12.11 -p 9090
Re: Ec2Snitch nodetool issue after upgrade to 0.8.5
Can you create a Jira ticket? On Sat, Sep 10, 2011 at 6:52 AM, Sasha Dolgy sdo...@gmail.com wrote: maybe it's related to this ... https://issues.apache.org/jira/browse/CASSANDRA-3114 odd thing is, we haven't moved to Ec2Snitch ... been using it for quite a long time now ... On Sat, Sep 10, 2011 at 1:42 PM, Sasha Dolgy sdo...@gmail.com wrote: Upgraded one ring that has four nodes from 0.8.0 to 0.8.5 with only one minor problem. It relates to Ec2Snitch when running a 'nodetool ring' from two of the four nodes. the rest are all working fine: Address DC Rack Status State Load Owns Token 148362247927262972740864614603570725035 172.16.12.11 ap-southeast1a Up Normal 1.58 MB 24.02% 1909554714494251628118265338228798 172.16.12.10 ap-southeast1a Up Normal 1.63 MB 22.11% 56713727820156410577229101238628035242 172.16.14.10 ap-southeast1b Up Normal 1.85 MB 33.33% 113427455640312821154458202477256070484 172.16.14.12 ap-southeast1b Up Normal 1.36 MB 20.53% 14836224792726297274086461460357072503 works ... on 2 nodes which happen to be on the 172.16.14.0/24 network. the nodes where the error appears are on the 172.16.12.0/24 network and this is what is shown when nodetool ring is run: Address DC Rack Status State Load Owns Token 148362247927262972740864614603570725035 172.16.12.11 ap-southeast1a Up Normal 1.58 MB 24.02% 1909554714494251628118265338228798 172.16.12.10 ap-southeast1a Up Normal 1.62 MB 22.11% 56713727820156410577229101238628035242 Exception in thread main java.lang.NullPointerException at org.apache.cassandra.locator.Ec2Snitch.getDatacenter(Ec2Snitch.java:93) at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122) at org.apache.cassandra.locator.EndpointSnitchInfo.getDatacenter(EndpointSnitchInfo.java:49) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:93) at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:27) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:208) at com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:120) at com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:262) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:836) at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:761) at javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1427) at javax.management.remote.rmi.RMIConnectionImpl.access$200(RMIConnectionImpl.java:72) at javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1265) at javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1360) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:788) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) I've stopped the node and started it ... still doesn't make a difference. I've also shut down all nodes in the ring so that it was fully offline, and then brought them all back up ... issue still persists on two of the nodes. There are no firewall rules restricting traffic between these nodes. For example, on a node where the ring throws
Re: Ec2Snitch nodetool issue after upgrade to 0.8.5
Of course. Hoping one day I create an issue related to Ec2 that CAN be reproduced... https://issues.apache.org/jira/browse/CASSANDRA-3175 On Sat, Sep 10, 2011 at 10:10 PM, Jonathan Ellis jbel...@gmail.com wrote: Can you create a Jira ticket?
Re: Ec2Snitch
Yup, work perfectly now. Thanks, V. On 10. Aug (Wednesday) v 20:45:36 -0500 2011, Brandon Williams wrote: You probably have other nodes that are NOT using the snitch yet, so they haven't populated DC/RACK info yet. The exceptions will stop when all snitches have been changed. On Wed, Aug 10, 2011 at 7:55 PM, Viliam Holub viliam.ho...@ucd.ie wrote: Hi, I tried to switch to Ec2Snith. Although it correctly found the region: INFO 23:18:00,643 EC2Snitch using region: eu-west, zone: 1a. it started to report NullPointerException every second: ERROR 00:23:40,268 Internal error processing get_slice java.lang.NullPointerException at org.apache.cassandra.locator.Ec2Snitch.getDatacenter(Ec2Snitch.java:93) at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122) at org.apache.cassandra.locator.OldNetworkTopologyStrategy.calculateNaturalEndpoints(OldNetworkTopologyStrategy.java:64) at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:99) at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1708) at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1702) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:511) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:480) at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:126) at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:280) at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:362) at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:323) at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Am I doing something wrong? Thanks, Viliam -- Viliam Holub School of Computer Science and Informatics University College Dublin, Ireland
Ec2Snitch
Hi, I tried to switch to Ec2Snith. Although it correctly found the region: INFO 23:18:00,643 EC2Snitch using region: eu-west, zone: 1a. it started to report NullPointerException every second: ERROR 00:23:40,268 Internal error processing get_slice java.lang.NullPointerException at org.apache.cassandra.locator.Ec2Snitch.getDatacenter(Ec2Snitch.java:93) at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122) at org.apache.cassandra.locator.OldNetworkTopologyStrategy.calculateNaturalEndpoints(OldNetworkTopologyStrategy.java:64) at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:99) at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1708) at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1702) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:511) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:480) at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:126) at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:280) at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:362) at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:323) at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Am I doing something wrong? Thanks, Viliam
Re: Ec2Snitch
You probably have other nodes that are NOT using the snitch yet, so they haven't populated DC/RACK info yet. The exceptions will stop when all snitches have been changed. On Wed, Aug 10, 2011 at 7:55 PM, Viliam Holub viliam.ho...@ucd.ie wrote: Hi, I tried to switch to Ec2Snith. Although it correctly found the region: INFO 23:18:00,643 EC2Snitch using region: eu-west, zone: 1a. it started to report NullPointerException every second: ERROR 00:23:40,268 Internal error processing get_slice java.lang.NullPointerException at org.apache.cassandra.locator.Ec2Snitch.getDatacenter(Ec2Snitch.java:93) at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122) at org.apache.cassandra.locator.OldNetworkTopologyStrategy.calculateNaturalEndpoints(OldNetworkTopologyStrategy.java:64) at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:99) at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1708) at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1702) at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:511) at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:480) at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:126) at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:280) at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:362) at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:323) at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033) at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889) at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) Am I doing something wrong? Thanks, Viliam
Re: Ec2Snitch + NetworkTopologyStrategy if only in one region?
Also for the new users like me, don't assume DC1 is a keyword like I did. A
working example of a keyspace in EC2 is:
create keyspace test with replication_factor=3 and strategy_options =
[{us-east:3}] and
placement_strategy='org.apache.cassandra.locator.NetworkTopologyStrategy';
For a single DC in EC2 deployment. I felt silly afterwards, but I couldn't
find official docs on the structure of strategy_options anywhere.
will
On Wed, Apr 13, 2011 at 5:14 PM, William Oberman
ober...@civicscience.comwrote:
One last coda, for other noobs to cassandra like me. If you use
NetworkTopologyStrategy with replication_factor 1, make sure you have EC2
instance in multiple availability zones. I was doing baby steps, and tried
doing a cluster in one AZ (before spreading to multiple AZs) and was getting
the most baffling errors (cassandra_UnavailableException). I finally
thought to check the cassandra server logs (after debugging the client code,
firewalls, etc... painstakingly for connectivity problems), and it ends up
my cassandra cluster was considering itself unavailable as it couldn't
replicate as much as it wanted to. I kind of wish a different word than
unavailable was chosen for this error condition :-)
will
On Tue, Apr 12, 2011 at 10:37 PM, aaron morton aa...@thelastpickle.comwrote:
If you can use standard + encoded I would go with that.
Aaron
On 13 Apr 2011, at 07:07, William Oberman wrote:
Excellent to know! (and yes, I figure I'll expand someday, so I'm glad I
found this out before digging a hole).
The other issue I've been pondering is a normal column family of encoded
objects (in my case JSON) vs. a super column. Based on my use case, things
I've read, etc... right now I'm coming down on normal + encoded.
will
On Tue, Apr 12, 2011 at 2:57 PM, Jonathan Ellis jbel...@gmail.comwrote:
NTS is overkill in the sense that it doesn't really benefit you in a
single DC, but if you think you may expand to another DC in the future
it's much simpler if you were already using NTS, than first migrating
to NTS (changing strategy is painful).
I can't think of any downsides to using NTS in a single-DC
environment, so that's the safe option.
On Tue, Apr 12, 2011 at 1:15 PM, William Oberman
ober...@civicscience.com wrote:
Hi,
I'm getting closer to commiting to cassandra, and now I'm in system/IT
issues and questions. I'm in the amazon EC2 cloud. I previously used
this
forum to discover the best practice for disk layouts (large instance +
the
two ephemeral disks in RAID0 for data + root volume for everything
else).
Now I'm hoping to confirm bits and pieces of things I've read about for
snitch/replication strategies. I was thinking of using
endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch
placement_strategy='org.apache.cassandra.locator.NetworkTopologyStrategy'
(for people hitting this from the mailing list or google, I feel
obligated
to note that the former setting is in cassandra.yaml, and the latter is
an
option on a keyspace).
But, I'm only in one region. Is using the amazon snitch/networktopology
overkill given everything I have is in one DC (I believe region==DC and
availability_zone==rack). I'm using multiple availability zones for
some
level of redundancy, I'm just not yet to the point I'm using multiple
regions. If someday I move to using multiple regions, would that
change the
answer?
Thanks!
--
Will Oberman
Civic Science, Inc.
3030 Penn Avenue., First Floor
Pittsburgh, PA 15201
(M) 412-480-7835
(E) ober...@civicscience.com
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
--
Will Oberman
Civic Science, Inc.
3030 Penn Avenue., First Floor
Pittsburgh, PA 15201
(M) 412-480-7835
(E) ober...@civicscience.com
--
Will Oberman
Civic Science, Inc.
3030 Penn Avenue., First Floor
Pittsburgh, PA 15201
(M) 412-480-7835
(E) ober...@civicscience.com
--
Will Oberman
Civic Science, Inc.
3030 Penn Avenue., First Floor
Pittsburgh, PA 15201
(M) 412-480-7835
(E) ober...@civicscience.com
Re: Ec2Snitch + NetworkTopologyStrategy if only in one region?
One last coda, for other noobs to cassandra like me. If you use NetworkTopologyStrategy with replication_factor 1, make sure you have EC2 instance in multiple availability zones. I was doing baby steps, and tried doing a cluster in one AZ (before spreading to multiple AZs) and was getting the most baffling errors (cassandra_UnavailableException). I finally thought to check the cassandra server logs (after debugging the client code, firewalls, etc... painstakingly for connectivity problems), and it ends up my cassandra cluster was considering itself unavailable as it couldn't replicate as much as it wanted to. I kind of wish a different word than unavailable was chosen for this error condition :-) will On Tue, Apr 12, 2011 at 10:37 PM, aaron morton aa...@thelastpickle.comwrote: If you can use standard + encoded I would go with that. Aaron On 13 Apr 2011, at 07:07, William Oberman wrote: Excellent to know! (and yes, I figure I'll expand someday, so I'm glad I found this out before digging a hole). The other issue I've been pondering is a normal column family of encoded objects (in my case JSON) vs. a super column. Based on my use case, things I've read, etc... right now I'm coming down on normal + encoded. will On Tue, Apr 12, 2011 at 2:57 PM, Jonathan Ellis jbel...@gmail.com wrote: NTS is overkill in the sense that it doesn't really benefit you in a single DC, but if you think you may expand to another DC in the future it's much simpler if you were already using NTS, than first migrating to NTS (changing strategy is painful). I can't think of any downsides to using NTS in a single-DC environment, so that's the safe option. On Tue, Apr 12, 2011 at 1:15 PM, William Oberman ober...@civicscience.com wrote: Hi, I'm getting closer to commiting to cassandra, and now I'm in system/IT issues and questions. I'm in the amazon EC2 cloud. I previously used this forum to discover the best practice for disk layouts (large instance + the two ephemeral disks in RAID0 for data + root volume for everything else). Now I'm hoping to confirm bits and pieces of things I've read about for snitch/replication strategies. I was thinking of using endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch placement_strategy='org.apache.cassandra.locator.NetworkTopologyStrategy' (for people hitting this from the mailing list or google, I feel obligated to note that the former setting is in cassandra.yaml, and the latter is an option on a keyspace). But, I'm only in one region. Is using the amazon snitch/networktopology overkill given everything I have is in one DC (I believe region==DC and availability_zone==rack). I'm using multiple availability zones for some level of redundancy, I'm just not yet to the point I'm using multiple regions. If someday I move to using multiple regions, would that change the answer? Thanks! -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) ober...@civicscience.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) ober...@civicscience.com -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) ober...@civicscience.com
Ec2Snitch + NetworkTopologyStrategy if only in one region?
Hi, I'm getting closer to commiting to cassandra, and now I'm in system/IT issues and questions. I'm in the amazon EC2 cloud. I previously used this forum to discover the best practice for disk layouts (large instance + the two ephemeral disks in RAID0 for data + root volume for everything else). Now I'm hoping to confirm bits and pieces of things I've read about for snitch/replication strategies. I was thinking of using endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch placement_strategy='org.apache.cassandra.locator.NetworkTopologyStrategy' (for people hitting this from the mailing list or google, I feel obligated to note that the former setting is in cassandra.yaml, and the latter is an option on a keyspace). But, I'm only in one region. Is using the amazon snitch/networktopology overkill given everything I have is in one DC (I believe region==DC and availability_zone==rack). I'm using multiple availability zones for some level of redundancy, I'm just not yet to the point I'm using multiple regions. If someday I move to using multiple regions, would that change the answer? Thanks! -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) ober...@civicscience.com
Re: Ec2Snitch + NetworkTopologyStrategy if only in one region?
NTS is overkill in the sense that it doesn't really benefit you in a single DC, but if you think you may expand to another DC in the future it's much simpler if you were already using NTS, than first migrating to NTS (changing strategy is painful). I can't think of any downsides to using NTS in a single-DC environment, so that's the safe option. On Tue, Apr 12, 2011 at 1:15 PM, William Oberman ober...@civicscience.com wrote: Hi, I'm getting closer to commiting to cassandra, and now I'm in system/IT issues and questions. I'm in the amazon EC2 cloud. I previously used this forum to discover the best practice for disk layouts (large instance + the two ephemeral disks in RAID0 for data + root volume for everything else). Now I'm hoping to confirm bits and pieces of things I've read about for snitch/replication strategies. I was thinking of using endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch placement_strategy='org.apache.cassandra.locator.NetworkTopologyStrategy' (for people hitting this from the mailing list or google, I feel obligated to note that the former setting is in cassandra.yaml, and the latter is an option on a keyspace). But, I'm only in one region. Is using the amazon snitch/networktopology overkill given everything I have is in one DC (I believe region==DC and availability_zone==rack). I'm using multiple availability zones for some level of redundancy, I'm just not yet to the point I'm using multiple regions. If someday I move to using multiple regions, would that change the answer? Thanks! -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) ober...@civicscience.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com
Re: Ec2Snitch + NetworkTopologyStrategy if only in one region?
Excellent to know! (and yes, I figure I'll expand someday, so I'm glad I found this out before digging a hole). The other issue I've been pondering is a normal column family of encoded objects (in my case JSON) vs. a super column. Based on my use case, things I've read, etc... right now I'm coming down on normal + encoded. will On Tue, Apr 12, 2011 at 2:57 PM, Jonathan Ellis jbel...@gmail.com wrote: NTS is overkill in the sense that it doesn't really benefit you in a single DC, but if you think you may expand to another DC in the future it's much simpler if you were already using NTS, than first migrating to NTS (changing strategy is painful). I can't think of any downsides to using NTS in a single-DC environment, so that's the safe option. On Tue, Apr 12, 2011 at 1:15 PM, William Oberman ober...@civicscience.com wrote: Hi, I'm getting closer to commiting to cassandra, and now I'm in system/IT issues and questions. I'm in the amazon EC2 cloud. I previously used this forum to discover the best practice for disk layouts (large instance + the two ephemeral disks in RAID0 for data + root volume for everything else). Now I'm hoping to confirm bits and pieces of things I've read about for snitch/replication strategies. I was thinking of using endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch placement_strategy='org.apache.cassandra.locator.NetworkTopologyStrategy' (for people hitting this from the mailing list or google, I feel obligated to note that the former setting is in cassandra.yaml, and the latter is an option on a keyspace). But, I'm only in one region. Is using the amazon snitch/networktopology overkill given everything I have is in one DC (I believe region==DC and availability_zone==rack). I'm using multiple availability zones for some level of redundancy, I'm just not yet to the point I'm using multiple regions. If someday I move to using multiple regions, would that change the answer? Thanks! -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) ober...@civicscience.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) ober...@civicscience.com
Re: Ec2Snitch + NetworkTopologyStrategy if only in one region?
If you can use standard + encoded I would go with that. Aaron On 13 Apr 2011, at 07:07, William Oberman wrote: Excellent to know! (and yes, I figure I'll expand someday, so I'm glad I found this out before digging a hole). The other issue I've been pondering is a normal column family of encoded objects (in my case JSON) vs. a super column. Based on my use case, things I've read, etc... right now I'm coming down on normal + encoded. will On Tue, Apr 12, 2011 at 2:57 PM, Jonathan Ellis jbel...@gmail.com wrote: NTS is overkill in the sense that it doesn't really benefit you in a single DC, but if you think you may expand to another DC in the future it's much simpler if you were already using NTS, than first migrating to NTS (changing strategy is painful). I can't think of any downsides to using NTS in a single-DC environment, so that's the safe option. On Tue, Apr 12, 2011 at 1:15 PM, William Oberman ober...@civicscience.com wrote: Hi, I'm getting closer to commiting to cassandra, and now I'm in system/IT issues and questions. I'm in the amazon EC2 cloud. I previously used this forum to discover the best practice for disk layouts (large instance + the two ephemeral disks in RAID0 for data + root volume for everything else). Now I'm hoping to confirm bits and pieces of things I've read about for snitch/replication strategies. I was thinking of using endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch placement_strategy='org.apache.cassandra.locator.NetworkTopologyStrategy' (for people hitting this from the mailing list or google, I feel obligated to note that the former setting is in cassandra.yaml, and the latter is an option on a keyspace). But, I'm only in one region. Is using the amazon snitch/networktopology overkill given everything I have is in one DC (I believe region==DC and availability_zone==rack). I'm using multiple availability zones for some level of redundancy, I'm just not yet to the point I'm using multiple regions. If someday I move to using multiple regions, would that change the answer? Thanks! -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) ober...@civicscience.com -- Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com -- Will Oberman Civic Science, Inc. 3030 Penn Avenue., First Floor Pittsburgh, PA 15201 (M) 412-480-7835 (E) ober...@civicscience.com
Ec2Snitch Other snitches...
Hi Everyone, Can the Ec2Snitch be enabled by adjusting the parameter in the cassandra.yaml and restarting the node? More, I suppose the question I'm after is, can the snitch method be adjusted adhoc (with node restart) or once it's changed from SimpleSnitch to Ec2Snitch that's it? What influence does one node have over the Gossiper? If i'm not happy with the Ec2Snitch, I want to change to PropertyFileSnitch ... but if I have 8 nodes at the moment. ... i don't want to create additional headaches... -- Sasha Dolgy sasha.do...@gmail.com
Re: Ec2Snitch Other snitches...
On Tue, Mar 22, 2011 at 7:19 AM, Sasha Dolgy sdo...@gmail.com wrote: More, I suppose the question I'm after is, can the snitch method be adjusted adhoc (with node restart) or once it's changed from SimpleSnitch to Ec2Snitch that's it? You can change Snitches on a cluster with data on it, as long as you are very careful about what you are doing and you are in a particular case which you are probably not in if you want to change your Snitch. The snitch meaningfully determines replica placement strategy, and in general when changing snitches you need the replica placement strategy to stay exactly the same. Unfortunately the point of changing a snitch is usually.. changing your replica placement strategy. Simplest case is if the replica placement strategy actually stays the same, like for example when Digg replaced its custom version of the PropertyFileSnitch with SimpleSnitch in prep for going single-DC, because we weren't actually using the functionality of PFS. In that case, I simply generated a set of input which hashed correctly such that I had one piece of input per node. I then verified the topology based on this input before and after changing my snitch, and got the same results both times, confirming that my change of the Snitch was a no-op. A less simple, but still tractable case is if the topology changes such that one or more replicas is different but at least one is still the same. In this case, repair would be likely to repair.. most.. of your data. But honestly if you have to change strategy that much (and are not running IP-partitioned counts, which make this operation much more difficult) you probably just want to dump and reload your data into a new cluster which has the topology and snitch you want. =Rob
