Re: Riak and host names
Hi folks, After trying a couple of configurations, I'd recommend using a VPC on EC2. I decided to share my experience with a wider audience and wrote about it on a blog post http://deepakbala.me/2013/02/08/deploying-riak-on-ec2/. The post details (among other things) why VPC can make your life much easier. Thank you again for all your suggestions. Is there a Riak wiki that I can contribute this information to ? It would be easier for users to choose between deployment solutions if they know what options are available. Thanks -Deepak On Sat, Jan 12, 2013 at 1:02 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: Thanks for all the suggestions. I will take some time to review and try them out. I'll get back to this thread when I do. -Deepak On Thu, Jan 10, 2013 at 3:04 PM, Richard Shaw rich...@basho.com wrote: Hi Matt, For stopping and starting rather than rebooting, you need to use elastic IPs inside of Amazon's virtual private cloud[1] which allows for persistent static private addresses. [1]http://aws.amazon.com/vpc/ Regards Richard On Thu, Jan 10, 2013 at 6:09 AM, Matt Black matt.bl...@jbadigital.com wrote: A quick update on this subject. Using an Elastic IP won't help with AWS since that only binds to the public interface - not the internal private one. The hostname command still returns the same internal IP address as before, which is what's seen by Riak. In AWS an internal IP address will actually persist across reboots. It does not persist across shutdown and startup. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Hi Deepak, On Fri, Feb 8, 2013 at 3:15 AM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: Hi folks, After trying a couple of configurations, I'd recommend using a VPC on EC2. I decided to share my experience with a wider audience and wrote about it on a blog post. The post details (among other things) why VPC can make your life much easier. Thank you again for all your suggestions. Nice. Thanks for taking the time to write the post. Is there a Riak wiki that I can contribute this information to ? It would be easier for users to choose between deployment solutions if they know what options are available. Absolutely. All our docs are on GitHub [0] and we love (cherish) contributions. You'll probably want to add it to the Performance Tuning for AWS section [1], and at the very least we should have a link to the blog post. Thanks again. Mark twitter.com/pharkmillups A Riak Conference in NYC! - ricon.io/east.html [0] https://github.com/basho/basho_docs [1] http://docs.basho.com/riak/latest/cookbooks/Performance-Tuning-AWS/ Thanks -Deepak On Sat, Jan 12, 2013 at 1:02 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: Thanks for all the suggestions. I will take some time to review and try them out. I'll get back to this thread when I do. -Deepak On Thu, Jan 10, 2013 at 3:04 PM, Richard Shaw rich...@basho.com wrote: Hi Matt, For stopping and starting rather than rebooting, you need to use elastic IPs inside of Amazon's virtual private cloud[1] which allows for persistent static private addresses. [1]http://aws.amazon.com/vpc/ Regards Richard On Thu, Jan 10, 2013 at 6:09 AM, Matt Black matt.bl...@jbadigital.com wrote: A quick update on this subject. Using an Elastic IP won't help with AWS since that only binds to the public interface - not the internal private one. The hostname command still returns the same internal IP address as before, which is what's seen by Riak. In AWS an internal IP address will actually persist across reboots. It does not persist across shutdown and startup. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Awesome. I'll make a pull request soon. Thanks -Deepak On Sat, Feb 9, 2013 at 12:08 AM, Mark Phillips m...@basho.com wrote: Hi Deepak, On Fri, Feb 8, 2013 at 3:15 AM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: Hi folks, After trying a couple of configurations, I'd recommend using a VPC on EC2. I decided to share my experience with a wider audience and wrote about it on a blog post. The post details (among other things) why VPC can make your life much easier. Thank you again for all your suggestions. Nice. Thanks for taking the time to write the post. Is there a Riak wiki that I can contribute this information to ? It would be easier for users to choose between deployment solutions if they know what options are available. Absolutely. All our docs are on GitHub [0] and we love (cherish) contributions. You'll probably want to add it to the Performance Tuning for AWS section [1], and at the very least we should have a link to the blog post. Thanks again. Mark twitter.com/pharkmillups A Riak Conference in NYC! - ricon.io/east.html [0] https://github.com/basho/basho_docs [1] http://docs.basho.com/riak/latest/cookbooks/Performance-Tuning-AWS/ Thanks -Deepak On Sat, Jan 12, 2013 at 1:02 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: Thanks for all the suggestions. I will take some time to review and try them out. I'll get back to this thread when I do. -Deepak On Thu, Jan 10, 2013 at 3:04 PM, Richard Shaw rich...@basho.com wrote: Hi Matt, For stopping and starting rather than rebooting, you need to use elastic IPs inside of Amazon's virtual private cloud[1] which allows for persistent static private addresses. [1]http://aws.amazon.com/vpc/ Regards Richard On Thu, Jan 10, 2013 at 6:09 AM, Matt Black matt.bl...@jbadigital.com wrote: A quick update on this subject. Using an Elastic IP won't help with AWS since that only binds to the public interface - not the internal private one. The hostname command still returns the same internal IP address as before, which is what's seen by Riak. In AWS an internal IP address will actually persist across reboots. It does not persist across shutdown and startup. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Hurray! We love doc contributions! On Feb 8, 2013, at 10:38 AM, Mark Phillips m...@basho.com wrote: Hi Deepak, On Fri, Feb 8, 2013 at 3:15 AM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: Hi folks, After trying a couple of configurations, I'd recommend using a VPC on EC2. I decided to share my experience with a wider audience and wrote about it on a blog post. The post details (among other things) why VPC can make your life much easier. Thank you again for all your suggestions. Nice. Thanks for taking the time to write the post. Is there a Riak wiki that I can contribute this information to ? It would be easier for users to choose between deployment solutions if they know what options are available. Absolutely. All our docs are on GitHub [0] and we love (cherish) contributions. You'll probably want to add it to the Performance Tuning for AWS section [1], and at the very least we should have a link to the blog post. Thanks again. Mark twitter.com/pharkmillups A Riak Conference in NYC! - ricon.io/east.html [0] https://github.com/basho/basho_docs [1] http://docs.basho.com/riak/latest/cookbooks/Performance-Tuning-AWS/ Thanks -Deepak On Sat, Jan 12, 2013 at 1:02 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: Thanks for all the suggestions. I will take some time to review and try them out. I'll get back to this thread when I do. -Deepak On Thu, Jan 10, 2013 at 3:04 PM, Richard Shaw rich...@basho.com wrote: Hi Matt, For stopping and starting rather than rebooting, you need to use elastic IPs inside of Amazon's virtual private cloud[1] which allows for persistent static private addresses. [1]http://aws.amazon.com/vpc/ Regards Richard On Thu, Jan 10, 2013 at 6:09 AM, Matt Black matt.bl...@jbadigital.com wrote: A quick update on this subject. Using an Elastic IP won't help with AWS since that only binds to the public interface - not the internal private one. The hostname command still returns the same internal IP address as before, which is what's seen by Riak. In AWS an internal IP address will actually persist across reboots. It does not persist across shutdown and startup. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Thanks for all the suggestions. I will take some time to review and try them out. I'll get back to this thread when I do. -Deepak On Thu, Jan 10, 2013 at 3:04 PM, Richard Shaw rich...@basho.com wrote: Hi Matt, For stopping and starting rather than rebooting, you need to use elastic IPs inside of Amazon's virtual private cloud[1] which allows for persistent static private addresses. [1]http://aws.amazon.com/vpc/ Regards Richard On Thu, Jan 10, 2013 at 6:09 AM, Matt Black matt.bl...@jbadigital.com wrote: A quick update on this subject. Using an Elastic IP won't help with AWS since that only binds to the public interface - not the internal private one. The hostname command still returns the same internal IP address as before, which is what's seen by Riak. In AWS an internal IP address will actually persist across reboots. It does not persist across shutdown and startup. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Hi Matt, For stopping and starting rather than rebooting, you need to use elastic IPs inside of Amazon's virtual private cloud[1] which allows for persistent static private addresses. [1]http://aws.amazon.com/vpc/ Regards Richard On Thu, Jan 10, 2013 at 6:09 AM, Matt Black matt.bl...@jbadigital.com wrote: A quick update on this subject. Using an Elastic IP won't help with AWS since that only binds to the public interface - not the internal private one. The hostname command still returns the same internal IP address as before, which is what's seen by Riak. In AWS an internal IP address will actually persist across reboots. It does not persist across shutdown and startup. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
A quick update on this subject. Using an Elastic IP won't help with AWS since that only binds to the public interface - not the internal private one. The hostname command still returns the same internal IP address as before, which is what's seen by Riak. In AWS an internal IP address will actually persist across reboots. It does not persist across shutdown and startup. On 8 January 2013 10:16, Richard Shaw rich...@basho.com wrote: Hi Matt, Just to add to Charlie's suggestion, you could take a look at EC2 elastic IP addresses which would allow you to permanently map a public and private address to an EC2 instance, assignDNS hostnames and not have them change on reboot[1] [1] http://aws.amazon.com/articles/1346 Regards Richard On 7 Jan 2013, at 23:03, Charlie Voiselle cvoise...@basho.com wrote: Matt: You would need to use (or implement your own) DNS service that you could programmatically access--Route 53 has an API that you could use to create DNS entries that point to the internal addresses of your nodes. In very carefully re-reading the thread Deepak mentions, one problem that will occur is that each node needs to be able to resolve the other nodes by name also. The only way for this to occur reasonably, would be to register the internal addresses with a single point that they share. Some examples of free services that you might use for this are DynDns[1], DNSDynamic[2], or DNS-O-Matic[3]. I have also seen some projects floating around the web that might enable you to create a self-hosted dynamic DNS like opendyn[4] and GnuDIP[5]; however, I have had no occasion to use something like this in my own environment. Some additional discussion about creating your own Dynamic DNS server is also at http://unix.stackexchange.com/questions/29049/how-to-create-a-custom-dynamic-dns-solution Hope this helps! Charlie [1] http://www.dyn.com [2] http://www.dnsdynamic.org [3] http://www.dnsomatic.com [4] http://code.google.com/p/opendyn/ [5] http://gnudip2.sourceforge.net/ On Jan 7, 2013, at 5:00 PM, Matt Black matt.bl...@jbadigital.com wrote: Thanks for this Charlie. I'm running a production Riak cluster on AWS which runs constantly, and I've been wondering how I might be able to easliy stop and start AWS nodes for a testing and benchmarking cluster (to save on cost). By using the 'riaknode1.priv' hostname method you describe, would I be able to stop and then restart a whole cluster of nodes at once? (As described by Deepak, AWS assigns new IPs when a VM starts). Thanks Matt On 8 January 2013 01:31, Charlie Voiselle cvoise...@basho.com wrote: Deepak: When you name a node in app.config with -name it has to have a '.' in it, like r...@hostname.net As you have surmised, you can get around that if you use the -sname argument instead. They have to be done consistently. In your example, had you used the -sname argument, `riak@riaknode1` would work. Making a host entry `riaknode1.priv` that points to the local address would work with the -name argument. The inportant thing about -name and -sname is that they can't mix within a cluster. Cluster replace is designed to replace a node with a new one and transfer all the partitions. You can cheat and use it to rename a node though. The process to do this would look like the following: • Stop the node to rename with `riak stop` • Mark it 'down' from another node in the cluster using `riak-admin down «old nodename». • Rename the node in vm.args. • Delete the ring directory. • Start the node with `riak start`. • It will come up as a single instance which you can verify with `riak-admin member-status`. • Join the node to the cluster with `riak-admin cluster join «cluster nodename» ` • Set it to replace the old instance of itself with `riak-admin cluster replace «old nodename» «new nodename» • Plan the changes with `riak-admin cluster plan` • Commit the changes with `riak-admin cluster commit` As you can see, this is a very large effort, so best to use hostnames that aren't moving around. Apologies for you getting this twice, Deepak. I failed to reply to the list as well. Hope this makes sense... Charlie On Jan 1, 2013, at 2:43 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. 1. Preventive measures Someone on this thread dated May 2011 suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to
Re: Riak and host names
Deepak: When you name a node in app.config with -name it has to have a '.' in it, like r...@hostname.net As you have surmised, you can get around that if you use the -sname argument instead. They have to be done consistently. In your example, had you used the -sname argument, `riak@riaknode1` would work. Making a host entry `riaknode1.priv` that points to the local address would work with the -name argument. The inportant thing about -name and -sname is that they can't mix within a cluster. Cluster replace is designed to replace a node with a new one and transfer all the partitions. You can cheat and use it to rename a node though. The process to do this would look like the following: Stop the node to rename with `riak stop` Mark it 'down' from another node in the cluster using `riak-admin down «old nodename». Rename the node in vm.args. Delete the ring directory. Start the node with `riak start`. It will come up as a single instance which you can verify with `riak-admin member-status`. Join the node to the cluster with `riak-admin cluster join «cluster nodename» ` Set it to replace the old instance of itself with `riak-admin cluster replace «old nodename» «new nodename» Plan the changes with `riak-admin cluster plan` Commit the changes with `riak-admin cluster commit` As you can see, this is a very large effort, so best to use hostnames that aren't moving around. Apologies for you getting this twice, Deepak. I failed to reply to the list as well. Hope this makes sense... Charlie On Jan 1, 2013, at 2:43 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. 1. Preventive measures Someone on this thread dated May 2011 suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? 2. Use -sname Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) 3. Use cluster replace a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? b . What happens if 2 nodes go down and one was joined with the other. They both have 2 different IP addresses on restart. How will 'cluster replace' work now ? Do let me know your thoughts. Thanks -Deepak ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Thanks for this Charlie. I'm running a production Riak cluster on AWS which runs constantly, and I've been wondering how I might be able to easliy stop and start AWS nodes for a testing and benchmarking cluster (to save on cost). By using the 'riaknode1.priv' hostname method you describe, would I be able to stop and then restart a whole cluster of nodes at once? (As described by Deepak, AWS assigns new IPs when a VM starts). Thanks Matt On 8 January 2013 01:31, Charlie Voiselle cvoise...@basho.com wrote: Deepak: When you name a node in app.config with -name it has to have a '.' in it, like r...@hostname.net As you have surmised, you can get around that if you use the -sname argument instead. They have to be done consistently. In your example, had you used the -sname argument, `riak@riaknode1` would work. Making a host entry `riaknode1.priv` that points to the local address would work with the -name argument. The inportant thing about -name and -sname is that they can't mix within a cluster. Cluster replace is designed to replace a node with a new one and transfer all the partitions. You can cheat and use it to rename a node though. The process to do this would look like the following: - Stop the node to rename with `riak stop` - Mark it 'down' *from another node in the cluster *using `riak-admin down «old nodename». - Rename the node in vm.args. - Delete the ring directory. - Start the node with `riak start`. - It will come up as a single instance which you can verify with `riak-admin member-status`. - Join the node to the cluster with `riak-admin cluster join «cluster nodename» ` - Set it to replace the old instance of itself with `riak-admin cluster replace «old nodename» «new nodename» - Plan the changes with `riak-admin cluster plan` - Commit the changes with `riak-admin cluster commit` As you can see, this is a very large effort, so best to use hostnames that aren't moving around. Apologies for you getting this twice, Deepak. I failed to reply to the list as well. Hope this makes sense... Charlie On Jan 1, 2013, at 2:43 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. *1. Preventive measures* Someone on this thread dated May 2011http://riak-users.197444.n3.nabble.com/EC2-and-node-names-td2892047.html suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? *2. Use -sname* Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) *3. Use cluster replace * a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? b . What happens if 2 nodes go down and one was joined with the other. They both have 2 different IP addresses on restart. How will 'cluster replace' work now ? Do let me know your thoughts. Thanks -Deepak ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Matt: You would need to use (or implement your own) DNS service that you could programmatically access--Route 53 has an API that you could use to create DNS entries that point to the internal addresses of your nodes. In very carefully re-reading the thread Deepak mentions, one problem that will occur is that each node needs to be able to resolve the other nodes by name also. The only way for this to occur reasonably, would be to register the internal addresses with a single point that they share. Some examples of free services that you might use for this are DynDns[1], DNSDynamic[2], or DNS-O-Matic[3]. I have also seen some projects floating around the web that might enable you to create a self-hosted dynamic DNS like opendyn[4] and GnuDIP[5]; however, I have had no occasion to use something like this in my own environment. Some additional discussion about creating your own Dynamic DNS server is also at http://unix.stackexchange.com/questions/29049/how-to-create-a-custom-dynamic-dns-solution Hope this helps! Charlie [1] http://www.dyn.com [2] http://www.dnsdynamic.org [3] http://www.dnsomatic.com [4] http://code.google.com/p/opendyn/ [5] http://gnudip2.sourceforge.net/ On Jan 7, 2013, at 5:00 PM, Matt Black matt.bl...@jbadigital.com wrote: Thanks for this Charlie. I'm running a production Riak cluster on AWS which runs constantly, and I've been wondering how I might be able to easliy stop and start AWS nodes for a testing and benchmarking cluster (to save on cost). By using the 'riaknode1.priv' hostname method you describe, would I be able to stop and then restart a whole cluster of nodes at once? (As described by Deepak, AWS assigns new IPs when a VM starts). Thanks Matt On 8 January 2013 01:31, Charlie Voiselle cvoise...@basho.com wrote: Deepak: When you name a node in app.config with -name it has to have a '.' in it, like r...@hostname.net As you have surmised, you can get around that if you use the -sname argument instead. They have to be done consistently. In your example, had you used the -sname argument, `riak@riaknode1` would work. Making a host entry `riaknode1.priv` that points to the local address would work with the -name argument. The inportant thing about -name and -sname is that they can't mix within a cluster. Cluster replace is designed to replace a node with a new one and transfer all the partitions. You can cheat and use it to rename a node though. The process to do this would look like the following: Stop the node to rename with `riak stop` Mark it 'down' from another node in the cluster using `riak-admin down «old nodename». Rename the node in vm.args. Delete the ring directory. Start the node with `riak start`. It will come up as a single instance which you can verify with `riak-admin member-status`. Join the node to the cluster with `riak-admin cluster join «cluster nodename» ` Set it to replace the old instance of itself with `riak-admin cluster replace «old nodename» «new nodename» Plan the changes with `riak-admin cluster plan` Commit the changes with `riak-admin cluster commit` As you can see, this is a very large effort, so best to use hostnames that aren't moving around. Apologies for you getting this twice, Deepak. I failed to reply to the list as well. Hope this makes sense... Charlie On Jan 1, 2013, at 2:43 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. 1. Preventive measures Someone on this thread dated May 2011 suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? 2. Use -sname Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) 3. Use cluster replace a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? b . What happens if 2 nodes go down and one was joined with the other. They
Re: Riak and host names
Hi Matt, Just to add to Charlie's suggestion, you could take a look at EC2 elastic IP addresses which would allow you to permanently map a public and private address to an EC2 instance, assignDNS hostnames and not have them change on reboot[1] [1] http://aws.amazon.com/articles/1346 Regards Richard On 7 Jan 2013, at 23:03, Charlie Voiselle cvoise...@basho.com wrote: Matt: You would need to use (or implement your own) DNS service that you could programmatically access--Route 53 has an API that you could use to create DNS entries that point to the internal addresses of your nodes. In very carefully re-reading the thread Deepak mentions, one problem that will occur is that each node needs to be able to resolve the other nodes by name also. The only way for this to occur reasonably, would be to register the internal addresses with a single point that they share. Some examples of free services that you might use for this are DynDns[1], DNSDynamic[2], or DNS-O-Matic[3]. I have also seen some projects floating around the web that might enable you to create a self-hosted dynamic DNS like opendyn[4] and GnuDIP[5]; however, I have had no occasion to use something like this in my own environment. Some additional discussion about creating your own Dynamic DNS server is also at http://unix.stackexchange.com/questions/29049/how-to-create-a-custom-dynamic-dns-solution Hope this helps! Charlie [1] http://www.dyn.com [2] http://www.dnsdynamic.org [3] http://www.dnsomatic.com [4] http://code.google.com/p/opendyn/ [5] http://gnudip2.sourceforge.net/ On Jan 7, 2013, at 5:00 PM, Matt Black matt.bl...@jbadigital.com wrote: Thanks for this Charlie. I'm running a production Riak cluster on AWS which runs constantly, and I've been wondering how I might be able to easliy stop and start AWS nodes for a testing and benchmarking cluster (to save on cost). By using the 'riaknode1.priv' hostname method you describe, would I be able to stop and then restart a whole cluster of nodes at once? (As described by Deepak, AWS assigns new IPs when a VM starts). Thanks Matt On 8 January 2013 01:31, Charlie Voiselle cvoise...@basho.com wrote: Deepak: When you name a node in app.config with -name it has to have a '.' in it, like r...@hostname.net As you have surmised, you can get around that if you use the -sname argument instead. They have to be done consistently. In your example, had you used the -sname argument, `riak@riaknode1` would work. Making a host entry `riaknode1.priv` that points to the local address would work with the -name argument. The inportant thing about -name and -sname is that they can't mix within a cluster. Cluster replace is designed to replace a node with a new one and transfer all the partitions. You can cheat and use it to rename a node though. The process to do this would look like the following: • Stop the node to rename with `riak stop` • Mark it 'down' from another node in the cluster using `riak-admin down «old nodename». • Rename the node in vm.args. • Delete the ring directory. • Start the node with `riak start`. • It will come up as a single instance which you can verify with `riak-admin member-status`. • Join the node to the cluster with `riak-admin cluster join «cluster nodename» ` • Set it to replace the old instance of itself with `riak-admin cluster replace «old nodename» «new nodename» • Plan the changes with `riak-admin cluster plan` • Commit the changes with `riak-admin cluster commit` As you can see, this is a very large effort, so best to use hostnames that aren't moving around. Apologies for you getting this twice, Deepak. I failed to reply to the list as well. Hope this makes sense... Charlie On Jan 1, 2013, at 2:43 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. 1. Preventive measures Someone on this thread dated May 2011 suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? 2. Use -sname Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of
Re: Riak and host names
I'm not sure how much this will help, but I will throw my two cents into the hat. Given that I don't know the details of your requirements, and I am fairly new to riak and aws, please take everything with a grain of salt. On Tue, Jan 1, 2013 at 11:43 AM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. *1. Preventive measures* Someone on this thread dated May 2011http://riak-users.197444.n3.nabble.com/EC2-and-node-names-td2892047.html suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 If you add something that looks like a legitimate FQDN (like riaknode1.mydomaind.com) into /etc/hosts and utilize that instead of just riaknode1, that should work just fine. The caveat is that all nodes you your cluster need to have entries in /etc/hosts for all the other nodes. This is where you may have some management headaches. It seems silly to me that erlang requires you to use a different command line option depending on what the host name looks like, but then there are many things that elrang does that seem silly to me. There my be a good technical explanation, but I can't come up with one. Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? *2. Use -sname* Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) As far as erlang is concerned, you absolutely could do this and it should work fine for riak. But the caveate is that you likely will have to rework the riak init scripts or simply write your own. RabbitMQ is another erlang application I have used and it's init script requires that you use the short name with no dots in it and passes that to erlang with -sname. That bugs me a bit as I perfer to use the FQDN, but the amount of work to manage and maintain my own init scripts was just not work it for something that is seemingly just asthetics. *3. Use cluster replace * a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? Rather than micro managing the ips in app.config, I've opted to just have riak listing on any IP by specifying 0.0.0.0 and then restrict access using IP tables to deny access base on the interface name which is generally consistent across all nodes. This means the configs are much more generic and easy to maintain for me. This might be a suitable option for you as well. b . What happens if 2 nodes go down and one was joined with the other. They both have 2 different IP addresses on restart. How will 'cluster replace' work now ? I've not played around with cluster replace much, but given riak likes to do everything by hostname (which it then resolved to an IP), it shouldn't matter if the ip changes as long was the node names don't change and those names resolve to the correct IP addresses. But then that means each time a node comes up, you have to update DNS or /etc/hosts as appropriate which will be a pain. Do let me know your thoughts. Ultimately, I wouldn't want to be building a riak cluster where the IP address of the nodes could/would change any time the node was rebooted. Surely there is a way to make sure the private IP of the nodes say the same across reboots? That will make your life a lot easier. Thanks -Deepak ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Perhaps I should ask the question differently. Can someone elaborate on what 'cluster replace' does and how the command is affected by node names that change after riak starts up ? Thanks -Deepak On Wed, Jan 2, 2013 at 1:13 AM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. *1. Preventive measures* Someone on this thread dated May 2011http://riak-users.197444.n3.nabble.com/EC2-and-node-names-td2892047.html suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? *2. Use -sname* Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) *3. Use cluster replace * a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? b . What happens if 2 nodes go down and one was joined with the other. They both have 2 different IP addresses on restart. How will 'cluster replace' work now ? Do let me know your thoughts. Thanks -Deepak ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak and host names
I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. *1. Preventive measures* Someone on this thread dated May 2011http://riak-users.197444.n3.nabble.com/EC2-and-node-names-td2892047.html suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? *2. Use -sname* Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) *3. Use cluster replace * a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? b . What happens if 2 nodes go down and one was joined with the other. They both have 2 different IP addresses on restart. How will 'cluster replace' work now ? Do let me know your thoughts. Thanks -Deepak ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
