Hi James,
Gonna move this back to the list so that others can add to the
discussion. Definitely sounds like it may be a firewall issue. You
should get "Firewall is stopped" if it is actually stopped.
$ sudo service iptables status
Firewall is stopped.
From your output, it is still running. Here are some instructions on
how to permanently turn it off.
http://www.cyberciti.biz/faq/disable-linux-firewall-under-centos-rhel-fedora/
Since it sounds like you are safe in your companies intranet, I would
suggest trying the above and turn off the firewall completely.
In regards to our install - for a production solution, we build from
sources in a manner that we can actually have multiple installs of both
Erlang and Riak (we are 100% Erlang) to allow us to roll back quickly.
This gives us fine grain control over 1) when we want to upgrade to a
new release, 2) the mount point for our data (we use bitcask), and 3)
the ability to roll back to a previous release. This is true for both
Erlang and Riak. This also assumes that a RIAK upgrade does not
transform physical data - which it has never yet as far as I am aware.
Our solution is pretty simple and is based on a blog post by Robert Aloi:
http://aloiroberto.wordpress.com/2010/11/24/how-to-manage-multiple-erlang-installations/
Here are the steps that I follow (sorry, havent made a shell script out
of it). Would love to hear any feedback from anyone on the list on
where we could be doing better. Following this, I can build a new 5
node cluster from scratch within an hour.
Configuring Erlang
./configure --prefix=/home/riak/erlang/R15B01 --enable-ssl=yes
--enable-kernel-poll --enable-hipe --enable-smp-support --enable-threads
--enable-m64-build
FOR THE MAC:
CFLAGS=-O0 ./configure --enable-hipe --enable-smp-support
--enable-threads --enable-kernel-poll --enable-darwin-64bit
--prefix=/opt/erlang/R15B01
As root:
1. useradd -m -G wheel riak
2. visudo to enable wheel sudoers
3. passwd riak <whatever>
4. su - riak
As riak:
[building erlang]
5. sudo yum install gcc gcc-c++ glibc-devel make ncurses-devel
openssl-devel autoconf git
NOTE: If yum can not install git, do the following:
wget
http://packages.sw.be/rpmforge-release/rpmforge-release-0.3.6-1.el5.rf.i386.rpm
rpm -Uvh http://repo.webtatic.com/yum/centos/5/latest.rpm
yum install --enablerepo=webtatic git-all
6. mkdir erlang
7. cd erlang
8. mkdir R15B01
9. wget http://erlang.org/download/otp_src_R15B01.tar.gz
10. tar zxvf otp_src_R15B01.tar.gz
11. cd otp_src_R15B01
12. ./configure --prefix=/home/riak/erlang/R15B01 --enable-ssl=yes
--enable-kernel-poll --enable-hipe --enable-smp-support --enable-threads
--enable-m64-build
13. make
14. make install
15. cd ~
16. mkdir bin
17. vi .bashrc (add)
RIAK_HOME=$HOME/riak-1.2.0/rel/riak
PATH=$PATH:/$HOME/bin:/sbin:/usr/bin:$RIAK_HOME/bin
18. . .bashrc
19. cd bin
20. vi e15 (add)
#!/bin/bash
env PATH=/home/riak/erlang/R15B01/bin:$PATH "$@"
21. chmod +x e15
to test, type the following on the command line:
e15 erl
22. Do the following. Append these in /etc/sysctl.conf then run sysctl
-p to apply them. No need to reboot, now your kernel should be able to
handle a lot more open connections, yay.
# General gigabit tuning:
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# this gives the kernel more memory for tcp
# which you need with many (100k+) open socket connections
net.ipv4.tcp_mem = 50576 64768 98152
net.core.netdev_max_backlog = 2500
23. Make sure the file max is high enough
sysctl fs.file-max
[building riak]
24. cd /home/riak
25. wget
http://downloads.basho.com.s3-website-us-east-1.amazonaws.com/riak/CURRENT/riak-1.2.0.tar.gz
26. tar zxvf riak-1.2.0.tar.gz
27. cd riak-1.2.0
28. e15 make rel
29. ifconfig to get ipaddress
[configuring the firewall (if needed)]
30. Add
## Firewall
{ kernel, [
{inet_dist_listen_min, 6000},
{inet_dist_listen_max, 7999}
]},
to app.config at top level (same as risk.core)
31. sudo vi /etc/sysconfig/iptables (and add the following to
RH-Firewall-1-INPUT just above the icmp-host-prohibited)
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 4369
-j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8099
-j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport
7010:7015 -j ACCEPT
32. sudo service iptables restart
Cheers,
Bryan
On 10/30/12 8:14 PM, SWEENEY, JAMES wrote:
Hey thanks for the reply Bryan, I did go through the instructions
below. Plus I had checked the iptables right away after I installed,
and iptables does not seem to be running on any of my centos servers
ie this is the result I get when running service iptables status --
[root@essd-riak-test-server ~]# sudo service iptables status
Table: filter
Chain INPUT (policy ACCEPT)
num target prot opt source destination
Chain FORWARD (policy ACCEPT)
num target prot opt source destination
Chain OUTPUT (policy ACCEPT)
num target prot opt source destination
[root@essd-riak-test-server ~]#
The type of install I have done is installed from the dev package from
the riak site, not installed from source code. Does that matter for
clustering? Everything about my install seems fine except that it my
cluster commands fail and I can't join nodes for a clustered environment.
The nodes are on a cloud server that is controlled by another group in
my company. We have made sure the ports that riak needs are opened
for centos to centos communication, and I can telnet from one centos /
riak server to another successfully, but just can join a node.
*From:*riak-users [mailto:[email protected]] *On
Behalf Of *Bryan Hughes
*Sent:* Tuesday, October 30, 2012 5:36 PM
*To:* [email protected]
*Subject:* Re: Issue with clustering Riak nodes on CentOS servers.
Hi James,
We have a 5 node cluster running in production with no problems on
CentOS. We have each node isolated on a private lan. Our application
server has two interfaces, one to the outside world with a very
restricted firewall, and the second connected to the private lan with
the 5 nodes.
Assuming you set the node names accordingly and followed:
http://docs.basho.com/riak/latest/cookbooks/Basic-Cluster-Setup/
One thing to check would your firewall, which is by default on with
CentOS.
>sudo service iptables status
If you have your 4 nodes on a private lan, or perhaps well protected
behind a firewall as an internal deployment, then it is relatively
straight forward. You can just turn off your firewall on the 4 machines.
If your machines are not on a isolated private lan and are exposed to
the internet, or just need to be protected behind a firewall, you will
need to do the following.
http://docs.basho.com/riak/1.2.0/cookbooks/Network-Security-and-Firewall-Configurations/
First, in your app.config, you will need to add the following at the
top level (same as riak.core). You can pick whatever min and max
range you want - here I arbitrarily chosen 4 ports:
{ kernel, [
{inet_dist_listen_min, 7010},
{inet_dist_listen_max, 7014}
]},
Next, on each or our riak nodes, you will need to edit your iptables.
I tend to edit them manually using sudo.
> sudo vi /etc/sysconfig/iptables (and add the following to
RH-Firewall-1-INPUT just above the icmp-host-prohibited)
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 4369
-j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8087
-j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport 8099
-j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m state --state NEW -m tcp --dport
7010:7014 -j ACCEPT
> sudo service iptables restart
After doing this on all the nodes, they should all be visible to each
other. If you wanted to lock down your firewall even further, you can
specify a range of IP addresses.
Finally, on the machine that your erlang application or Riak client,
you will need to do the following:
sudo vi /etc/sysconf/iptable (and add the following to
RH-Firewall-1-INPUT just above the icmp-host-prohibited)
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8098
-j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8087
-j ACCEPT
> sudo service iptables restart
Hope this helps...
Bryan
On 10/30/12 11:39 AM, SWEENEY, JAMES wrote:
Hello, I am attempting to create a 4 node riak installation on 4 separate
CentOS servers. Installation of Riak went well. We have run simple tests
like riak ping, sudo curl \-vhttp://127.0.0.1:8098/riak/test, ect, and
everything seems fine with the riak servers, they all start up with no errors.
We used the default ports in the app.config files. Ie: epmd listener:
TCP:4369
handoff_port listener: TCP:8099
web_port: TCP:8098
pb_port: TCP:8087
We have updated all ip addresses in the config files as instructed on the
riak site. Also, we have verified by telnet that the ports are all open and
that I can telnet between the riak servers. Still after all of that, when I
try to do a cluster command I get the following response:
[root@essd-riak-test-server ~]# riak-admin cluster [email protected]
<mailto:[email protected]>
Attempting to restart script through sudo -H -u riak
[email protected] <mailto:[email protected]> is not reachable!
[root@essd-riak-test-server ~]#
I have been stuck on this issue for quite a while and tried everything I
found on the riak web site to resolve this. Any help you could offer would
greatly be appreciated. Thanks in advance.
Sincerely,
James Sweeney
_______________________________________________
riak-users mailing list
[email protected] <mailto:[email protected]>
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
--
Bryan Hughes
CTO and Founder / *Go Factory*
(415) 515-7916
http://www.go-factory.net
/"Art is never finished, only abandoned. - Leonardo da Vinci"/
--
Bryan Hughes
CTO and Founder / *Go Factory*
(415) 515-7916
http://www.go-factory.net
/"Art is never finished, only abandoned. - Leonardo da Vinci"/
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
