Hi whirr users
I'm trying to deploy a small hbase cluster on ec2, and I'm hitting the
following network issue
Here's my config file :
whirr.cluster-name=my-hbase-cluster
whirr.instance-templates=1
zookeeper+hadoop-namenode+hadoop-jobtracker+hbase-master,2
hadoop-datanode+hadoop-tasktracker+hbase-regionserver
hbase-site.dfs.replication=1
whirr.provider=aws-ec2
whirr.identity=<my_id>
whirr.credential=<my_cred>
whirr.hardware-id=m1.large
whirr.image-id=eu-west-1/ami-895069fd
whirr.location-id=eu-west-1
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_whirr
whirr.public-key-file=${whirr.private-key-file}.pub
whirr.hbase.tarball.url=
http://apache.cict.fr/hbase/hbase-0.90.4/hbase-0.90.4.tar.gz
whirr.hadoop.tarball.url=
http://archive.cloudera.com/cdh/3/hadoop-0.20.2-cdh3u1.tar.gz
So the configuration is pretty standard...
The problem is : the regions servers can't talk to the master server,
because port 60000 does not seem to be opened (the hbase master rpc port if
I get it correctly)
* Througt telnet :
telnet ip-10-58-170-126.eu-west-1.compute.internal 60000
Trying 10.58.170.126...
telnet: Unable to connect to remote host: Connection refused
* In the region server log :
2012-03-12 12:53:20,889 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting connect to
Master server at ip-10-58-170-126.eu-west-1.compute.internal:60000
2012-03-12 12:54:21,000 WARN
org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to connect to
master. Retrying. Error was:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:408)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:328)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:883)
at
org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:750)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:257)
at $Proxy5.getProtocolVersion(Unknown Source)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:419)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:393)
at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:444)
at
org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:349)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.getMaster(HRegionServer.java:1462)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1515)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.tryReportForDuty(HRegionServer.java:1499)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:572)
at java.lang.Thread.run(Thread.java:662)
Note that other hadoop / hbase related ports look fine : i can telnet from
the region server to the master on port 60010 for example.
The hadoop logs on the region servers (who also act as datanodes /
tasktrackers) look fine
The EC2 security group also look fine : ports 1 - 65535 for tcp and udp
seem to be opened for the whole security group.
I'm using whirr 0.7.1, and tried various ubuntu AMIs / hbase+hadoop
combinations
Any idea on what's going on here ?
Best regards
Fred
