Actually found the problem - bad pasting in vi! Doh! <name>hbase.rootdir</name> <value>hdfs://hdfs://svrh124:9000/hbase</value>
Looks good now. Going to test each server and make sure it's working. -----Original Message----- From: Anthony Ikeda [mailto:[email protected]] Sent: Thursday, 3 June 2010 3:51 PM To: [email protected] Subject: RE: Trying to get the region servers working.... Okay, here's the situation. My Master is not starting. I think I'm missing a setting: 2010-06-03 15:43:17,813 ERROR org.apache.hadoop.hbase.master.HMaster: Can not start master java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorA ccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingCons tructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.master.HMaster.doMain(HMaster.java:1230) at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1271) Caused by: java.lang.NumberFormatException: For input string: "" at java.lang.NumberFormatException.forInputString(NumberFormatException.jav a:48) at java.lang.Integer.parseInt(Integer.java:470) at java.lang.Integer.parseInt(Integer.java:499) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:146) at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java :156) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileS ystem.java:81) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95) at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:192) ... 6 more I'm just not sure which one..... -----Original Message----- From: Anthony Ikeda [mailto:[email protected]] Sent: Thursday, 3 June 2010 3:02 PM To: [email protected] Subject: RE: Trying to get the region servers working.... Cool, all network issues sorted, but now this exception: KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/master I've sent an ruok within ZooKeeper and this is the reponse: [zkee...@svrh124 zookeeper-3.2.2]$ echo ruok | nc 127.0.0.1 2181 2010-06-03 14:57:52,009 - INFO [NIOServerCxn.Factory:2181:nioserverc...@651] - Processing ruok command from /127.0.0.1:60343 2010-06-03 14:57:52,010 - WARN [NIOServerCxn.Factory:2181:nioserverc...@518] - Exception causing close of session 0x0 due to java.io.IOException: Responded to info probe 2010-06-03 14:57:52,010 - INFO [NIOServerCxn.Factory:2181:nioserverc...@857] - closing session:0x0 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/127.0.0.1:2181 remote=/127.0.0.1:60343] imok [zkee...@svrh124 zookeeper-3.2.2]$ So it looks like ZooKeeper is working. How do I force the creation of the /hbase/master in ZooKeeper? Anthony -----Original Message----- From: Anthony Ikeda [mailto:[email protected]] Sent: Thursday, 3 June 2010 2:23 PM To: [email protected] Subject: RE: Trying to get the region servers working.... Thanks St.ack, I've found out there is an issue with the hosts file - apparently we are not using them and there is some other config to be sorted - so once the administrators have fixed this issue I will: * Set up a port in core-site.xml for hadoop * Configure the port for the hbase.rootdir for HBase * Remove the hbase.master property * Stop HBase, ZooKeeper and Hadoop * Start Hadoop, ZooKeeper and then HBase Just out of curiosity, is ZooKeeper dependant on Hadoop at all, or is this a side-by-side running app? i.e, the order I start Hadoop and ZooKeeper is irrelevant? Anthony -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Stack Sent: Thursday, 3 June 2010 2:14 PM To: [email protected] Subject: Re: Trying to get the region servers working.... On Wed, Jun 2, 2010 at 5:37 PM, Anthony Ikeda <[email protected] > wrote: > However the region servers seem to keep resolving the master server to > 127.0.0.1:60000 > > > Your machines are finding 127.0.0.1 when they try to figure their machinename it seems. Can you fix that? Both master and regionserver are doing this to figure out who they are: String addressStr = DNS.getDefaultHost( conf.get("hbase.master.dns.interface","default"), conf.get("hbase.master.dns.nameserver","default")); .. then they are hoisting their address up into zk. >From what I can tell in the ZooKeeper logs, it has started successfully and > is communicating. > > ${HBASE_HOME/logs/ hbase-hbase-zookeeper-SVRH124.log > > 2010-06-03 10:05:47,286 INFO org.apache.zookeeper.server.ZooKeeperServer: > Created server > > 2010-06-03 10:05:47,288 INFO org.apache.zookeeper.server.quorum.Follower: > Following /172.28.2.136:2888 > > Its interesting that zk can figure out hostname. > The hbase-site.xml for each server is configured as: > > <configuration> > > <property> > > <name>hbase.rootdir</name> > > <value>hdfs://172.28.1.138/hbase</value> > > </property> > > Don't you want to add a port on there? > <property> > > <name>hbase.master</name> > > <value>172.28.1.138:60000</value> > > </property> > > This property is not used any more (presuming you are using a >= 0.20.x hbase. > Question: > > Why can't the region servers contact the master? I've checked the > /etc/hosts file and there are 2 entries to resolve the server name > (127.0.0.1 and 172.28.x.x) with 127.0.0.1 coming first. But I've been told > not to change this as it affects other functions of the server. > > Try playing w/ port above. See if that'll fix it. St.Ack _____________________________________________________________________ This e-mail has been scanned for viruses by MCI's Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.mci.com ********************************************************************** This e-mail message and any attachments are intended only for the use of the addressee(s) named above and may contain information that is privileged and confidential. If you are not the intended recipient, any display, dissemination, distribution, or copying is strictly prohibited. If you believe you have received this e-mail message in error, please immediately notify the sender by replying to this e-mail message or by telephone to (02) 9646 9222. Please delete the email and any attachments and do not retain the email or any attachments in any form. ********************************************************************** _____________________________________________________________________ This e-mail has been scanned for viruses by MCI's Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.mci.com ********************************************************************** This e-mail message and any attachments are intended only for the use of the addressee(s) named above and may contain information that is privileged and confidential. If you are not the intended recipient, any display, dissemination, distribution, or copying is strictly prohibited. If you believe you have received this e-mail message in error, please immediately notify the sender by replying to this e-mail message or by telephone to (02) 9646 9222. Please delete the email and any attachments and do not retain the email or any attachments in any form. ********************************************************************** _____________________________________________________________________ This e-mail has been scanned for viruses by MCI's Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.mci.com ********************************************************************** This e-mail message and any attachments are intended only for the use of the addressee(s) named above and may contain information that is privileged and confidential. If you are not the intended recipient, any display, dissemination, distribution, or copying is strictly prohibited. If you believe you have received this e-mail message in error, please immediately notify the sender by replying to this e-mail message or by telephone to (02) 9646 9222. Please delete the email and any attachments and do not retain the email or any attachments in any form. ********************************************************************** _____________________________________________________________________ This e-mail has been scanned for viruses by MCI's Internet Managed Scanning Services - powered by MessageLabs. For further information visit http://www.mci.com ********************************************************************** This e-mail message and any attachments are intended only for the use of the addressee(s) named above and may contain information that is privileged and confidential. If you are not the intended recipient, any display, dissemination, distribution, or copying is strictly prohibited. If you believe you have received this e-mail message in error, please immediately notify the sender by replying to this e-mail message or by telephone to (02) 9646 9222. Please delete the email and any attachments and do not retain the email or any attachments in any form. **********************************************************************
