check if password less ssh is enabled for local server or not. if you have already created it try to remove .ssh folder in /home and try recreating .some time its problem related to this also. just a wild guess. have a look on this.
Configuring SSH Hadoop requires SSH access to manage its nodes, i.e. remote machines plus your local machine if you want to use Hadoop on it (which is what we want to do in this short tutorial). For our single-node setup of Hadoop, we therefore need to configure SSH access to localhost for the hadoop user we create in the previous section. I assume that you have SSH up and running on your machine and configured it to allow SSH public key authentication. If not, there are several guides<http://ubuntuguide.org/> available. First, we have to generate an SSH key for the <tt>hadoop</tt> user. noll@ubuntu:~$ su - hadoop hadoop@ubuntu:~$ ssh-keygen -t rsa -P "" Generating public/private rsa key pair. Enter file in which to save the key (/home/hadoop/.ssh/id_rsa): Created directory '/home/hadoop/.ssh'. Your identification has been saved in /home/hadoop/.ssh/id_rsa. Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub. The key fingerprint is: 9d:47:ab:d7:22:54:f0:f9:b9:3b:64:93:12:75:81:27 hadoop@ubuntu hadoop@ubuntu:~$ The second line will create an RSA key pair with an empty password. Generally, using an empty password is not recommended, but in this case it is needed to unlock the key without your interaction (you don't want to enter the passphrase every time Hadoop interacts with its nodes). Second, you have to enable SSH access to your local machine with this newly created key. hadoop@ubuntu:~$ cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys The final step is to test the SSH setup by connecting to your local machine with the hadoop user. The step is also needed to save your local machine's host key fingerprint to the hadoop user'sknown_hosts file. If you have any special SSH configuration for your local machine like a non-standard SSH port, you can define host-specific SSH options in $HOME/.ssh/config (see man ssh_config for more information). hadoop@ubuntu:~$ ssh localhost The authenticity of host 'localhost (127.0.0.1)' can't be established. RSA key fingerprint is 76:d7:61:86:ea:86:8f:31:89:9f:68:b0:75:88:52:72. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'localhost' (RSA) to the list of known hosts. Ubuntu 7.04 ... hadoop@ubuntu:~$ If the SSH connect should fail, these general tips might help: - Enable debugging with ssh -vvv localhost and investigate the error in detail. - Check the SSH server configuration in /etc/ssh/sshd_config, in particular the options PubkeyAuthentication (which should be set to yes) and AllowUsers (if this option is active, add the <tt>hadoop</tt> user to it). If you made any changes to the SSH server configuration file, you can force a configuration reload with sudo /etc/init.d/ssh reload. Disabling IPv6 I have not found out yet how to configure Hadoop to listen on *all IPv4* (again: IPv4) network interfaces. Using 0.0.0.0 for the various networking-related Hadoop configuration options will result in Hadoop binding to the *IPv6* addresses on my Ubuntu box. As a workaround (and realizing that there's no practical point in enabling IPv6 on a box when you are not connected to any IPv6 network), I simply disabled IPv6 on my Ubuntu machine. To disable IPv6 on Ubuntu Linux, open /etc/modprobe.d/blacklist in the editor of your choice and add the following lines to the end of the file: # disable IPv6 blacklist ipv6 You have to reboot your machine in order to make the changes take effect. 2011/12/16 exp <[email protected]> > hi Mohammad Tariq, > > > thanks for reply. > > > I follow your instruction, change the hosts to this: > master: > 127.0.0.1 localhost 127.0.0.1 localhost ubuntu 10.66.201.243 master > 10.66.201.244 slave1 10.66.201.245 slave2 > > > slave1: > 127.0.0.1 localhost 127.0.0.1 slave1 ubuntu 10.66.201.243 master > 10.66.201.244 slave1 10.66.201.245 slave2 > > slave2: > 127.0.0.1 localhost > 127.0.0.1 slave2 ubuntu > 10.66.201.243 master > 10.66.201.244 slave1 > 10.66.201.245 slave2 > > > It still doesn't work. the same problem. > The regionserver try to connect to localhost:60000. And the hmaster still > start at localhost:60000. > > > > > > > > At 2011-12-16 15:37:53,"Mohammad Tariq" <[email protected]> wrote: > >Hi exp, > > > > Do not remove this line, instead make it 127.0.0.1..and copy the > >hadoop-core-0.20.204.0.jar from your HADOOP_HOME and > >commons-configuration-1.6.jar from the HADOOP_HOME/lib folder to the > >HBASE_HOME/lib folder. It should work then..Please let me know if it > >works for you. > > > >Regards, > > Mohammad Tariq > > > > > > > >2011/12/16 exp <[email protected]>: > >> hi, > >> > >> > >> I'm using Hadoop 0.20.204.0 > >> > >> > >> After I remove the 127.0.1.1 lines, the HMaster cannot start. I get > this exception: > >> > >> > >> 2011-12-16 13:37:11,899 ERROR > org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master > java.lang.RuntimeException: Failed construction of Master: class > org.apache.hadoop.hbase.master.HMaster at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1060) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:142) > at > org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:102) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at > org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76) > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1074) Caused > by: java.net.BindException: Cannot assign requested address at > sun.nio.ch.Net.bind(Native Method) at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:126) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:59) at > org.apache.hadoop.hbase.ipc.HBaseServer.bind(HBaseServer.j > ava > >> :201) at > org.apache.hadoop.hbase.ipc.HBaseServer$Listener.<init>(HBaseServer.java:270) > at org.apache.hadoop.hbase.ipc.HBaseServer.<init>(HBaseServer.java:1168) at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.<init>(HBaseRPC.java:544) at > org.apache.hadoop.hbase.ipc.HBaseRPC.getServer(HBaseRPC.java:514) at > org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:196) at > sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1055) > ... 5 more > >> > >> > >> what address should the hmaster rpcserver bind to? > >> > >> > >> > >> > >> cheers > >> exception > >> > >> > >> At 2011-12-16 02:37:43,"Mohammad Tariq" <[email protected]> wrote: > >>>I agree with J-D and Shashwat. BTW, which version of Hadoop are you > using?? > >>>Regards, > >>> Mohammad Tariq > >>> > >>> > >>> > >>>On Thu, Dec 15, 2011 at 11:56 PM, shashwat shriparv > >>><[email protected]> wrote: > >>>> make 127.0.1.1 to 127.0.0.1 that will solve lot of problems > >>>> > >>>> On Thu, Dec 15, 2011 at 11:54 PM, Jean-Daniel Cryans < > [email protected]>wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> A few notes: > >>>>> > >>>>> Remove the 127.0.1.1 lines, they usually mess things up. > >>>>> > >>>>> The hbase.master configuration has been removed from the HBase code > >>>>> more than 2 years ago, you can remove it too. > >>>>> > >>>>> Setting hbase.master.dns.interface alone without > >>>>> hbase.master.dns.nameserver doesn't do anything if I remember > >>>>> correctly, also I'm pretty sure you don't have an interface named > >>>>> "master", it's more likely "eth0" and whatnot. > >>>>> > >>>>> I don't know if this will fix your DNS resolution, but at least it > >>>>> will clean up a bit :) > >>>>> > >>>>> J-D > >>>>> > >>>>> 2011/12/15 exp <[email protected]>: > >>>>> > hi all, > >>>>> > > >>>>> > I am installing hbase on a small cluster of 3 machines. The > RegionServer > >>>>> unable connect to the master. This is the log: > >>>>> > > >>>>> > > >>>>> > 2011-12-15 13:46:43,415 INFO > >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Attempting > connect to > >>>>> Master server at localhost:60000 > >>>>> > 2011-12-15 13:47:43,473 WARN > >>>>> org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to > connect to > >>>>> master. Retrying. Error was: > >>>>> > java.net.ConnectException: Connection refused > >>>>> > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > >>>>> > at > >>>>> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567) > >>>>> > at > >>>>> > org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206) > >>>>> > ...... > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > The lod shows this RegionServer connect to the master at a wrong > >>>>> address(localhost:60000). > >>>>> > > >>>>> > > >>>>> > as far as I know, the RegionServer use ZooKeeper to find out the > >>>>> master's address. so I check the HMaster's log. > >>>>> > > >>>>> > > >>>>> > this is what i find out: > >>>>> > ... > >>>>> > 2011-12-15 13:44:34,946 INFO > >>>>> org.apache.hadoop.hbase.master.metrics.MasterMetrics: Initialized > >>>>> > 2011-12-15 13:44:35,010 INFO > >>>>> org.apache.hadoop.hbase.master.ActiveMasterManager: > Master=localhost:60000 > >>>>> > 2011-12-15 13:44:37,476 INFO org.apache.zookeeper.ZooKeeper: > Initiating > >>>>> client connection, connectString=master:2181,slave2:2181,slave > >>>>> > 1:2181 sessionTimeout=180000 watcher=hconnection > >>>>> > ... > >>>>> > > >>>>> > > >>>>> > The HMaster starts at localhost:60000! I think the HMaster should > start > >>>>> at master:60000. "master" is the hosts where JobTracker runs. > >>>>> > > >>>>> > > >>>>> > I am using hbase 0.90.3 and zookeeper 3.3.4. This is the source > code > >>>>> where the HMaster get the address: > >>>>> > > >>>>> > > >>>>> > HServerAddress a = new HServerAddress(getMyAddress(this.conf)); > >>>>> > int numHandlers = conf.getInt("hbase.regionserver.handler.count", > 10); > >>>>> > this.rpcServer = HBaseRPC.getServer(this, > >>>>> > new Class<?>[]{HMasterInterface.class, > >>>>> HMasterRegionInterface.class}, > >>>>> > a.getBindAddress(), a.getPort(), > >>>>> > numHandlers, > >>>>> > 0, // we dont use high priority handlers in master > >>>>> > false, conf, > >>>>> > 0); // this is a DNC w/o high priority handlers > >>>>> > this.address = new HServerAddress(rpcServer.getListenerAddress()); > >>>>> > > >>>>> > > >>>>> > private static String getMyAddress(final Configuration c) > >>>>> > throws UnknownHostException { > >>>>> > // Find out our address up in DNS. > >>>>> > String s = > >>>>> DNS.getDefaultHost(c.get("hbase.master.dns.interface","default"), > >>>>> > c.get("hbase.master.dns.nameserver","default")); > >>>>> > s += ":" + c.get(HConstants.MASTER_PORT, > >>>>> > Integer.toString(HConstants.DEFAULT_MASTER_PORT)); > >>>>> > return s; > >>>>> > } > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > As you can see in the code, the HMaster find its address in DNS! I > have > >>>>> no idea how to setup the hbase to let the HMaster find the correct > address. > >>>>> > > >>>>> > > >>>>> > this is my hbase-site.xml: > >>>>> > > >>>>> > > >>>>> > <configuration> > >>>>> > <property> > >>>>> > <name>hbase.rootdir</name> > >>>>> > <value>hdfs://master:9000/user/hadoop/hbase</value> > >>>>> > </property> > >>>>> > <property> > >>>>> > <name>hbase.cluster.distributed</name> > >>>>> > <value>true</value> > >>>>> > </property> > >>>>> > > >>>>> > > >>>>> > <property> > >>>>> > <name>hbase.master.dns.interface</name> > >>>>> > <value>master</value> > >>>>> > </property> > >>>>> > > >>>>> > > >>>>> > <property> > >>>>> > <name>zookeeper.znode.master</name> > >>>>> > <value>master</value> > >>>>> > </property> > >>>>> > > >>>>> > > >>>>> > <property> > >>>>> > <name>hbase.master</name> > >>>>> > <value>master:60000</value> > >>>>> > </property> > >>>>> > </configuration> > >>>>> > > >>>>> > > >>>>> > and all the three machines are virtual machine with Ubuntu 11.04 > >>>>> > > >>>>> > > >>>>> > this is the /etc/hosts : > >>>>> > master: > >>>>> > 127.0.0.1 localhost ubt > >>>>> > 127.0.0.1 ubt ubuntu > >>>>> > 10.66.201.243 master > >>>>> > 10.66.201.244 slave1 > >>>>> > 10.66.201.245 slave2 > >>>>> > > >>>>> > > >>>>> > slave1: > >>>>> > 127.0.0.1 localhost > >>>>> > 127.0.1.1 slave1 ubuntu > >>>>> > 10.66.201.243 master > >>>>> > 10.66.201.244 slave1 > >>>>> > 10.66.201.245 slave2 > >>>>> > > >>>>> > > >>>>> > slave2: > >>>>> > 127.0.0.1 localhost > >>>>> > 127.0.1.1 slave2 ubuntu > >>>>> > 10.66.201.243 master > >>>>> > 10.66.201.244 slave1 > >>>>> > 10.66.201.245 slave2 > >>>>> > > >>>>> > > >>>>> > any idea on this problems? > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> > cheers > >>>>> > > >>>>> > > >>>>> > exception > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Shashwat Shriparv > >>>> 09900059620 > >>>> 09663531241 > >>>> > >>>> > >>>> > >>>> <iframe src=" > >>>> > http://rcm.amazon.com/e/cm?t=shriparv-20&o=1&p=48&l=ur1&category=kindlerotating&f=ifr > " > >>>> width="728" height="90" scrolling="no" border="0" marginwidth="0" > >>>> style="border:none;" frameborder="0"></iframe> > -- Shashwat Shriparv 09900059620 09663531241 <iframe src=" http://rcm.amazon.com/e/cm?t=shriparv-20&o=1&p=48&l=ur1&category=kindlerotating&f=ifr"; width="728" height="90" scrolling="no" border="0" marginwidth="0" style="border:none;" frameborder="0"></iframe>
