This would be a good FAQ entry Andrew, your explanations are way better than mine. The master publishing itself is new in 0.20 and I guess others will hit this issue with Ubuntu.
J-D On Tue, Nov 24, 2009 at 2:03 PM, Andrew Purtell <[email protected]> wrote: > Good to hear Mark. > > HBase is sensitive to this in a way that Hadoop is not because HBase region > servers and clients do not use a static configuration to locate the master. > If you check your Hadoop config, I suspect you'll find you configured the > namenode location using either a hostname that did not resolve to a localhost > address, or a non-localhost IP address. Likewise perhaps the jobtracker > location. So datanodes, tasktrackers, clients, etc. would use the static > configuration no matter what IP address the namenode, jobtracker, etc. was > self-discovering. However the HBase master publishes its location to clients > via Zookeeper. This is done to support multimaster operation (failover). So > if the HBase master self-discovers its location as a localhost address, then > it will publish that. Region servers or clients which go to Zookeeper for the > master location will get back an address in that case only useful if they > happen to be co-located with the master. > > - Andy > > > > ________________________________ > From: Mark Vigeant <[email protected]> > To: "[email protected]" <[email protected]> > Sent: Tue, November 24, 2009 1:41:15 PM > Subject: RE: Way to Specify HBase master? > > Ah, thank you so much Andy! > > You know, I had just gotten hadoop working again (it turns out it too was > running with master 127.0.1.1 which is odd because it worked while hbase > didn't...) by changing the hostnames on my machines and doing a lot of > frustrated restarting but finally everything worked. > > And I just deleted the 127/8 things in my /etc/hosts and it looks like hbase > is FINALLY running! Woo! > > -Mark > > -----Original Message----- > From: Andrew Purtell [mailto:[email protected]] > Sent: Tuesday, November 24, 2009 4:28 PM > To: [email protected] > Subject: Re: Way to Specify HBase master? > > If you have an entry in your /etc/hosts file on the master that causes it to > resolve its hostname to a localhost address (anything in 127/8) , then this > will happen: > 1) The master will publish its address into zookeeper as a localhost address. > 2) The region servers will attempt to contact the master on the localhost > address... which is their localhost of course > 3) The connection fails unless the region server and the master happen to be > running on the same node. > > In your case, the solution is to remove all entries in /etc/hosts which cause > resolution of the local host name to a localhost address and substitute them > with either a functional DNS setup or a /etc/hosts file which resolves your > host names to the IP addresses assigned on the shared LAN segment (i.e. eth0 > interfaces). > > - Andy > > > > > ________________________________ > From: Mark Vigeant <[email protected]> > To: "[email protected]" <[email protected]> > Sent: Tue, November 24, 2009 9:21:01 AM > Subject: RE: Way to Specify HBase master? > > Wait, I think there's been some confusion: the master resolves itself to > 127.0.1.1, not 127.0.0.1 > > I've looked at the HMaster logs (http://pastebin.com/m4ecf071e) and it says > that in fact what the region servers are reading is true: the master is > running as "hadoop 127.0.1.1." A few interesting things to note are: the ROOT > region is not set, and the /hbase/rs node is completely empty. > > Is this still the matter of a host-name issue? I notice there are dns > properties for all aspects of hbase, should I tamper with them? > > -----Original Message----- > From: [email protected] [mailto:[email protected]] On Behalf Of Jean-Daniel > Cryans > Sent: Tuesday, November 24, 2009 11:40 AM > To: [email protected] > Subject: Re: Way to Specify HBase master? > > It seems to be the same problems isn't it? The master resolves itself > to 127.0.0.1 and all region servers, when communicating with > Zookeeper, read the master's znode with the local address but since > there isn't a master there, it can't connect. > > The reason why you can't specify a master is because HBase is > dynamically multi-master e.g. you can start 100 masters (one is > elected), kill some backups, get new ones, etc. So each node, master > and region server, in HBase has to be able to publish its own address. > > Like Stack said, playing with your hosts file should be enough, just > make sure that the hadoop config is still sane (and try to see why it > fails if so). > > If you have a DNS server, you can also set the following values: > > <property> > <name>hbase.zookeeper.dns.interface</name> > <value>default</value> > <description>The name of the Network Interface from which a ZooKeeper > server > should report its IP address. > </description> > </property> > <property> > <name>hbase.zookeeper.dns.nameserver</name> > <value>default</value> > <description>The host name or IP address of the name server (DNS) > which a ZooKeeper server should use to determine the host name used by > the > master for communication and display purposes. > </description> > </property> > > J-D > > On Tue, Nov 24, 2009 at 8:28 AM, Mark Vigeant > <[email protected]> wrote: >> So I tried to change the hostname and it messed up my hadoop cluster... it >> appears that everything else runs fine on the host hadoop 127.0.1.1 >> >> I'll change strategies here: my regionservers see the host and try to >> connect to it, but the connection is refused. Does anyone have any ideas on >> how I can change that? >> >> Here's a snippet of the log from my regionserver: >> http://pastebin.com/m6bbcb432 >> -----Original Message----- >> From: [email protected] [mailto:[email protected]] On Behalf Of stack >> Sent: Monday, November 23, 2009 5:28 PM >> To: [email protected] >> Subject: Re: Way to Specify HBase master? >> >> This sounds like basic networking issue where host lookup is finding >> 127.0.0.1. Can you get someone to help you setup your networking on your >> machine. Else play w/ your /etc/hosts. Seems like your host name resolves >> to 127.0.0.1. >> St.Ack >> >> On Mon, Nov 23, 2009 at 12:31 PM, Mark Vigeant <[email protected] >>> wrote: >> >>> I'm trying to play around with HBase but it hasn't been working for me. >>> From the look of the logs, it appears that the master is running on >>> 127.0.1.1 instead of my machine (fry) and so all of the regionservers start, >>> but the logs say they cannot connect to the master. From the master logs, it >>> thinks there are 0 regionservers and thus won't do anything for me. I'm >>> using HBase 20.2. How can I set the IP address of the master machine? >>> >>> Mark Vigeant >>> RiskMetrics Group, Inc. > > > > > This email message and any attachments are for the sole use of the intended > recipients and may contain proprietary and/or confidential information which > may be privileged or otherwise protected from disclosure. Any unauthorized > review, use, disclosure or distribution is prohibited. If you are not an > intended recipient, please contact the sender by reply email and destroy the > original message and any copies of the message as well as any attachments to > the original message. > > > >
