THANK YOU. It is now listening on port 60000 Michael
On Thu, Sep 16, 2010 at 3:51 PM, Ryan Rawson <[email protected]> wrote: > Hey, > > Ok the picture is all clear. > > So HBase is a minimally configured system... You dont want to specify > the bind address in your config file, because usually you have 1 file > that you distribute to dozens or even potentially hundreds of systems. > So specifying configuration for 1 system is just not really the way > to go with clustered software. > > So what does hbase do? We need to know the node's identity so when we > register ourselves we know what our IP is, and that IP goes into the > META table. So we grab the hostname (as per 'hostname' on most > systems). Then reverse DNS it, use that IP to bind to. > > In this case, the problem is your hostname is reversing to the > external IP which your host doesnt actually have an interface to. If > you want to run internal network services behind a NAT you will need > to have local IPs and hostnames and not reuse your external name/IP as > internal hostnames. > > So, change your hostname to 'myhost' and make sure it DNS reverses to > 10.0.0.2 (your real IP) and you should be off to the races. > > -ryan > > On Thu, Sep 16, 2010 at 1:06 PM, Michael Scott <[email protected]> > wrote: > > Thanks again. This changes the behavior, but it does not yet fix my > > problem. The hbase.rootdir property forces the hbase master to stay > alive > > for a little while, so I had a moment of short-lived euphoria when > Hmaster > > appeared in the jps list, but this only lasts while it tries to connect > to > > localhost:9000 (which is not open), and it still doesn't open port 60000 > and > > it still thinks it is named my-static-ip.com (i.e., same error message > as > > before). The removal of localhost.localdomain from /etc/hosts made no > > difference one way or the other. I still am looking for a way to try to > > have hbase bind to localhost:6000 instead of my-static-ip.com:6000. I > will > > also try to see why localhost:9000 is not open (though that appears later > in > > the log file, so I don't think it is causing the failure to open 60000). > > Thanks for the help so far, I will post again with further info. > > > > Michael > > > > On Thu, Sep 16, 2010 at 12:53 PM, N.N. Gesli <[email protected]> wrote: > > > >> I have this in hbase-site.xml: > >> > >> <property> > >> <name>hbase.rootdir</name> > >> <value>hdfs://localhost:9000/hbase</value> > >> <description>The directory shared by region servers. > >> Should be fully-qualified to include the filesystem to use. > >> E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR > >> </description> > >> </property> > >> > >> <property> > >> <name>hbase.cluster.distributed</name> > >> <value>true</value> > >> <description>For psuedo-distributed, you want to set this to true. > >> false means that HBase tries to put Master + RegionServers in one > >> process. > >> Pseudo-distributed = seperate processes/pids</description> > >> </property> > >> > >> <property> > >> <name>hbase.regionserver.hlog.replication</name> > >> <value>1</value> > >> <description>For HBase to offer good data durability, we roll logs if > >> filesystem replication falls below a certain amount. In > >> psuedo-distributed > >> mode, you normally only have the local filesystem or 1 HDFS DataNode, > so > >> you > >> don't want to roll logs constantly.</description> > >> </property> > >> > >> <property> > >> <name>hbase.tmp.dir</name> > >> <value>/tmp/hbase-testing</value> > >> <description>Temporary directory on the local > filesystem.</description> > >> </property> > >> > >> I also hase Hadoop conf directory in HBASE_CLASSPATH (hbase-env.sh). > >> > >> I just tried etc/hosts with "127.0.0.1 > localhost.localdomain > >> localhost" line. I got the same error I was getting before. I switched > it > >> back to "127.0.0.1 localhost" and it worked. In between those > >> changes, > >> I stopped hbase, hadoop and killed still running region server. I hope > that > >> helps. > >> > >> N.Gesli > >> > >> > >> > >> > >> On Thu, Sep 16, 2010 at 7:04 AM, Michael Scott <[email protected]> > >> wrote: > >> > >> > This sounds promising, I have one quick question about your steps: > where > >> > in > >> > the Hbase config *site*.xml did you make the change back to localhost? > >> My > >> > hbase master is using the public IP address (97.86.88.18), and I don't > >> > think > >> > I've told it to. I want to convince hbase to get rid of the line in > the > >> > log > >> > file that says something like: > >> > > >> > 2010-09-16 09:59:21,727 INFO org.apache.hadoop.hbase.master.HMaster: > My > >> > address is 97-86-88-18.static.aldl.mi.charter.com:60000 > >> > > >> > (Note that my /etc/hosts has only the one line > >> > 127.0.0.1 localhost.localdomain localhost > >> > since I'm not running ipv6, but somehow hbase knows that the interface > is > >> a > >> > comcast static address. I can use /etc/hosts to change that to the > >> > registered domain name for 97-86-88-18, but this doesn't help.) > >> > > >> > To reply to Ryan's question, my ifconfig gives: > >> > > >> > eth0 Link encap:Ethernet HWaddr 00:24:E8:01:DA:B8 > >> > inet addr:10.0.0.2 Bcast:10.0.0.255 Mask:255.255.255.0 > >> > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > >> > RX packets:319475 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:290698 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:1000 > >> > RX bytes:108186958 (103.1 MiB) TX bytes:187845633 (179.1 > MiB) > >> > Interrupt:28 Base address:0xa000 > >> > > >> > lo Link encap:Local Loopback > >> > inet addr:127.0.0.1 Mask:255.0.0.0 > >> > UP LOOPBACK RUNNING MTU:16436 Metric:1 > >> > RX packets:370795 errors:0 dropped:0 overruns:0 frame:0 > >> > TX packets:370795 errors:0 dropped:0 overruns:0 carrier:0 > >> > collisions:0 txqueuelen:0 > >> > RX bytes:108117402 (103.1 MiB) TX bytes:108117402 (103.1 > MiB) > >> > > >> > Thanks a bunch! > >> > > >> > Michael > >> > > >> > On Thu, Sep 16, 2010 at 1:12 AM, N.N. Gesli <[email protected]> > wrote: > >> > > >> > > Hi Michael, > >> > > > >> > > I was having a similar problem and following this thread for any > >> > > suggestions. I tried everything suggested and more. > >> > > > >> > > I was trying to run Hadoop/Hbase pseudo distributed version on my > Mac. > >> I > >> > > initially started with Hadoop 21.0 and Hbase 0.89 versions. I had > >> exactly > >> > > the same error that you were getting. Then switched to Hadoop 20.2 > and > >> > > Hbase > >> > > 20.6 - still HMaster was not starting. Then finally it worked. Below > >> are > >> > my > >> > > steps to success :) > >> > > > >> > > * stopped hbase > >> > > * stopped hadoop > >> > > * run jps; RegionServer was still running; killed it manually > >> > > * in tmp directory (where hadoop namenode and *.pid files are > stored) I > >> > > removed everything related to hadoop and hbase, including the > >> > directories. > >> > > (I had no data in Hadoop, so I could do this) > >> > > * changed the ports back to default 600** > >> > > * changed back Hadoop and Hbase configurations to "localhost" in > >> > *site*.xml > >> > > and regionservers. (Only I will be using this - no remote > connection) > >> > > * changed back my /etc/hosts to the original version. It looks like > >> this: > >> > > 127.0.0.1 localhost > >> > > ::1 localhost > >> > > fe80::1%lo0 localhost > >> > > * reformatted the Hadoop namenode > >> > > * started Hadoop > >> > > * started HBase and it worked :) > >> > > > >> > > Let me know if you want to know any specific configuration. > >> > > > >> > > N.Gesli > >> > > > >> > > On Wed, Sep 15, 2010 at 10:41 PM, Ryan Rawson <[email protected]> > >> > wrote: > >> > > > >> > > > What is your ifconfig output looking like? > >> > > > > >> > > > > >> > > > > >> > > > On Wed, Sep 15, 2010 at 10:07 PM, Michael Scott < > >> [email protected]> > >> > > > wrote: > >> > > > > Thanks for the continued advice. I am still confused by the > >> > different > >> > > > > behaviors of hadoop and hbase. As I said before, I can't get > hbase > >> to > >> > > > work > >> > > > > on any of the ports that hadoop works on, so I guess hadoop and > >> hbase > >> > > are > >> > > > > using different interfaces. Why is this, and can't I ask hbase > to > >> > use > >> > > > the > >> > > > > interface that hadoop uses? What interfaces are hadoop and > hbase > >> > > using? > >> > > > > > >> > > > > Also (and maybe this is the wrong forum for this question), how > can > >> I > >> > > get > >> > > > my > >> > > > > OS to allow me to open 60000 using the IP address? I have > >> > temporarily > >> > > > > disabled selinux and iptables, as I thought that this would > simply > >> > > allow > >> > > > all > >> > > > > port connections. Still, this works just fine: > >> > > > > bash-4.0$ nc -l 60000 > /tmp/nc.out > >> > > > > > >> > > > > but this does not: > >> > > > > bash-4.0$ nc -l 97.86.88.18 60000 > /tmp/nc.out > >> > > > > (returns "nc: Cannot assign requested address"; I get the same > >> error > >> > > for > >> > > > the > >> > > > > hostname instead of the IP address, and for 10.0.0.1, but > 10.0.0.0 > >> is > >> > > > > allowed) > >> > > > > > >> > > > > I am trying to get hbase running for a socorro server, which > will > >> > > running > >> > > > > locally. I don't know if that matters. > >> > > > > > >> > > > > Thanks, > >> > > > > Michael > >> > > > > > >> > > > > On Wed, Sep 15, 2010 at 6:04 PM, Ryan Rawson < > [email protected]> > >> > > wrote: > >> > > > > > >> > > > >> Hey, > >> > > > >> > >> > > > >> If you bind to localhost you wont actually be reachable by > anyone! > >> > > > >> > >> > > > >> The question is why is your OS disallowing binds to a specific > >> > > > >> interface/port combo? > >> > > > >> > >> > > > >> HBase does not really run in a blended/multihomed > environment... > >> > > > >> meaning if you have multiple interfaces, you have to choose one > >> that > >> > > > >> we work over. This is because we need to know a singular > >> canonical > >> > > > >> IP/name for any given server because we put that info up inside > >> > > > >> ZooKeeper and META tables. So it's not just an artificial > >> > constraint, > >> > > > >> but exists for cluster management needs. > >> > > > >> > >> > > > >> Having said that, we do work on multihomed machines, eg: ec2, > you > >> > > > >> might bind hbase to the internal interface taking advantage of > the > >> > > > >> unmetered/faster network. Also better for security as well. > >> > > > >> > >> > > > >> Let us know if you need more background on how we use the > network > >> > and > >> > > > why. > >> > > > >> -ryan > >> > > > >> > >> > > > >> On Wed, Sep 15, 2010 at 10:18 AM, Michael Scott < > >> > [email protected] > >> > > > > >> > > > >> wrote: > >> > > > >> > Hi again, > >> > > > >> > > >> > > > >> > I think the hbase server master is not starting because it is > >> > > > attempting > >> > > > >> to > >> > > > >> > open port 60000 on its public IP address, rather than using > >> > > localhost. > >> > > > I > >> > > > >> > cannot seem to figure out how to force it (well, configure > it) > >> to > >> > > > attempt > >> > > > >> to > >> > > > >> > bind to localhost:60000 instead. As far as I can see, this > is > >> > set > >> > > in > >> > > > >> the > >> > > > >> > file: > >> > > > >> > > >> > > > >> > org/apache/hadoop/hbase/master/HMaster.java > >> > > > >> > > >> > > > >> > I don't know much about java, so I'd prefer not to edit the > >> source > >> > > if > >> > > > >> there > >> > > > >> > is an option, but I will if necessary. Can someone please > point > >> > me > >> > > to > >> > > > >> the > >> > > > >> > way to change this setting? Any help would be greatly > >> > appreciated. > >> > > > >> > > >> > > > >> > Thanks, > >> > > > >> > Michael > >> > > > >> > > >> > > > >> > On Wed, Sep 15, 2010 at 12:42 AM, Michael Scott < > >> > > [email protected] > >> > > > >> >wrote: > >> > > > >> > > >> > > > >> >> Hi again, > >> > > > >> >> > >> > > > >> >> IPV6 was enabled. I shut it off, rebooted to be sure, > verified > >> > it > >> > > > was > >> > > > >> >> still off, and encountered the same problem once again. > >> > > > >> >> > >> > > > >> >> I also tried to open port 60000 by hand with a small php > file. > >> I > >> > > can > >> > > > do > >> > > > >> >> this (as any user) for localhost. I can NOT do this (not > even > >> as > >> > > > root) > >> > > > >> for > >> > > > >> >> the IP address which matches the fully qualified domain > name, > >> > which > >> > > > is > >> > > > >> what > >> > > > >> >> hbase is trying to use. Is there some way for me to > configure > >> > > hbase > >> > > > to > >> > > > >> use > >> > > > >> >> localhost instead of the fully qualified domain name for the > >> > > master? > >> > > > I > >> > > > >> >> would have thought this was done by default, or that there > >> would > >> > be > >> > > > an > >> > > > >> >> obvious line in some conf file, but I can't find it. > >> > > > >> >> > >> > > > >> >> Thanks again, > >> > > > >> >> > >> > > > >> >> Michael > >> > > > >> >> > >> > > > >> >> On Tue, Sep 14, 2010 at 12:23 PM, Todd Lipcon < > >> [email protected] > >> > > > >> > > > >> wrote: > >> > > > >> >> > >> > > > >> >>> Hi Michael, > >> > > > >> >>> > >> > > > >> >>> It might be related to IPV6. Do you have IPV6 enabled on > this > >> > > > machine? > >> > > > >> >>> > >> > > > >> >>> Check out this hadoop JIRA that might be related for some > >> tips: > >> > > > >> >>> https://issues.apache.org/jira/browse/HADOOP-6056 > >> > > > >> >>> > >> > > > >> >>> <https://issues.apache.org/jira/browse/HADOOP-6056>-Todd > >> > > > >> >>> > >> > > > >> >>> On Tue, Sep 14, 2010 at 10:17 AM, Michael Scott < > >> > > > [email protected] > >> > > > >> >>> >wrote: > >> > > > >> >>> > >> > > > >> >>> > That's correct. I tried a number of different ports to > see > >> if > >> > > > there > >> > > > >> was > >> > > > >> >>> > something weird, and then I shut down the hadoop server > and > >> > > tried > >> > > > to > >> > > > >> >>> > connect > >> > > > >> >>> > to 50010 (which of course should have been free at that > >> point) > >> > > but > >> > > > >> got > >> > > > >> >>> the > >> > > > >> >>> > same "cannot assign to requested address" error. If I > start > >> > > > hadoop, > >> > > > >> >>> > netstat > >> > > > >> >>> > shows a process listening on 50010. > >> > > > >> >>> > > >> > > > >> >>> > I am going to try this on a different OS, I am wondering > if > >> > FC11 > >> > > > is > >> > > > >> my > >> > > > >> >>> > problem. > >> > > > >> >>> > > >> > > > >> >>> > Michael > >> > > > >> >>> > > >> > > > >> >>> > On Tue, Sep 14, 2010 at 11:41 AM, Stack < > [email protected]> > >> > > wrote: > >> > > > >> >>> > > >> > > > >> >>> > > On Tue, Sep 14, 2010 at 9:33 AM, Michael Scott < > >> > > > >> [email protected]> > >> > > > >> >>> > > wrote: > >> > > > >> >>> > > > I don't see why hadoop binds > >> > > > >> >>> > > > to a port but hbase does not (I even tried starting > >> hbase > >> > > with > >> > > > >> >>> hadoop > >> > > > >> >>> > off > >> > > > >> >>> > > > and binding to 50010, which hadoop uses). > >> > > > >> >>> > > > > >> > > > >> >>> > > > >> > > > >> >>> > > Using 50010 worked for hadoop but not for hbase? (Odd. > >> We > >> > > > hadoop > >> > > > >> >>> > > their mechanism essentially). > >> > > > >> >>> > > > >> > > > >> >>> > > St.Ack > >> > > > >> >>> > > > >> > > > >> >>> > > >> > > > >> >>> > >> > > > >> >>> > >> > > > >> >>> > >> > > > >> >>> -- > >> > > > >> >>> Todd Lipcon > >> > > > >> >>> Software Engineer, Cloudera > >> > > > >> >>> > >> > > > >> >> > >> > > > >> >> > >> > > > >> > > >> > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> > > >
