Re: Namenode cannot accept connection from datanode

Dennis Kubes Mon, 14 May 2007 10:31:59 -0700

We have run into this problem before. If you have a static address forthe machine, make sure that your hosts file is pointing to the staticaddress for the namenode host name as opposed to the 127.0.0.1 address.It should look something like this with the values replaced with yourvalues.


127.0.0.1               localhost.localdomain localhost
192.x.x.x               yourhost.yourdomain.com yourhost


Dennis Kubes

Cedric Ho wrote:

Oh, and I also tried to use 192.168.1.179 as a datanode itself, and
only this datanode connect to the namenode on this same host
successfully.

On 5/14/07, Cedric Ho <[EMAIL PROTECTED]> wrote:

I performed more testing on this. While the namenode is running, I
cannot connect to 192.168.1.179:9000 from other machines, but I can
connect to it locally. It seems that the serverSocket only bind to the
127.0.0.1:9000 but not 192.168.1.179:9000.

I've also confirmed that there's no firewall, connection bocking etc
on this machine. In fact I've written a small Java program that open a
serverSocket on 9000, started with the same user on the same machine.
And I am able to connect to it from all other machines.

So is there some settings that will cause the namenode to only bind to
the 9000 port on the local interface ?

Cedric


On 5/12/07, Michael Bieniosek <[EMAIL PROTECTED]> wrote:
> I would try to debug this as a network problem - when the namenode is

> running, can you connect to 192.168.1.179:9000 from the machine thedatanode

> is on?
>

> While the namenode does use a lot of RAM as the cluster sizeincreases, an

> overloaded namenode will typically start panicking in its log messages.

> This doesn't occur in your namenode logs - it doesn't appear anydatanodes

> connected at all.
>
> -Michael
>
> On 5/10/07 7:39 PM, "Cedric Ho" <[EMAIL PROTECTED]> wrote:
>
> > Hi all,
> >
> > We were trying to setup hadoop in our linux environments. When we
> > tried to use a slow machine as the namenode (some Pentium III machine

> > with 512Mb ram). It seems that it was unable to accept connectionfrom

> > other datanodes. (I can access its status from http at port 50070
> > however).
> >
> > But it works fine on a faster machine (Pentium4 3Ghz with 3Gb ram).
> > The settings, etc are exactly the same.
> >

> > The problem seems to be on the namenode. Is it because the machineis slow ?

> >
> > The version we use is 0.12.3
> >
> > Any help is appreciated.
> >
> >
> > Following is the log from the abnormal namenode.
> >

> > 2007-05-09 18:18:46,998 INFO org.apache.hadoop.dfs.StateChange:STATE*

> > Network topology has 0 racks and 0 datanodes

> > 2007-05-09 18:18:47,000 INFO org.apache.hadoop.dfs.StateChange:STATE*

> > UnderReplicatedBlocks has 0 blocks
> > 2007-05-09 18:18:47,432 INFO org.mortbay.util.Credential: Checking
> > Resource aliases

> > 2007-05-09 18:18:48,051 INFO org.mortbay.http.HttpServer: VersionJetty/5.1.4

> > 2007-05-09 18:18:50,524 INFO org.mortbay.util.Container: Started
> > [EMAIL PROTECTED]
> > 2007-05-09 18:18:51,064 INFO org.mortbay.util.Container: Started
> > WebApplicationContext[/,/]
> > 2007-05-09 18:18:51,065 INFO org.mortbay.util.Container: Started
> > HttpContext[/logs,/logs]
> > 2007-05-09 18:18:51,065 INFO org.mortbay.util.Container: Started
> > HttpContext[/static,/static]
> > 2007-05-09 18:18:51,147 INFO org.mortbay.http.SocketListener: Started
> > SocketListener on 0.0.0.0:50070
> > 2007-05-09 18:18:51,148 INFO org.mortbay.util.Container: Started
> > [EMAIL PROTECTED]
> > 2007-05-09 18:18:51,223 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 9000: starting
> > 2007-05-09 18:18:51,226 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 9000: starting
> > 2007-05-09 18:18:51,227 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 9000: starting
> > 2007-05-09 18:18:51,228 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 9000: starting
> > 2007-05-09 18:18:51,229 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 9000: starting
> > 2007-05-09 18:18:51,391 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 4 on 9000: starting
> > 2007-05-09 18:18:51,392 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 5 on 9000: starting
> > 2007-05-09 18:18:51,393 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 6 on 9000: starting
> > 2007-05-09 18:18:51,394 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 7 on 9000: starting
> > 2007-05-09 18:18:51,395 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 8 on 9000: starting
> > 2007-05-09 18:18:51,397 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 9 on 9000: starting
> >
> >
> > And these are from the datanode
> >
> > 2007-05-09 18:35:13,263 INFO org.apache.hadoop.ipc.Client: Retrying