Just to close the loop on this, and to make sure someone else doesn't have the
same problem, this turned out to be a case of cockpit error.
I had mis-read the documentation concerning
mapred.task.tracker.report.bindAddress and had set it to point to the master
node. I should have left this set to 127.0.0.1 and instead tweaked
mapred.job.tracker and pointed that to the master node. When I realized the
error of my ways I changed it and everything came up on the first try.
I could nit about the documentation needing to be a bit more clear, but I'm
writing this off as mostly a case of error due to sleep deprivation.
Thanks,
C G
Arun C Murthy <[EMAIL PROTECTED]> wrote:
> 2007-12-17 14:03:59,756 ERROR org.apache.hadoop.mapred.TaskTracker:
Can not start task tracker because java.net.BindException: Problem
binding to master/10.2.13.1:0
> at org.apache.hadoop.ipc.Server.bind(Server.java:163)
> at org.apache.hadoop.ipc.Server$Listener.(Server.java:215)
> at org.apache.hadoop.ipc.Server.(Server.java:657)
> at org.apache.hadoop.ipc.RPC$Server.(RPC.java:363)
> at org.apache.hadoop.ipc.RPC.getServer(RPC.java:333)
> at
org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:409)
> at
org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:717)
> at
org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1880)
That tells me you need to check who is running on that port...
Arun
C G wrote:
> Sorry, forgot to paste these in.....
>
>
> 2007-12-17 14:03:59,371 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = master/10.2.13.1
> STARTUP_MSG: args = []
> ************************************************************/
> 2007-12-17 14:03:59,481 INFO org.mortbay.util.Credential: Checking Resource
> aliases
> 2007-12-17 14:03:59,531 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
> 2007-12-17 14:03:59,531 INFO org.mortbay.util.Container: Started
> HttpContext[/static,/static]
> 2007-12-17 14:03:59,531 INFO org.mortbay.util.Container: Started
> HttpContext[/logs,/logs]
> 2007-12-17 14:03:59,712 INFO org.mortbay.util.Container: Started [EMAIL
> PROTECTED]
> 2007-12-17 14:03:59,739 INFO org.mortbay.util.Container: Started
> WebApplicationContext[/,/]
> 2007-12-17 14:03:59,742 INFO org.mortbay.http.SocketListener: Started
> SocketListener on 0.0.0.0:50060
> 2007-12-17 14:03:59,742 INFO org.mortbay.util.Container: Started [EMAIL
> PROTECTED]
> 2007-12-17 14:03:59,747 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2007-12-17 14:03:59,756 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 43913: starting
> 2007-12-17 14:03:59,756 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 0 on 43913: starting
> 2007-12-17 14:03:59,757 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 43913: starting
> 2007-12-17 14:03:59,757 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 2 on 43913: starting
> 2007-12-17 14:03:59,757 INFO org.apache.hadoop.mapred.TaskTracker:
> TaskTracker up at: /10.2.13.1:43913
> 2007-12-17 14:03:59,757 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> tracker master:/10.2.13.1:43913
> 2007-12-17 14:03:59,759 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 3 on 43913: starting
> 2007-12-17 14:03:59,784 INFO org.apache.hadoop.mapred.TaskTracker: Starting
> thread: Map-events fetcher for all reduce tasks on master:/10.2.13.1:43913
>
> 2007-12-17 14:03:59,370 INFO org.apache.hadoop.mapred.TaskTracker:
> STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting TaskTracker
> STARTUP_MSG: host = worker1/10.2.14.1
> STARTUP_MSG: args = []
> ************************************************************/
> 2007-12-17 14:03:59,481 INFO org.mortbay.util.Credential: Checking Resource
> aliases
> 2007-12-17 14:03:59,529 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4
> 2007-12-17 14:03:59,530 INFO org.mortbay.util.Container: Started
> HttpContext[/static,/static]
> 2007-12-17 14:03:59,530 INFO org.mortbay.util.Container: Started
> HttpContext[/logs,/logs]
> 2007-12-17 14:03:59,714 INFO org.mortbay.util.Container: Started [EMAIL
> PROTECTED]
> 2007-12-17 14:03:59,741 INFO org.mortbay.util.Container: Started
> WebApplicationContext[/,/]
> 2007-12-17 14:03:59,744 INFO org.mortbay.http.SocketListener: Started
> SocketListener on 0.0.0.0:50060
> 2007-12-17 14:03:59,744 INFO org.mortbay.util.Container: Started [EMAIL
> PROTECTED]
> 2007-12-17 14:03:59,749 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> Initializing JVM Metrics with processName=TaskTracker, sessionId=
> 2007-12-17 14:03:59,756 INFO org.apache.hadoop.mapred.TaskTracker:
> SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down TaskTracker at worker1/10.2.14.1
> ************************************************************/
>
>
> C G
wrote:
> Hi All:
>
> I am migrating from a small grid to a larger one. The small grid runs fine
> with no issues. On the larger grid, with nearly identical configuration files
> (just changing host names and file paths), I can get dfs to run, but not all
> the TaskTrackers. Specifically, the task trackers on the slave nodes fail to
> initialize, failing on a bind error to the master node using port 0.
>
> The task tracker logs are below for the master (which starts up successfully)
> and from one of the slaves (which fails to start). Any thoughts/comments
> would be most appreciated.
>
> Note that if I log in to the worker node, and using Python do a
> socket.connect() to the master on the master's port (43913 for this run) I
> can connect successfully.
>
> How do the slave nodes know what port to use when connecting?
>
> Any help appreciate as I am tearing out what little is left of my hair :-).
>
> Thanks,
> C G
>
>
>
> ---------------------------------
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
>
>
> ---------------------------------
> Looking for last minute shopping deals? Find them fast with Yahoo! Search.
---------------------------------
Looking for last minute shopping deals? Find them fast with Yahoo! Search.
