I'm looking to deploy Hadoop at my University (University of
Washington) this upcoming school year, but I have a few questions as
to the development progress of the project.
~ Hadoop's DFS is supposed to be distributed, but I can't tell if it's
spreading to other machines besides the name node, or even to the
other drives on the name node. How do I monitor this?
~ What happens if a name-node goes down? Is there redundancy
currently in the system?
~ I'm trying to do a crawl with Nutch, but I keep getting timeout,
rpc, and other errors. They could be configuration based, but I doubt
it. Some help running through this would be very helpful. I've
duplicated the tutorials, but still have problems. The current
pressing error, which seems to be on each machine which is supposed to
be running the crawl is pasted from the log file below:
2006-08-04 00:01:49,739 WARN mapred.JobTracker - Starting tracker
java.net.SocketTimeoutException: timed out waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:312)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:150)
at org.apache.hadoop.dfs.$Proxy0.isDir(Unknown Source)
at org.apache.hadoop.dfs.DFSClient.isDirectory(DFSClient.java:325)
at
org.apache.hadoop.dfs.DistributedFileSystem.isDirectory(DistributedFileSystem.java:135)
at org.apache.hadoop.fs.FileSystem.delete(FileSystem.java:414)
at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:430)
at org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:65)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:1133)
2006-08-04 00:01:54,589 WARN fs.DFSClient - Problem renewing lease
for DFSClient_773448646: java.net.SocketTimeoutException: timed out
waiting for rpc response
at org.apache.hadoop.ipc.Client.call(Client.java:312)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:150)
at org.apache.hadoop.dfs.$Proxy0.renewLease(Unknown Source)
at org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:437)
at java.lang.Thread.run(Thread.java:595)
Obviously the DFSClient_773448646 number changes on each machine, but
my log files are just filled with these messages every minute or so.
Much Thanks,
Alden
