Steve Loughran 写道:
David Wei wrote:
Steve Loughran ??:
David Wei wrote:
Error initializing attempt_200810220716_0004_m_000001_0:
java.lang.IllegalArgumentException: Wrong FS:
hdfs://192.168.52.129:9000/tmp/hadoop-root/mapred/system/job_200810220716_0004/job.xml,
expected: hdfs://datacenter5:9000 at
org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:320)
And if I configure the masters/slaves with hostname, datanode will
not be able to connect to the master!
No one knows how to solve this?
that implies DNS problems. what does
nslookup datacenter5
return on the command line of all the slaves?
Steve,
the output is:
Server: 192.168.18.10
Address: 192.168.18.10#53
Non-authoritative answer:
Name: datacenter5.papa
Address: 61.140.3.66
And none of them is the address of datacenter5(192.168.52.129). How can
I solve this problem?
well, your datacenter5 hostname may be the public IP address; the
192.168. addresses are the addresses within the cluster. Because they
are different, and because the namenode thinks it is called datacenter5,
things are breaking
Some options
-bring up your own DNS server and add the entries for the hostnames
-edit /etc/hosts in the different servers (this becomes expensive after
a while, and if one machine gets inconsistent, you will have a hard time
tracking down the problem)
-somehow get the namenode to come up with a filesystem called
hdfs://192.168.52.129:9000/. That should be something you can do in the
site fs.default.name option.
I had ready try the last 2 options and found that:
In 2nd option, the SAME problem. I edit all the nodes's hosts file and
append the ip/hostname mapping inside it, so that all the nodes can
access each other with hostname(can ping and ssh to each other)
In 3rd option, I remove all the setting inside hosts file and find that
running reduce is almost impossible. We always got this kind of error:
java.lang.NullPointerException
at
org.apache.hadoop.mapred.ReduceTask$ReduceCopier.fetchOutputs(ReduceTask.java:1777)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:254)
at
org.apache.hadoop.mapred.TaskTracker$Child.main(TaskTracker.java:2207)
-------
Task attempt_200810220833_0001_r_000000_0 failed to report status for 601
seconds. Killing!
I had googled for set up of hadoop and heard that if setting totally based on IP, doing Reduce will meet this kind of problem.
Is this a bug of hadoop?
BTW, datacenter5 is the only master in my cluster
thx