I search on the internet and most of threads pointing that the problem
might come from the configuration of /etc/hosts, /etc/hostname file.
I check my /etc/hosts which contains all machines, including master
and slaves, and /etc/hostname has correct name setup. Also, firewall
opens for ports that are required for hadoop system, such as 9000,
9001, and so on.
In addition to these setting, any other place I can check for
investigating such problem?
I appreciate any suggestion.
My testing environment setting is as below:
Machines used: 1 master, 3 slaves
master: debian
slaves: cloud1, cloud2, cloud3
And all machines share the same user name - cloud
OS debian testing, java 1.6.0_23, hadoop 0.20.2.
/etc/hosts on all machines
127.0.0.1 localhost
###.###.###.### debian.domain.com debian
###.###.###.### cloud1.domain.com cloud1
###.###.###.### cloud2.domain.com cloud2
###.###.###.### cloud3.domain.com cloud3
hadoop conf in debian (master)
core-site.xml
<property>
<name>fs.default.name</name>
<value>hdfs://debian:9000</value>
</property>
hdfs-site.xml
<property>
<name>dfs.name.dir</name>
<value>/home/cloud/dfs/name</value>
</property>
mapred-site.xml
<property>
<name>mapred.job.tracker</name>
<value>debian:9001</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/cloud/mapred/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/cloud/mapred/local</value>
</property>
hadoop conf in cloud1 (slave):
core-site.xml:
<property>
<name>fs.default.name</name>
<value>hdfs://debian:9000</value>
</property>
hdfs-site.xml:
<property>
<name>dfs.data.dir</name>
<value>/home/cloud/dfs/data</value>
</property>
mapred-site.xml
<property>
<name>mapred.job.tracker</name>
<value>debian:9001</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/cloud/mapred/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/cloud/mapred/local</value>
</property>
hadoop conf in cloud2 (slave):
core-site.xml:
<property>
<name>fs.default.name</name>
<value>hdfs://debian:9000</value>
</property>
hdfs-site.xml:
<property>
<name>dfs.data.dir</name>
<value>/home/cloud/dfs/data</value>
</property>
mapred-site.xml:
<property>
<name>mapred.job.tracker</name>
<value>debian:9001</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/cloud/mapred/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/cloud/mapred/local</value>
</property>
hadoop conf in cloud3 (slave):
core-site.xml:
<property>
<name>fs.default.name</name>
<value>hdfs://debian:9000</value>
</property>
hdfs-site.xml:
<property>
<name>dfs.data.dir</name>
<value>/home/cloud/dfs/data</value>
</property>
mapred-site:xml
<property>
<name>mapred.job.tracker</name>
<value>debian:9001</value>
</property>
<property>
<name>mapred.map.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.reduce.tasks</name>
<value>3</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/cloud/mapred/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/cloud/mapred/local</value>
</property>
