Re: Reduce hangs

Fri, 18 Jan 2008 17:57:21 -0800 

Looks like we still have this unsolved mysterious problem:

http://issues.apache.org/jira/browse/HADOOP-1374

Could it be related to HADOOP-1246? Arun?

Thanks,
--Konstantin

Yunhong Gu1 wrote:


Hi,
If someone knows how to fix the problem described below, please help me out. Thanks!
I am testing Hadoop on 2-node cluster and the "reduce" always hangs at some stage, even if I use different clusters. My OS is Debian Linux kernel 2.6 (AMD Opteron w/ 4GB Mem). Hadoop verision is 0.15.2. Java version is 1.5.0_01-b08.
I simply tried "./bin/hadoop jar hadoop-0.15.2-test.jar mrbench" and when the map stage finishes, the reduce stage will hang somewhere in the middle, sometimes at 0%. I also tried any other mapreduce program I can find in the example jar package but they all hang. 

The log file simply print
2008-01-18 15:15:50,831 INFO org.apache.hadoop.mapred.TaskTracker: task_200801181424_0004_r_000000_0 0.0% reduce > copy > 2008-01-18 15:15:56,841 INFO org.apache.hadoop.mapred.TaskTracker: task_200801181424_0004_r_000000_0 0.0% reduce > copy > 2008-01-18 15:16:02,850 INFO org.apache.hadoop.mapred.TaskTracker: task_200801181424_0004_r_000000_0 0.0% reduce > copy > 

forever.

The program does work if I start Hadoop only on single node.

Below is my hadoop-site.xml configuration:

<configuration>

<property>
   <name>fs.default.name</name>
   <value>10.0.0.1:60000</value>
</property>

<property>
   <name>mapred.job.tracker</name>
   <value>10.0.0.1:60001</value>
</property>

<property>
   <name>dfs.data.dir</name>
   <value>/raid/hadoop/data</value>
</property>

<property>
   <name>mapred.local.dir</name>
   <value>/raid/hadoop/mapred</value>
</property>

<property>
  <name>hadoop.tmp.dir</name>
  <value>/raid/hadoop/tmp</value>
</property>

<property>
  <name>mapred.child.java.opts</name>
  <value>-Xmx1024m</value>
</property>

<property>
  <name>mapred.tasktracker.tasks.maximum</name>
  <value>4</value>
</property>

<!--
<property>
  <name>mapred.map.tasks</name>
  <value>7</value>
</property>

<property>
  <name>mapred.reduce.tasks</name>
  <value>3</value>
</property>
-->

<property>
  <name>fs.inmemory.size.mb</name>
  <value>200</value>
</property>

<property>
  <name>dfs.block.size</name>
  <value>134217728</value>
</property>

<property>
  <name>io.sort.factor</name>
  <value>100</value>
</property>

<property>
  <name>io.sort.mb</name>
  <value>200</value>
</property>

<property>
  <name>io.file.buffer.size</name>
  <value>131072</value>
</property>

</configuration>

Reply via email to