Hi,
not sure if this is the same source of problem, but I also run in
problems with a hanging reduce.
It is reproducible for me, though I did not find the source of the
problem yet.
I run a series of jobs and my last job, the last reduce task hangs for
about 15 to 20 minutes doing nothing, but than resumes. I running
hadoop 15.1
Below the log entries during the hang. So I think it is not the copy
problem mentioned before. I also checked our dfs is healthy.
2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1 Need 2 map output(s)
2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1: Got 0 new map-outputs & 0 obsolete
map-outputs from tasktracker and 0 map-outputs from previous failures
2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1 Got 2 known map output location(s);
scheduling...
2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1 Scheduled 2 of 2 known outputs (0
slow hosts and 0 dup hosts)
2008-01-22 21:22:09,327 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1 Copying
task_200801221313_0003_m_000035_0 output from hadoop5.dev.company.com.
2008-01-22 21:22:09,328 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1 Copying
task_200801221313_0003_m_000040_0 output from hadoop1.dev.company.com.
2008-01-22 21:22:11,243 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1 done copying
task_200801221313_0003_m_000040_0 output from hadoop1.dev.company.com.
2008-01-22 21:22:11,610 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1 done copying
task_200801221313_0003_m_000035_0 output from hadoop5.dev.company.com.
2008-01-22 21:22:11,611 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1 Copying of all map outputs complete.
Initiating the last merge on the remaining files in ramfs://
mapoutput169937755
2008-01-22 21:22:11,635 INFO org.apache.hadoop.mapred.ReduceTask:
task_200801221313_0003_r_000046_1 Merge of the 1 files in
InMemoryFileSystem complete. Local file is /home/hadoop/data/hadoop-
hadoop/mapred/local/task_200801221313_0003_r_000046_1/map_34.out
Any ideas? Thanks!
Stefan
