Hi! I'm running streaming tasks on hadoop 0.17.0, and wondered, if anyone has an approach to debugging the following situation:
-) map have all finished (100% in http display), -) some reducers are hanging, with the messages below. Notice, that the task had 100 map tasks at allo, so 58 seems like an extraordinary high number of missing parts, long after map has officially finished. Plus it seems to be deterministic, it always stop at 3 reduce parts not finishing, although I haven't yet checked if they are always the same errors or not. > 2008-06-30 15:25:41,953 INFO org.apache.hadoop.mapred.ReduceTask: > task_200806300847_0002_r_000014_0 Need 58 map output(s) 2008-06-30 > 15:25:41,953 INFO org.apache.hadoop.mapred.ReduceTask: > task_200806300847_0002_r_000014_0: Got 0 new map-outputs & 0 obsolete > map-outputs from tasktracker and 0 map-outputs from previous failures > 2008-06-30 15:25:41,954 INFO org.apache.hadoop.mapred.ReduceTask: > task_200806300847_0002_r_000014_0 Got 0 known map output location(s); > scheduling... 2008-06-30 15:25:41,954 INFO > org.apache.hadoop.mapred.ReduceTask: task_200806300847_0002_r_000014_0 > Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts) 2008-06-30 > 15:25:46,770 INFO org.apache.hadoop.streaming.PipeMapRed: MRErrorThread > done 2008-06-30 15:25:46,963 INFO org.apache.hadoop.mapred.ReduceTask: > task_200806300847_0002_r_000014_0 Need 58 map output(s) 2008-06-30 > 15:25:46,963 INFO org.apache.hadoop.mapred.ReduceTask: > task_200806300847_0002_r_000014_0: Got 0 new map-outputs & 0 obsolete > map-outputs from tasktracker and 0 map-outputs from previous failures > 2008-06-30 15:25:46,964 INFO org.apache.hadoop.mapred.ReduceTask: > task_200806300847_0002_r_000014_0 Got 0 known map output location(s); > scheduling... 2008-06-30 15:25:46,964 INFO > org.apache.hadoop.mapred.ReduceTask: task_200806300847_0002_r_000014_0 > Scheduled 0 of 0 known outputs (0 slow hosts and 0 dup hosts) TIA, Andreas
signature.asc
Description: This is a digitally signed message part.
