Hi Brady, Could you please check the TaskTracker logs (TTs that ran the maps that are failing copy) and see what error you are seeing there?
Jothi On 4/24/09 5:15 AM, "Brady Chang" <bch...@atipa.com> wrote: > Hello all, > > One of my customer's getting the following when increase input size. > > same process runs fine with smaller input size. > > /tmp has plenty of space, > > each data node has 64 Gig memory. > > hadoop 0-19.1 > > Please let me know if you need any additional info. > > > > [r...@head attempt_200904181818_0020_r_000002_1]# cat syslog > > 2009-04-20 00:52:52,898 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: > Initializing JVM Metrics with processName=SHUFFLE, sessionId= > > 2009-04-20 00:52:53,005 INFO org.apache.hadoop.mapred.ReduceTask: > ShuffleRamManager: MemoryLimit=130514944, MaxSingleShuffleLimit=32628736 > > 2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging > on-disk files > > 2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 Thread started: Thread for merging in > memory files > > 2009-04-20 00:52:53,013 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 Thread waiting: Thread for merging > on-disk files > > 2009-04-20 00:52:53,015 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 Need another 168 map output(s) where 0 > is already in progress > > 2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1: Got 168 new map-outputs > > 2009-04-20 00:52:53,042 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0 > dup hosts) > > 2009-04-20 00:52:53,068 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000002_0 > > 2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000006_0 > > 2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000003_0 > > 2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000016_0 > > 2009-04-20 00:52:53,069 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000012_0 > > 2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000016_0 > > 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000006_0 > > 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000012_0 > > 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000003_0 > > 2009-04-20 00:52:53,073 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000002_0 > > 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000012_0 -> (-1, -1) from n2 > > 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000016_0 -> (-1, -1) from n8 > > 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000006_0 -> (-1, -1) from n4 > > 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000002_0 -> (-1, -1) from n5 > > 2009-04-20 00:52:53,074 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000003_0 -> (-1, -1) from n7 > > 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000007_0 > > 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000007_0 > > 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000007_0 -> (-1, -1) from n3 > > 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000037_0 > > 2009-04-20 00:52:53,078 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000037_0 > > 2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000037_0 -> (-1, -1) from n1 > > 2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000033_0 > > 2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000033_0 > > 2009-04-20 00:52:53,079 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000033_0 -> (-1, -1) from n6 > > 2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask: Ignoring > obsolete output of KILLED map-task: 'attempt_200904181818_0020_m_000166_1' > > 2009-04-20 00:52:55,021 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1: Got 1 obsolete map-outputs from > tasktracker > > 2009-04-20 00:52:55,022 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0 > dup hosts) > > 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000005_0 > > 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000005_0 > > 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000005_0 -> (-1, -1) from n1 > > 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000014_0 > > 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000004_0 > > 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000014_0 > > 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000004_0 > > 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000014_0 -> (-1, -1) from n4 > > 2009-04-20 00:52:55,024 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000004_0 -> (-1, -1) from n2 > > 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000019_0 > > 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000018_0 > > 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000019_0 > > 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000018_0 > > 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000019_0 -> (-1, -1) from n7 > > 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000018_0 -> (-1, -1) from n5 > > 2009-04-20 00:52:55,025 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000008_0 > > 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000008_0 > > 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000008_0 -> (-1, -1) from n8 > > 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000015_0 > > 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000015_0 > > 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000015_0 -> (-1, -1) from n3 > > 2009-04-20 00:52:55,026 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000001_0 > > 2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000001_0 > > 2009-04-20 00:52:55,027 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000001_0 -> (-1, -1) from n6 > > 2009-04-20 00:52:57,026 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0 > dup hosts) > > 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000029_0 > > 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000029_0 > > 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000029_0 -> (-1, -1) from n1 > > 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000011_0 > > 2009-04-20 00:52:57,028 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000020_0 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000026_0 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000011_0 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000020_0 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000011_0 -> (-1, -1) from n7 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000020_0 -> (-1, -1) from n2 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000026_0 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000047_0 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000026_0 -> (-1, -1) from n5 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000047_0 > > 2009-04-20 00:52:57,029 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000047_0 -> (-1, -1) from n3 > > 2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000022_0 > > 2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000022_0 > > 2009-04-20 00:52:57,030 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000022_0 -> (-1, -1) from n4 > > 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000049_0 > > 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000049_0 > > 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000049_0 -> (-1, -1) from n6 > > 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000000_0 > > 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000000_0 > > 2009-04-20 00:52:57,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000000_0 -> (-1, -1) from n8 > > 2009-04-20 00:52:59,029 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0 > dup hosts) > > 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000052_0 > > 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000052_0 > > 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000074_0 > > 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000052_0 -> (-1, -1) from n2 > > 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000074_0 > > 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000074_0 -> (-1, -1) from n5 > > 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000017_0 > > 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000017_0 > > 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000017_0 -> (-1, -1) from n6 > > 2009-04-20 00:52:59,031 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000038_0 > > 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000056_0 > > 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000038_0 > > 2009-04-20 00:52:59,032 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000056_0 > > 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000038_0 -> (-1, -1) from n4 > > 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000056_0 -> (-1, -1) from n8 > > 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000023_0 > > 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000023_0 > > 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000023_0 -> (-1, -1) from n3 > > 2009-04-20 00:52:59,033 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000013_0 > > 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000013_0 > > 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000013_0 -> (-1, -1) from n1 > > 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000027_0 > > 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000027_0 > > 2009-04-20 00:52:59,034 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000027_0 -> (-1, -1) from n7 > > 2009-04-20 00:53:01,033 INFO org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 Scheduled 8 outputs (0 slow hosts and0 > dup hosts) > > 2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000055_0 > > 2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000055_0 > > 2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000055_0 -> (-1, -1) from n3 > > 2009-04-20 00:53:01,035 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000025_0 > > 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000025_0 > > 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000025_0 -> (-1, -1) from n6 > > 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000035_0 > > 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000035_0 > > 2009-04-20 00:53:01,036 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000035_0 -> (-1, -1) from n7 > > 2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000021_0 > > 2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000021_0 > > 2009-04-20 00:53:01,037 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000021_0 -> (-1, -1) from n1 > > 2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Shuffling > 2 bytes (6 raw bytes) into RAM from attempt_200904181818_0020_m_000034_0 > > 2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Read 2 > bytes from map-output for attempt_200904181818_0020_m_000034_0 > > 2009-04-20 00:53:01,038 INFO org.apache.hadoop.mapred.ReduceTask: Rec #1 > from attempt_200904181818_0020_m_000034_0 -> (-1, -1) from n5 > > 2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 copy failed: > attempt_200904181818_0020_m_000044_0 from n2 > > 2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 copy failed: > attempt_200904181818_0020_m_000054_0 from n4 > > 2009-04-20 00:53:01,041 WARN org.apache.hadoop.mapred.ReduceTask: > attempt_200904181818_0020_r_000002_1 copy failed: > attempt_200904181818_0020_m_000032_0 from n8 > > 2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask: > java.io.FileNotFoundException: > http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818 > _0020_m_000044_0&reduce=2 > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces > sorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc > torAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at > sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136 > 0) > > at java.security.AccessController.doPrivileged(Native Method) > > at > sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne > ction.java:1354) > > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection > .java:1008) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr > eam(ReduceTask.java:1293) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu > t(ReduceTask.java:1231) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput( > ReduceTask.java:1144) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT > ask.java:1084) > > Caused by: java.io.FileNotFoundException: > http://n2:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818 > _0020_m_000044_0&reduce=2 > > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection > .java:1303) > > ... 4 more > > > > 2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask: > java.io.FileNotFoundException: > http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818 > _0020_m_000032_0&reduce=2 > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces > sorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc > torAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at > sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136 > 0) > > at java.security.AccessController.doPrivileged(Native Method) > > at > sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne > ction.java:1354) > > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection > .java:1008) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr > eam(ReduceTask.java:1293) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu > t(ReduceTask.java:1231) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput( > ReduceTask.java:1144) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT > ask.java:1084) > > Caused by: java.io.FileNotFoundException: > http://n8:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818 > _0020_m_000032_0&reduce=2 > > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection > .java:1303) > > ... 4 more > > > > 2009-04-20 00:53:01,042 WARN org.apache.hadoop.mapred.ReduceTask: > java.io.FileNotFoundException: > http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818 > _0020_m_000054_0&reduce=2 > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAcces > sorImpl.java:39) > > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc > torAccessorImpl.java:27) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:513) > > at > sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:136 > 0) > > at java.security.AccessController.doPrivileged(Native Method) > > at > sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConne > ction.java:1354) > > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection > .java:1008) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getInputStr > eam(ReduceTask.java:1293) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.getMapOutpu > t(ReduceTask.java:1231) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.copyOutput( > ReduceTask.java:1144) > > at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceT > ask.java:1084) > > Caused by: java.io.FileNotFoundException: > http://n4:50060/mapOutput?job=job_200904181818_0020&map=attempt_200904181818 > _0020_m_000054_0&reduce=2 > > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection > .java:1303) > > ... 4 more >
