hello guys: I'm doing some experiences on my 3 node virtual machine cluster, one for namenode and jobtracker while the other tow for datanode and tasktracker.with a 0.21.0 hadoop and when I 'm running a job ,I got such message
10/08/27 17:28:58 INFO mapreduce.Job: map 0% reduce 0% 10/08/27 17:29:10 INFO mapreduce.Job: map 50% reduce 0% 10/08/27 17:29:12 INFO mapreduce.Job: map 83% reduce 0% 10/08/27 17:29:19 INFO mapreduce.Job: map 83% reduce 16% 10/08/27 17:29:24 INFO mapreduce.Job: map 100% reduce 16% the reduce runs while the map task hasn't finished!! ( I read from the books that reduce task runs exactlly after the maps finish!!) is there something wrong with the cluster or my knowledge? by the way ,the reduce job stuck while it is in the progress 16.63%, I found that people on the internet also got this problem but I haven't found the solution. however, after some time , after the system report the error message , the job began to run again! like this 10/08/27 17:29:58 INFO mapreduce.Job: map 100% reduce 16% 10/08/27 17:30:01 INFO mapreduce.Job: map 50% reduce 16% 10/08/27 17:30:07 INFO mapreduce.Job: map 83% reduce 16% 10/08/27 17:30:19 INFO mapreduce.Job: map 100% reduce 16% 10/08/27 17:30:25 INFO mapreduce.Job: map 100% reduce 66% 10/08/27 17:30:31 INFO mapreduce.Job: map 100% reduce 100% and it runs pretty well! does someone know about this? belows are the message on the console, If the logs are needed ,let me know.thanks any suggestions and references are appreciated best regards xu 10/08/27 17:28:56 INFO security.Groups: Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping; cacheTimeout=300000 10/08/27 17:28:56 WARN conf.Configuration: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id 10/08/27 17:28:56 WARN mapreduce.JobSubmitter: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 10/08/27 17:28:56 INFO input.FileInputFormat: Total input paths to process : 1 10/08/27 17:28:56 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 10/08/27 17:28:56 INFO mapreduce.JobSubmitter: number of splits:2 10/08/27 17:28:57 INFO mapreduce.JobSubmitter: adding the following namenodes' delegation tokens:null 10/08/27 17:28:57 INFO mapreduce.Job: Running job: job_201008271725_0001 10/08/27 17:28:58 INFO mapreduce.Job: map 0% reduce 0% 10/08/27 17:29:10 INFO mapreduce.Job: map 50% reduce 0% 10/08/27 17:29:12 INFO mapreduce.Job: map 83% reduce 0% 10/08/27 17:29:19 INFO mapreduce.Job: map 83% reduce 16% 10/08/27 17:29:24 INFO mapreduce.Job: map 100% reduce 16% 10/08/27 17:29:48 INFO mapreduce.Job: Task Id : attempt_201008271725_0001_r_000000_0, Status : FAILED org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error in shuffle in fetcher#1 at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:124) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362) at org.apache.hadoop.mapred.Child$4.run(Child.java:217) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742) at org.apache.hadoop.mapred.Child.main(Child.java:211) Caused by: java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out. at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.checkReducerHealth(ShuffleScheduler.java:253) at org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.copyFailed(ShuffleScheduler.java:187) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:234) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:149) 10/08/27 17:29:48 WARN mapreduce.Job: Error reading task outputConnection refused 10/08/27 17:29:48 WARN mapreduce.Job: Error reading task outputConnection refused 10/08/27 17:29:49 INFO mapreduce.Job: map 100% reduce 0% 10/08/27 17:29:57 INFO mapreduce.Job: Task Id : attempt_201008271725_0001_m_000000_0, Status : FAILED Too many fetch-failures 10/08/27 17:29:57 WARN mapreduce.Job: Error reading task outputConnection refused 10/08/27 17:29:57 WARN mapreduce.Job: Error reading task outputConnection refused 10/08/27 17:29:58 INFO mapreduce.Job: map 100% reduce 16% 10/08/27 17:30:01 INFO mapreduce.Job: map 50% reduce 16% 10/08/27 17:30:07 INFO mapreduce.Job: map 83% reduce 16% 10/08/27 17:30:19 INFO mapreduce.Job: map 100% reduce 16% 10/08/27 17:30:25 INFO mapreduce.Job: map 100% reduce 66% 10/08/27 17:30:31 INFO mapreduce.Job: map 100% reduce 100% 10/08/27 17:30:33 INFO mapreduce.Job: Job complete: job_201008271725_0001 10/08/27 17:30:33 INFO mapreduce.Job: Counters: 33 FileInputFormatCounters BYTES_READ=76420532 FileSystemCounters FILE_BYTES_READ=155979268 FILE_BYTES_WRITTEN=239598906 HDFS_BYTES_READ=76424828 HDFS_BYTES_WRITTEN=78386951 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=2 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 Job Counters Data-local map tasks=3 Total time spent by all maps waiting after reserving slots (ms)=0 Total time spent by all reduces waiting after reserving slots (ms)=0 SLOTS_MILLIS_MAPS=47747 SLOTS_MILLIS_REDUCES=77241 Launched map tasks=3 Launched reduce tasks=2 Map-Reduce Framework Combine input records=999998 Combine output records=994179 Failed Shuffles=1 GC time elapsed (ms)=667 Map input records=499999 Map output bytes=80759850 Map output records=999998 Merged Map outputs=2 Reduce input groups=993814 Reduce input records=994179 Reduce output records=993814 Reduce shuffle bytes=83049802 Shuffled Maps =2 Spilled Records=2861309 SPLIT_RAW_BYTES=200
