Hi,
I need help to figure out why reducer failed. I am using nutch 1.2 and the hadoop shipped with nutch 1.2. I was using http://wiki.apache.org/nutch/NutchHadoopTutorial to configure the two. Below is the information: Hadoop Map/Reduce History Viewer _____ Available History Available Jobs Job tracker Host Name Job tracker Start time Job Id Name User localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0001 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_000 1&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0001_nutch_inject%2Burls> inject urls nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0002 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_000 2&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0002_nutch_crawldb%2Bmit%252Fcrawldb> crawldb mit/crawldb nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0003 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_000 3&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0003_nutch_generate%253A%2Bselect%2Bfrom%2Bmit%252Fcrawldb> generate: select from mit/crawldb nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0004 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_000 4&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0004_nutch_generate%253A%2Bpartition%2Bmit%252Fsegments%252F201108161 60509> generate: partition mit/segments/20110816160509 nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0005 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_000 5&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0005_nutch_fetch%2Bmit%252Fsegments%252F20110816160509> fetch mit/segments/20110816160509 nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0006 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_000 6&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0006_nutch_crawldb%2Bmit%252Fcrawldb> crawldb mit/crawldb nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0007 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_000 7&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0007_nutch_linkdb%2Bmit%252Flinkdb> linkdb mit/linkdb nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0008 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_000 8&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0008_nutch_index-lucene%2Bmit%252Findexes> index-lucene mit/indexes nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0009 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_000 9&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0009_nutch_dedup%2B1%253A%2Burls%2Bby%2Btime> dedup 1: urls by time nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0010 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_001 0&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0010_nutch_dedup%2B2%253A%2Bcontent%2Bby%2Bhash> dedup 2: content by hash nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0011 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_001 1&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0011_nutch_dedup%2B3%253A%2Bdelete%2Bfrom%2Bindex%2528es%2529> dedup 3: delete from index(es) nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0012 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_001 2&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0012_nutch_inject%2Burls> inject urls nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0013 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_001 3&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0013_nutch_crawldb%2Bmit%252Fcrawldb> crawldb mit/crawldb nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0014 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_001 4&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0014_nutch_generate%253A%2Bselect%2Bfrom%2Bmit%252Fcrawldb> generate: select from mit/crawldb nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0015 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_001 5&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0015_nutch_generate%253A%2Bpartition%2Bmit%252Fsegments%252F201108161 62211> generate: partition mit/segments/20110816162211 nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0016 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_001 6&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0016_nutch_fetch%2Bmit%252Fsegments%252F20110816162211> fetch mit/segments/20110816162211 nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0017 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_001 7&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0017_nutch_crawldb%2Bmit%252Fcrawldb> crawldb mit/crawldb nutch localhost Tue Aug 16 15:59:38 GMT 2011 job_201108161559_0018 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_001 8&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_201108 161559_0018_nutch_dump%2Bmit%252Fcrawldb> dump mit/crawldb nutch master Tue Aug 16 16:43:27 GMT 2011 job_201108161643_0001 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161643_000 1&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 643_0001_nutch_inject%2Burls> inject urls nutch The error came from the last row: inject urls. The inject details are: Hadoop Job job_201108161643_0001 on History Viewer <http://192.168.1.116:50030/jobhistory.jsp> User: nutch JobName: inject urls JobConf: hdfs://master:9000/nutch/filesystem/mapreduce/system/job_201108161643_0001/j ob.xml <http://192.168.1.116:50030/jobconf_history.jsp?jobid=job_201108161643_0001&; jobLogDir=file:/nutch/search/logs/history&jobUniqueString=master_13135130071 44_job_201108161643_0001> Submitted At: 16-Aug-2011 16:45:11 Launched At: 16-Aug-2011 16:45:15 (4sec) Finished At: 16-Aug-2011 16:46:22 (1mins, 7sec) Status: FAILED Analyse This Job <http://192.168.1.116:50030/analysejobhistory.jsp?jobid=job_201108161643_000 1&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 643_0001_nutch_inject%2Burls> _____ Kind Total Tasks(successful+failed+killed) Successful tasks Failed tasks Killed tasks Start Time Finish Time Setup 1 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=SETUP&status=all> 1 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=SETUP&status=SUCCESS> 0 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=SETUP&status=FAILED> 0 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=SETUP&status=KILLED> 16-Aug-2011 16:45:39 16-Aug-2011 16:45:41 (1sec) Map 3 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=MAP&status=all> 3 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=MAP&status=SUCCESS> 0 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=MAP&status=FAILED> 0 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=MAP&status=KILLED> 16-Aug-2011 16:45:42 16-Aug-2011 16:46:17 (34sec) Reduce 8 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=REDUCE&status=all> 0 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=REDUCE&status=SUCCESS> 8 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=REDUCE&status=FAILED> 0 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=REDUCE&status=KILLED> 16-Aug-2011 16:45:58 16-Aug-2011 16:46:36 (37sec) Cleanup 1 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=CLEANUP&status=all> 1 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=CLEANUP&status=SUCCESS> 0 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=CLEANUP&status=FAILED> 0 <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001&; logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816164 3_0001_nutch_inject%2Burls&taskType=CLEANUP&status=KILLED> 16-Aug-2011 16:46:37 16-Aug-2011 16:46:39 (1sec) Failed tasks attempts by nodes Hostname Failed Tasks slave_1 task_201108161643_0001_r_000000 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> , task_201108161643_0001_r_000001 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> , slave_2 task_201108161643_0001_r_000000 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> , task_201108161643_0001_r_000001 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> , The eight failed tasks are: FAILED REDUCE task list for job_201108161643_0001 <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161643_000 1&&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls> Task Id Start Time Finish Time Error task_201108161643_0001_r_000000 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> 16/08 16:45:58 16/08 16:46:06 (7sec) Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCo mpletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Redu ceTask.java:2605) task_201108161643_0001_r_000000 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> 16/08 16:46:19 16/08 16:46:24 (4sec) Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCo mpletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Redu ceTask.java:2605) task_201108161643_0001_r_000000 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> 16/08 16:46:25 16/08 16:46:30 (4sec) Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCo mpletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Redu ceTask.java:2605) task_201108161643_0001_r_000000 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> 16/08 16:46:31 16/08 16:46:36 (4sec) Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCo mpletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Redu ceTask.java:2605) task_201108161643_0001_r_000001 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> 16/08 16:46:08 16/08 16:46:16 (7sec) Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCo mpletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Redu ceTask.java:2605) task_201108161643_0001_r_000001 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> 16/08 16:46:19 16/08 16:46:24 (5sec) Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCo mpletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Redu ceTask.java:2605) task_201108161643_0001_r_000001 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> 16/08 16:46:25 16/08 16:46:30 (4sec) Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCo mpletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Redu ceTask.java:2605) task_201108161643_0001_r_000001 <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_00 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_20110816 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> 16/08 16:46:31 16/08 16:46:35 (4sec) Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCo mpletionEvents(ReduceTask.java:2683) at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Redu ceTask.java:2605)
