Can you search the internet for : Error: java.lang.NullPointerException at java.util.concurrent.ConcurrentHashMap.get
There are several pages on this subject of which at least one has a solution. But i don't know if that only applies to an older Hadoop version. On Tuesday 16 August 2011 11:59:27 jeffersonzhou wrote: > Hi, > > > > I need help to figure out why reducer failed. I am using nutch 1.2 and the > hadoop shipped with nutch 1.2. I was using > http://wiki.apache.org/nutch/NutchHadoopTutorial to configure the two. > > > > Below is the information: > > > > > Hadoop Map/Reduce History Viewer > > _____ > > > Available History > > > Available Jobs > > > Job tracker Host Name > > Job tracker Start time > > Job Id > > Name > > User > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0001 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 0 > 1&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0001_nutch_inject%2Burls> > > inject urls > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0002 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 0 > 2&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0002_nutch_crawldb%2Bmit%252Fcrawldb> > > crawldb mit/crawldb > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0003 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 0 > 3&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0003_nutch_generate%253A%2Bselect%2Bfrom%2Bmit%252Fcrawldb> > > generate: select from mit/crawldb > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0004 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 0 > 4&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 > 161559_0004_nutch_generate%253A%2Bpartition%2Bmit%252Fsegments%252F2011081 > 61 60509> > > generate: partition mit/segments/20110816160509 > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0005 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 0 > 5&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0005_nutch_fetch%2Bmit%252Fsegments%252F20110816160509> > > fetch mit/segments/20110816160509 > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0006 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 0 > 6&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0006_nutch_crawldb%2Bmit%252Fcrawldb> > > crawldb mit/crawldb > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0007 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 0 > 7&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0007_nutch_linkdb%2Bmit%252Flinkdb> > > linkdb mit/linkdb > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0008 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 0 > 8&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0008_nutch_index-lucene%2Bmit%252Findexes> > > index-lucene mit/indexes > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0009 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 0 > 9&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0009_nutch_dedup%2B1%253A%2Burls%2Bby%2Btime> > > dedup 1: urls by time > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0010 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 1 > 0&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0010_nutch_dedup%2B2%253A%2Bcontent%2Bby%2Bhash> > > dedup 2: content by hash > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0011 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 1 > 1&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0011_nutch_dedup%2B3%253A%2Bdelete%2Bfrom%2Bindex%2528es%2529> > > dedup 3: delete from index(es) > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0012 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 1 > 2&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0012_nutch_inject%2Burls> > > inject urls > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0013 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 1 > 3&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0013_nutch_crawldb%2Bmit%252Fcrawldb> > > crawldb mit/crawldb > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0014 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 1 > 4&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0014_nutch_generate%253A%2Bselect%2Bfrom%2Bmit%252Fcrawldb> > > generate: select from mit/crawldb > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0015 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 1 > 5&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 > 161559_0015_nutch_generate%253A%2Bpartition%2Bmit%252Fsegments%252F2011081 > 61 62211> > > generate: partition mit/segments/20110816162211 > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0016 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 1 > 6&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0016_nutch_fetch%2Bmit%252Fsegments%252F20110816162211> > > fetch mit/segments/20110816162211 > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0017 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 1 > 7&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0017_nutch_crawldb%2Bmit%252Fcrawldb> > > crawldb mit/crawldb > > nutch > > > > localhost > > Tue Aug 16 15:59:38 GMT 2011 > > job_201108161559_0018 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161559_00 > 1 > 8&logFile=file:/nutch/search/logs/history/localhost_1313510378870_job_2011 > 08 161559_0018_nutch_dump%2Bmit%252Fcrawldb> > > dump mit/crawldb > > nutch > > > > master > > Tue Aug 16 16:43:27 GMT 2011 > > job_201108161643_0001 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161643_00 > 0 > 1&logFile=file:/nutch/search/logs/history/master_1313513007144_job_2011081 > 61 643_0001_nutch_inject%2Burls> > > inject urls > > nutch > > > > > > > > The error came from the last row: inject urls. The inject details are: > > > > > Hadoop Job job_201108161643_0001 on History Viewer > <http://192.168.1.116:50030/jobhistory.jsp> > > > User: nutch > JobName: inject urls > JobConf: > hdfs://master:9000/nutch/filesystem/mapreduce/system/job_201108161643_0001/ > j ob.xml > <http://192.168.1.116:50030/jobconf_history.jsp?jobid=job_201108161643_0001 > & > jobLogDir=file:/nutch/search/logs/history&jobUniqueString=master_131351300 > 71 44_job_201108161643_0001> > Submitted At: 16-Aug-2011 16:45:11 > Launched At: 16-Aug-2011 16:45:15 (4sec) > Finished At: 16-Aug-2011 16:46:22 (1mins, 7sec) > Status: FAILED > Analyse This Job > <http://192.168.1.116:50030/analysejobhistory.jsp?jobid=job_201108161643_00 > 0 > 1&logFile=file:/nutch/search/logs/history/master_1313513007144_job_2011081 > 61 643_0001_nutch_inject%2Burls> > > _____ > > > Kind > > Total Tasks(successful+failed+killed) > > Successful tasks > > Failed tasks > > Killed tasks > > Start Time > > Finish Time > > > Setup > > 1 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=SETUP&status=all> > > 1 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=SETUP&status=SUCCESS> > > 0 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=SETUP&status=FAILED> > > 0 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=SETUP&status=KILLED> > > 16-Aug-2011 16:45:39 > > 16-Aug-2011 16:45:41 (1sec) > > > Map > > 3 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=MAP&status=all> > > 3 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=MAP&status=SUCCESS> > > 0 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=MAP&status=FAILED> > > 0 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=MAP&status=KILLED> > > 16-Aug-2011 16:45:42 > > 16-Aug-2011 16:46:17 (34sec) > > > Reduce > > 8 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=REDUCE&status=all> > > 0 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=REDUCE&status=SUCCESS> > > 8 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=REDUCE&status=FAILED> > > 0 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=REDUCE&status=KILLED> > > 16-Aug-2011 16:45:58 > > 16-Aug-2011 16:46:36 (37sec) > > > Cleanup > > 1 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=CLEANUP&status=all> > > 1 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=CLEANUP&status=SUCCESS> > > 0 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=CLEANUP&status=FAILED> > > 0 > <http://192.168.1.116:50030/jobtaskshistory.jsp?jobid=job_201108161643_0001 > & > logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108161 > 64 3_0001_nutch_inject%2Burls&taskType=CLEANUP&status=KILLED> > > 16-Aug-2011 16:46:37 > > 16-Aug-2011 16:46:39 (1sec) > > > > > Failed tasks attempts by nodes > > > Hostname > > Failed Tasks > > > slave_1 > > task_201108161643_0001_r_000000 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> , > task_201108161643_0001_r_000001 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> , > > > slave_2 > > task_201108161643_0001_r_000000 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> , > task_201108161643_0001_r_000001 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> , > > > > > > The eight failed tasks are: > > > > > FAILED REDUCE task list for job_201108161643_0001 > <http://192.168.1.116:50030/jobdetailshistory.jsp?jobid=job_201108161643_00 > 0 > 1&&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls> > > > Task Id > > Start Time > > Finish Time > > Error > > > task_201108161643_0001_r_000000 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> > > 16/08 16:45:58 > > 16/08 16:46:06 (7sec) > > Error: java.lang.NullPointerException at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapC > o mpletionEvents(ReduceTask.java:2683) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Red > u ceTask.java:2605) > > > task_201108161643_0001_r_000000 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> > > 16/08 16:46:19 > > 16/08 16:46:24 (4sec) > > Error: java.lang.NullPointerException at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapC > o mpletionEvents(ReduceTask.java:2683) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Red > u ceTask.java:2605) > > > task_201108161643_0001_r_000000 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> > > 16/08 16:46:25 > > 16/08 16:46:30 (4sec) > > Error: java.lang.NullPointerException at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapC > o mpletionEvents(ReduceTask.java:2683) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Red > u ceTask.java:2605) > > > task_201108161643_0001_r_000000 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000000> > > 16/08 16:46:31 > > 16/08 16:46:36 (4sec) > > Error: java.lang.NullPointerException at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapC > o mpletionEvents(ReduceTask.java:2683) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Red > u ceTask.java:2605) > > > task_201108161643_0001_r_000001 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> > > 16/08 16:46:08 > > 16/08 16:46:16 (7sec) > > Error: java.lang.NullPointerException at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapC > o mpletionEvents(ReduceTask.java:2683) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Red > u ceTask.java:2605) > > > task_201108161643_0001_r_000001 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> > > 16/08 16:46:19 > > 16/08 16:46:24 (5sec) > > Error: java.lang.NullPointerException at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapC > o mpletionEvents(ReduceTask.java:2683) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Red > u ceTask.java:2605) > > > task_201108161643_0001_r_000001 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> > > 16/08 16:46:25 > > 16/08 16:46:30 (4sec) > > Error: java.lang.NullPointerException at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapC > o mpletionEvents(ReduceTask.java:2683) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Red > u ceTask.java:2605) > > > task_201108161643_0001_r_000001 > <http://192.168.1.116:50030/taskdetailshistory.jsp?jobid=job_201108161643_0 > 0 > 01&logFile=file:/nutch/search/logs/history/master_1313513007144_job_201108 > 16 1643_0001_nutch_inject%2Burls&taskid=task_201108161643_0001_r_000001> > > 16/08 16:46:31 > > 16/08 16:46:35 (4sec) > > Error: java.lang.NullPointerException at > java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:922) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapC > o mpletionEvents(ReduceTask.java:2683) at > org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(Red > u ceTask.java:2605) -- Markus Jelsma - CTO - Openindex http://www.linkedin.com/in/markus17 050-8536620 / 06-50258350
