Hi, all: I use Nutch-1.0 to do a crawling job. Now I found there are 14 tasks have very small number of pages, while the left 2 tasks have a large number of pages. I want to what reason cause this unbalanced task distribution in Fetch. Please see the messages ( http://node1:50030/jobtasks.jsp?jobid=job_201108161314_0207&type=map&pagenum=1&state=completed):
Hadoop map task list for job_201108161314_0207<http://node1:50030/jobdetails.jsp?jobid=job_201108161314_0207>on node1 <http://node1:50030/jobtracker.jsp> ------------------------------ Completed TasksTaskCompleteStatusStart TimeFinish TimeErrorsCounters task_201108161314_0207_m_000002<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000002> 100.00% 0 threads, 200 pages, 158 errors, 0.0 pages/s, 10 kb/s, 13-Sep-2011 08:53:56 13-Sep-2011 10:26:59 (1hrs, 33mins, 3sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000002> task_201108161314_0207_m_000003<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000003> 100.00% 0 threads, 12 pages, 58 errors, 0.0 pages/s, 5 kb/s, 13-Sep-2011 08:53:56 13-Sep-2011 09:00:23 (6mins, 27sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000003> task_201108161314_0207_m_000004<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000004> 100.00% 0 threads, 2 pages, 5 errors, 0.0 pages/s, 2 kb/s, 13-Sep-2011 08:53:57 13-Sep-2011 08:54:45 (48sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000004> task_201108161314_0207_m_000005<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000005> 100.00% 0 threads, 1 pages, 0 errors, 1.0 pages/s, 299 kb/s, 13-Sep-2011 08:53:57 13-Sep-2011 08:54:03 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000005> task_201108161314_0207_m_000006<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000006> 100.00% 0 threads, 1 pages, 0 errors, 1.0 pages/s, 149 kb/s, 13-Sep-2011 08:53:59 13-Sep-2011 08:54:05 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000006> task_201108161314_0207_m_000007<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000007> 100.00% 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, 13-Sep-2011 08:53:59 13-Sep-2011 08:54:02 (3sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000007> task_201108161314_0207_m_000008<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000008> 100.00% 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, 13-Sep-2011 08:54:02 13-Sep-2011 08:54:08 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000008> task_201108161314_0207_m_000009<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000009> 100.00% 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, 13-Sep-2011 08:54:03 13-Sep-2011 08:54:09 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000009> task_201108161314_0207_m_000010<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000010> 100.00% 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, 13-Sep-2011 08:54:05 13-Sep-2011 08:54:11 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000010> task_201108161314_0207_m_000011<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000011> 100.00% 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, 13-Sep-2011 08:54:08 13-Sep-2011 08:54:14 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000011> task_201108161314_0207_m_000012<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000012> 100.00% 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, 13-Sep-2011 08:54:09 13-Sep-2011 08:54:15 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000012> task_201108161314_0207_m_000013<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000013> 100.00% 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, 13-Sep-2011 08:54:11 13-Sep-2011 08:54:17 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000013> task_201108161314_0207_m_000014<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000014> 100.00% 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, 13-Sep-2011 08:54:14 13-Sep-2011 08:54:20 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000014> task_201108161314_0207_m_000015<http://node1:50030/taskdetails.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000015> 100.00% 0 threads, 0 pages, 0 errors, 0.0 pages/s, 0 kb/s, 13-Sep-2011 08:54:15 13-Sep-2011 08:54:21 (6sec) 9<http://node1:50030/taskstats.jsp?jobid=job_201108161314_0207&tipid=task_201108161314_0207_m_000015> ------------------------------ <http://node1:50030/jobtracker.jsp> TIA
