Hi Michael, this question should be asked in the nutch-users list.
Take a look at a thread: So many Unfetched Pages using MapReduce G. On Tue, 2006-01-31 at 15:52 +0100, Michael Nebel wrote: > Hi, > > the last days I gave the mapred-branch a try and I was impressed! > > But I still have a problem with the incremental crawling. My setup: I > have 4 boxes (1x namenode/jobtracker - 3x datanode/tasktracker). Running > one round of "crawling" consists out of the steps: > > - generate (I set a limit of "-topN 10000000") > - fetch > - update > - index > - invertlinks > > For the first round, I injected a list of about 20.000 websites. When > running nutch, I expected, that the fetcher would be pretty busy and > went for a coffee. Ok: perhaps someone talked to wife and decided, I > should not drink so much coffee. But I think, I made a mistake.. But > after 100 URLs he stopped working. > > After some tweaking I got the installation to fetch about 10.000 pages, > but this is still not what I expect. First guess was the url-filter, but > I see the urls in the tasktracker log. I looked at the mailinglist and > got many ideas, but I still get more confused. > > I think, the following parameters have an influence on the number of > pages fetched (in the brackets are the values I selected): > > - mapred.map.tasks (100) > - mapred.reduce.tasks (3) > - mapred.task.timeout (3600000 [an other question]) > - mapred.tasktracker.tasks.maximum (10) > - fetcher.threads.fetch (100) > - fetcher.server.delay (5.0) > - fetcher.threads.per.host (10) > - generate.max.per.host (1000) > - http.content.limit (2000000) > > I don't like my parameters, but so I got the most results. Looking at > the jobmanager, each "map task" fetched between 70 - 100 pages. Having > 100 map.tasks: I have ~ 8000 new pages fetched in the end. That's nearly > the number the crawldb says too. > > Which parameter has an influence on the number of pages ONE task > fetches. By my observations, I would guess it's "fetcher.threads.fetch". > Increasing this number further means, to blast the load on the > tasktrackers. So there must be an other problem. > > Any help appreciated! > > Regards > > Michael > ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642 _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
