Hi, nutch can only control the number of threads per host and fetch delay to achieve the crawl politeness. you can see these fetcher properties of fetcher.threads.fetch,fetcher.threads.per.queue, fetcher.server.delay and etc. There are all related with Number of politeness.
On Fri, Mar 29, 2013 at 1:54 AM, Yves S. Garret <[email protected]>wrote: > I was able to look into ${APACHE_NUTCH_HOME}/conf/nutch-default.xml and it > listed a very good explanation of each term that I can use to throttle my > crawling. I should be all set for now unless there's something that I'm > seriously not getting. > > ---------- Forwarded message ---------- > From: Yves S. Garret <[email protected]> > Date: Thu, Mar 28, 2013 at 12:55 PM > Subject: How to set politeness in Nutch 2.1? > To: [email protected] > > > Hi, I got a really embarrassing question. After googling for this answer > for some time, I can't find out how to set the politeness level when I > crawl through a site. I don't want to bombard a site. Any thoughts or > pointers on how to do this? > -- Don't Grow Old, Grow Up... :-)
