I guess your looking to limit the bandwidth used by the Nutch crawler?
One idea is to set a "very low" thread count for the fetcher, maybe even going
right down to 1 depending on the effect your looking for. Another idea, which
would be much more controlled is to setup a throttling firewall on the same
machine or anywhere in-between it and your gateway.
If you go the thread route, you will need to place the below lines in your
nutch-site.xml file;
<property>
<name>fetcher.threads.fetch</name>
<value>1</value>
<description>The number of FetcherThreads the fetcher should use.
This is also determines the maximum number of requests that are
made at once (each FetcherThread handles one connection).</description>
</property>
----- Original Message ----
From: Harmesh <[EMAIL PROTECTED]>
To: [email protected]
Sent: Wednesday, March 7, 2007 2:01:50 AM
Subject: memory consumpition by nutch
while i am crawling most of the band with is used by nutch crawler is there
any way to optimize it
--
View this message in context:
http://www.nabble.com/memory-consumpition-by-nutch-tf3360441.html#a9347708
Sent from the Nutch - User mailing list archive at Nabble.com.
