Author: jnioche
Date: Fri Jun 27 07:38:45 2014
New Revision: 1605978
URL: http://svn.apache.org/r1605978
Log:
NUTCH-385 Improve description of thread related configuration for Fetcher
Modified:
nutch/trunk/CHANGES.txt
nutch/trunk/conf/nutch-default.xml
Modified: nutch/trunk/CHANGES.txt
URL:
http://svn.apache.org/viewvc/nutch/trunk/CHANGES.txt?rev=1605978&r1=1605977&r2=1605978&view=diff
==============================================================================
--- nutch/trunk/CHANGES.txt (original)
+++ nutch/trunk/CHANGES.txt Fri Jun 27 07:38:45 2014
@@ -2,6 +2,8 @@ Nutch Change Log
Nutch Current Development
+* NUTCH-385 Improve description of thread related configuration for Fetcher
(jnioche,lufeng)
+
* NUTCH-1633 slf4j is provided by hadoop and should not be included in the job
file (kaveh minooie via jnioche)
* NUTCH-1787 update and complete API doc overview page (snagel)
Modified: nutch/trunk/conf/nutch-default.xml
URL:
http://svn.apache.org/viewvc/nutch/trunk/conf/nutch-default.xml?rev=1605978&r1=1605977&r2=1605978&view=diff
==============================================================================
--- nutch/trunk/conf/nutch-default.xml (original)
+++ nutch/trunk/conf/nutch-default.xml Fri Jun 27 07:38:45 2014
@@ -684,7 +684,10 @@
<name>fetcher.server.delay</name>
<value>5.0</value>
<description>The number of seconds the fetcher will delay between
- successive requests to the same server.</description>
+ successive requests to the same server. Note that this might get
+ overriden by a Crawl-Delay from a robots.txt and is used ONLY if
+ fetcher.threads.per.queue is set to 1.
+ </description>
</property>
<property>
@@ -692,7 +695,7 @@
<value>0.0</value>
<description>The minimum number of seconds the fetcher will delay between
successive requests to the same server. This value is applicable ONLY
- if fetcher.threads.per.host is greater than 1 (i.e. the host blocking
+ if fetcher.threads.per.queue is greater than 1 (i.e. the host blocking
is turned off).</description>
</property>
@@ -723,7 +726,11 @@
<name>fetcher.threads.per.queue</name>
<value>1</value>
<description>This number is the maximum number of threads that
- should be allowed to access a queue at one time.
+ should be allowed to access a queue at one time. Setting it to
+ a value > 1 will cause the Crawl-Delay value from robots.txt to
+ be ignored and the value of fetcher.server.min.delay to be used
+ as a delay between successive requests to the same server instead
+ of fetcher.server.delay.
</description>
</property>
