I want to limit the crawling rate, for example one fetch every 23s.
Therefore I put
<property>
<name>fetcher.server.delay</name>
<value>23.0</value>
<description>The number of seconds the fetcher will delay between
successive requests to the same server.</description>
</property>
in nutch-site.xml, but Nutch still crawls every three seconds which is
the limit given in the robots.txt of the server.
How to slow down Nutch when the property above is not working?
mfg ar
--
Hier könnte keine Signatur stehen.
