Bandwidth target for fetcher rather than a thread count
-------------------------------------------------------
Key: NUTCH-207
URL: http://issues.apache.org/jira/browse/NUTCH-207
Project: Nutch
Type: New Feature
Components: fetcher
Versions: 0.8-dev
Reporter: Rod Taylor
Attachments: ratelimit.patch
Increases or decreases the number of threads from the starting value
(fetcher.threads.fetch) up to a maximum (fetcher.threads.maximum) to achieve a
target bandwidth (fetcher.threads.bandwidth).
It seems to be able to keep within 10% of the target bandwidth even when large
numbers of errors are found or when a number of large pages is run across.
To achieve more accurate tracking Nutch should keep track of protocol overhead
as well as the volume of pages downloaded.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems? Stop! Download the new AJAX search engine that makes
searching your log files as easy as surfing the web. DOWNLOAD SPLUNK!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=103432&bid=230486&dat=121642
_______________________________________________
Nutch-developers mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/nutch-developers
