[
https://issues.apache.org/jira/browse/NUTCH-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13211294#comment-13211294
]
behnam nikbakht commented on NUTCH-1278:
----------------------------------------
here is a primary patch, that has some changes in Fetcher.java ,Protocol.java
and it's plugins like lib-http
i use a file in local system for maintaining a hashtable that contains hosts
and their http.timeout
for each blocked response, there is a increment in timeout and for each
success, there is a decrement
we can use different increment and decrement rates so we can make a balance
between total time of fetch Job, and a relation between fetched and blocked
rates. for example it can configurable that if 90% of requests for some host
are seccess, there is no need to increase timeout.
> Fetch Improvement in threads per host
> -------------------------------------
>
> Key: NUTCH-1278
> URL: https://issues.apache.org/jira/browse/NUTCH-1278
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 1.4
> Reporter: behnam nikbakht
> Attachments: NUTCH-1278.zip
>
>
> the value of maxThreads is equal to fetcher.threads.per.host and is constant
> for every host
> there is a possibility with using of dynamic values for every host that
> influeced with number of blocked requests.
> this means that if number of blocked requests for one host increased, then we
> most decrease this value and increase http.timeout
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
