Chris A. Mattmann updated NUTCH-475:

    Fix Version/s:     (was: 1.1)

- pushing this out per http://bit.ly/c7tBv9

> Adaptive crawl delay
> --------------------
>                 Key: NUTCH-475
>                 URL: https://issues.apache.org/jira/browse/NUTCH-475
>             Project: Nutch
>          Issue Type: Improvement
>          Components: fetcher
>            Reporter: Doğacan Güney
>         Attachments: adaptive-delay_draft.patch
> Current fetcher implementation waits a default interval before making another 
> request to the same server (if crawl-delay is not specified in robots.txt). 
> IMHO, an adaptive implementation will be better. If the server is under 
> little load and can server requests fast, then fetcher can ask for more pages 
> in a given interval. Similarly, if the server is suffering from heavy load, 
> fetcher can slow down(w.r.t that host), easing the load on the server.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to