[
https://issues.apache.org/jira/browse/NUTCH-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma updated NUTCH-3120:
---------------------------------
Attachment: NUTCH-3120-1.15.patch
> Automatically increase crawl-delay on HTTP 429
> ----------------------------------------------
>
> Key: NUTCH-3120
> URL: https://issues.apache.org/jira/browse/NUTCH-3120
> Project: Nutch
> Issue Type: Improvement
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Priority: Major
> Fix For: 1.22
>
> Attachments: NUTCH-3120-1.15.patch
>
>
> Thought i remember a discussion or ticket on this subject, but it seems no
> code of this sort is in master at the moment.
> Anyway, small patch that adds HTTP429 to ProtocolStatus, the setter for that
> status to HttpBase, and the reading of that status in FetcherThread, so it
> can adjust the fetch speed (hardcoded to *3 right now).
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
