Sebastian Nagel created NUTCH-3011:
--------------------------------------
Summary: HttpRobotRulesParser: handle HTTP 429 Too Many Requests
same as server errors (HTTP 5xx)
Key: NUTCH-3011
URL: https://issues.apache.org/jira/browse/NUTCH-3011
Project: Nutch
Issue Type: Improvement
Affects Versions: 1.19
Reporter: Sebastian Nagel
Assignee: Sebastian Nagel
Fix For: 1.20
HttpRobotRulesParser should handle HTTP 429 Too Many Requests same as server
errors (HTTP 5xx), that is if configured signalize Fetcher to delay requests.
See also NUTCH-2573 and
https://support.google.com/webmasters/answer/9679690#robots_details
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
