Tejas Patil created NUTCH-1716:
----------------------------------
Summary: RobotRulesParser adds extra '*' to the robots name
Key: NUTCH-1716
URL: https://issues.apache.org/jira/browse/NUTCH-1716
Project: Nutch
Issue Type: Bug
Components: fetcher
Affects Versions: 2.2.1, 1.7
Reporter: Tejas Patil
Assignee: Tejas Patil
Fix For: 2.3, 1.8
In RobotRulesParser, when Nutch creates a agent string from multiple agents, it
combines agents from both 'http.agent.name' and 'http.robots.agents'. Along
with that it appends a wildcard (ie. *) to it in the end. This is sent to
crawler commons while parsing the rules. The wildcard gets matched first in
robots file with (User-agent: *) if that comes before any other matching rule
thus resulting in a allowed url being robots denied.
This bug was reported by @Markus Jelsma. The discussion over nutch-user can be
found here:
http://mail-archives.apache.org/mod_mbox/nutch-user/201401.mbox/%3CCAFKhtFzBRpVv4MULSxw8RDRR_wbivOt%3DnhFX-w621BR8q%2BxVDQ%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
