[
https://issues.apache.org/jira/browse/NUTCH-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13881695#comment-13881695
]
Hudson commented on NUTCH-1715:
-------------------------------
SUCCESS: Integrated in Nutch-trunk #2505 (See
[https://builds.apache.org/job/Nutch-trunk/2505/])
NUTCH-1715 RobotRulesParser adds additional * to the robots name (tejasp:
http://svn.apache.org/viewvc/nutch/trunk/?view=rev&rev=1561087)
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/src/java/org/apache/nutch/protocol/RobotRulesParser.java
> RobotRulesParser adds additional '*' to the robots name
> -------------------------------------------------------
>
> Key: NUTCH-1715
> URL: https://issues.apache.org/jira/browse/NUTCH-1715
> Project: Nutch
> Issue Type: Bug
> Components: fetcher
> Affects Versions: 1.7, 2.2.1
> Reporter: Tejas Patil
> Assignee: Tejas Patil
> Fix For: 2.3, 1.8
>
> Attachments: NUTCH-1715.2.x.patch, NUTCH-1715.trunk.patch
>
>
> In RobotRulesParser, when Nutch creates a agent string from multiple agents,
> it combines agents from both 'http.agent.name' and 'http.robots.agents'.
> Along with that it appends a wildcard (ie. *) to it in the end. This is sent
> to crawler commons while parsing the rules. The wildcard gets matched first
> in robots file with (User-agent: *) if that comes before any other matching
> rule thus resulting in a allowed url being robots denied.
> This issue was reported by [~markus17]. The discussion over nutch-user is
> here:
> http://mail-archives.apache.org/mod_mbox/nutch-user/201401.mbox/%3CCAFKhtFzBRpVv4MULSxw8RDRR_wbivOt%3DnhFX-w621BR8q%2BxVDQ%40mail.gmail.com%3E
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
