[ https://issues.apache.org/jira/browse/NUTCH-2002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17099977#comment-17099977 ]
Hudson commented on NUTCH-2002: ------------------------------- SUCCESS: Integrated in Jenkins build Nutch-trunk #3681 (See [https://builds.apache.org/job/Nutch-trunk/3681/]) NUTCH-2002 parse and index checkers to check robots.txt - applied (snagel: [https://github.com/apache/nutch/commit/46db3ed71355fefda42a008ece75094f51859ab2]) * (edit) src/java/org/apache/nutch/util/AbstractChecker.java * (edit) src/java/org/apache/nutch/indexer/IndexingFiltersChecker.java * (edit) src/java/org/apache/nutch/parse/ParserChecker.java > ParserChecker and IndexingFiltersChecker to check robots.txt > ------------------------------------------------------------ > > Key: NUTCH-2002 > URL: https://issues.apache.org/jira/browse/NUTCH-2002 > Project: Nutch > Issue Type: Improvement > Components: parser > Affects Versions: 1.9 > Reporter: Julien Nioche > Assignee: Sebastian Nagel > Priority: Minor > Fix For: 1.17 > > Attachments: NUTCH-2002.patch > > > ParserChecker could check whether a given URL is allowed by the robots.txt > directives. -- This message was sent by Atlassian Jira (v8.3.4#803005)