[ https://issues.apache.org/jira/browse/NUTCH-2630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebastian Nagel resolved NUTCH-2630. ------------------------------------ Resolution: Fixed Committed to master/1.x. > Fetcher to log skipped records by robots.txt > -------------------------------------------- > > Key: NUTCH-2630 > URL: https://issues.apache.org/jira/browse/NUTCH-2630 > Project: Nutch > Issue Type: Improvement > Components: fetcher > Affects Versions: 1.15 > Reporter: Markus Jelsma > Priority: Minor > Fix For: 1.16 > > > To analyze problems it would be helpful if fetcher logs URLs which are > disallowed in the robots.txt - see [discussion on user mailing > list|https://lists.apache.org/thread.html/7fe5b02104ea866aba183d009a5fad59ad4e4daf8954593ef0123dd6@%3Cuser.nutch.apache.org%3E]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)