Sebastian Nagel created NUTCH-2300: -------------------------------------- Summary: Fetcher to optionally save robots.txt Key: NUTCH-2300 URL: https://issues.apache.org/jira/browse/NUTCH-2300 Project: Nutch Issue Type: Improvement Components: fetcher, protocol, segment Reporter: Sebastian Nagel Fix For: 1.13
For debugging or archival purposes it may be useful to let Fetcher store the robots.txt response (content and HTTP status). Of course, this should be optional and not by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)