Sebastian Nagel created NUTCH-3083: -------------------------------------- Summary: Add RobotRulesParser to bin/nutch Key: NUTCH-3083 URL: https://issues.apache.org/jira/browse/NUTCH-3083 Project: Nutch Issue Type: Improvement Components: bin Affects Versions: 1.21 Reporter: Sebastian Nagel Fix For: 1.21
The main method of the class {{org.apache.nutch.protocol.RobotRulesParser}} is quite useful if it's about verifying whether and how robots.txt files are parsed. It should be added to bin/nutch as *robotschecker*, similar to "parsechecker", "filterchecker", etc. -- This message was sent by Atlassian Jira (v8.20.10#820010)