sebastian-nagel opened a new pull request #537:
URL: https://github.com/apache/nutch/pull/537
- if no agent names are given as command-line arguments use values
ofhttp.agent.name and http.robots.agents as agent names to be checked
- update command-line help
```
$> nutch org.apache.nutch.protocol.RobotRulesParser \
-Dhttp.agent.name='mybot' \
-Dhttp.robots.agents='nutch,goodbot' \
robots.txt urls.txt
Testing robots.txt for agent names: mybot,nutch,goodbot
allowed: https://www.example.com/
# command-line overwrite:
$> nutch org.apache.nutch.protocol.RobotRulesParser \
-Dhttp.agent.name='mybot' \
-Dhttp.robots.agents='nutch,goodbot' \
robots.txt urls.txt \
badbot,anybot
Testing robots.txt for agent names: badbot,anybot
not allowed: https://www.example.com/
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
