sebastian-nagel commented on a change in pull request #539:
URL: https://github.com/apache/nutch/pull/539#discussion_r453188606
##########
File path: conf/nutch-default.xml
##########
@@ -118,7 +118,7 @@
</property>
<property>
- <name>http.robot.rules.whitelist</name>
+ <name>http.robot.rules.allowlist</name>
Review comment:
Just thinking about the naming of the property and whether
`http.robotstxt.ignore` might be the better naming. Or is there is a more
precise alternative?
Let's first state what the property effects: If a host or domain is listed
in the property
- no robots.txt is fetched and an empty robots rules set is assumed for this
host. Consequently, for this host:
- all URLs are allowed (no URL is disallowed)
- the Crawl-Delay is ignored
- no sitemaps are detected
In short, it has many more effects than "allow" any URL from the specified
host. Also, robots rules in HTML metadata or HTTP headers (noindex, nofollow)
are not affected and may still cause that pages from the specified host are not
indexed.
Also important to emphasize: robots.txt rules tell search engine crawlers
what is (not) worth to index: exclude duplicates (same content with different
view for print, re-sorted, re-formatted), skip over undesired content
(private), etc. That's why it might be smarter to [follow the rules of a more
privileged
bot](http://www.thesempost.com/apples-applebot-follows-googlebots-instructions-in-robots-txt-files/).
All these disadvantages and site-effects of ignoring the robots.txt should
be documented, see
[NUTCH-2808](https://issues.apache.org/jira/browse/NUTCH-2808) and
[NUTCH-2807](https://issues.apache.org/jira/browse/NUTCH-2807).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
