Magnus Manske (2014-10-20 13:33):
[...]
AFAIK, all tools use a default .lighttp configuration by default. Is
that replaced or extended by a local config file?
If it's replaced, the default config could exclude Googlebot, and even
a blank local config file would re-enable Googlebot again, for those
who want it.
Options are merged not replaced (as docs say [docs]), but +1 for default
option to block crawlers would be fine. Though there should be
information on how to unblock them (probably in those docs).
[docs] =
https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Configuring_the_web_server
_______________________________________________
Labs-l mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/labs-l
