Brion VIBBER has uploaded a new change for review. https://gerrit.wikimedia.org/r/92552
Change subject: Whitelist API mobileview for robots.txt ...................................................................... Whitelist API mobileview for robots.txt Per request from Google. Will allow Googlebot etc to run scraping through the Android app and through the mobile web site with active JS as we add more dynamic section loading to it. Note that while these API requests probably do not get fully HTTP-cached they do aggressively use the parser cache, and should be directed to API servers to limit load. Expected activity from Googlebot is maximum 50 hits/second, which will be a modest increase over existing traffic of approx 150 hits/second from apps and mobile web. If there are performance issues, please feel free to revert this change and contact [email protected] to let them know there were problems. They also recommend returning 503 errors as the best kill switch. Change-Id: I753eebd965bf7f6b63ad734fa728379b3701ad73 --- M robots.txt 1 file changed, 6 insertions(+), 0 deletions(-) git pull ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config refs/changes/52/92552/1 diff --git a/robots.txt b/robots.txt index caa3b29..9be241f 100644 --- a/robots.txt +++ b/robots.txt @@ -136,7 +136,13 @@ # bot supports such a thing using the 'Crawl-delay' or another # instruction, please let us know. # +# There is a special exception for API mobileview to allow dynamic +# mobile web & app views to load section content. +# These views aren't HTTP-cached but use parser cache aggressively +# and don't expose special: pages etc. +# User-agent: * +Allow: /w/api.php?action=mobileview& Disallow: /w/ Disallow: /trap/ Disallow: /wiki/Especial:Search -- To view, visit https://gerrit.wikimedia.org/r/92552 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I753eebd965bf7f6b63ad734fa728379b3701ad73 Gerrit-PatchSet: 1 Gerrit-Project: operations/mediawiki-config Gerrit-Branch: master Gerrit-Owner: Brion VIBBER <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
