Dr0ptp4kt has uploaded a new change for review. https://gerrit.wikimedia.org/r/95548
Change subject: WIP: DO NOT MERGE YET. Allow Google's bots to scrape bits. ...................................................................... WIP: DO NOT MERGE YET. Allow Google's bots to scrape bits. * Google's algorithms identify mobile-friendliness and the like. * But bits.wikimedia.org's robots.txt disallows indexing. * Thus, currently Google's bots can't do comparisons. * Only select bots will be allowed access. * THIS IS A WIP. DO NOT MERGE YET. Change-Id: I267803e4b2a6532ec75543e140b9ee4bdd459479 --- M docroot/bits/robots.txt A robots-bits.txt 2 files changed, 9 insertions(+), 1 deletion(-) git pull ssh://gerrit.wikimedia.org:29418/operations/mediawiki-config refs/changes/48/95548/1 diff --git a/docroot/bits/robots.txt b/docroot/bits/robots.txt index 2843912..cc59a2d 120000 --- a/docroot/bits/robots.txt +++ b/docroot/bits/robots.txt @@ -1 +1 @@ -/apache/common/robots-private.txt \ No newline at end of file +/apache/common/robots-bits.txt \ No newline at end of file diff --git a/robots-bits.txt b/robots-bits.txt new file mode 100644 index 0000000..d454248 --- /dev/null +++ b/robots-bits.txt @@ -0,0 +1,8 @@ +User-agent: * +Disallow: / + +User-agent: Googlebot +Disallow: + +User-agent: Google-Mobile +Disallow: \ No newline at end of file -- To view, visit https://gerrit.wikimedia.org/r/95548 To unsubscribe, visit https://gerrit.wikimedia.org/r/settings Gerrit-MessageType: newchange Gerrit-Change-Id: I267803e4b2a6532ec75543e140b9ee4bdd459479 Gerrit-PatchSet: 1 Gerrit-Project: operations/mediawiki-config Gerrit-Branch: master Gerrit-Owner: Dr0ptp4kt <[email protected]> _______________________________________________ MediaWiki-commits mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
