GWicke has uploaded a new change for review.
https://gerrit.wikimedia.org/r/257922
Change subject: Add a robots.txt to disallow indexing of API content
......................................................................
Add a robots.txt to disallow indexing of API content
But, make some limited effort to get the docs indexed.
Bug: T119786
Change-Id: I338a15398cb6583d417e6869acfe428f9fa3aa5e
---
M modules/restbase/templates/config.labs.yaml.erb
M modules/restbase/templates/config.yaml.erb
2 files changed, 24 insertions(+), 0 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/operations/puppet
refs/changes/22/257922/1
diff --git a/modules/restbase/templates/config.labs.yaml.erb
b/modules/restbase/templates/config.labs.yaml.erb
index 0b92d0d..25a6f18 100644
--- a/modules/restbase/templates/config.labs.yaml.erb
+++ b/modules/restbase/templates/config.labs.yaml.erb
@@ -86,6 +86,18 @@
# global domain
/{domain:wikimedia.org}: *wikimedia.org
+ # A robots.txt to make sure that the content isn't indexed.
+ /robots.txt:
+ get:
+ x-request-handler:
+ - static:
+ return:
+ headers:
+ content-type: text/plain
+ body: |
+ User-agent: *
+ Allow: /*/v1/?doc
+ Disallow: /
# Finally, a standard service-runner config.
diff --git a/modules/restbase/templates/config.yaml.erb
b/modules/restbase/templates/config.yaml.erb
index 0baff58..007655e 100644
--- a/modules/restbase/templates/config.yaml.erb
+++ b/modules/restbase/templates/config.yaml.erb
@@ -829,6 +829,18 @@
# global domain
/{domain:wikimedia.org}: *wikimedia.org
+ # A robots.txt to make sure that the content isn't indexed.
+ /robots.txt:
+ get:
+ x-request-handler:
+ - static:
+ return:
+ headers:
+ content-type: text/plain
+ body: |
+ User-agent: *
+ Allow: /*/v1/?doc
+ Disallow: /
# Finally, a standard service-runner config.
--
To view, visit https://gerrit.wikimedia.org/r/257922
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I338a15398cb6583d417e6869acfe428f9fa3aa5e
Gerrit-PatchSet: 1
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: GWicke <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
