Dzahn has submitted this change and it was merged.
Change subject: Add a robots.txt to disallow indexing of API content
......................................................................
Add a robots.txt to disallow indexing of API content
But, make some limited effort to get the docs indexed.
Bug: T119786
Change-Id: I338a15398cb6583d417e6869acfe428f9fa3aa5e
---
M modules/restbase/templates/config.labs.yaml.erb
M modules/restbase/templates/config.yaml.erb
2 files changed, 24 insertions(+), 0 deletions(-)
Approvals:
Eevans: Looks good to me, but someone else must approve
Mobrovac: Looks good to me, but someone else must approve
jenkins-bot: Verified
Dzahn: Looks good to me, approved
diff --git a/modules/restbase/templates/config.labs.yaml.erb
b/modules/restbase/templates/config.labs.yaml.erb
index 0b92d0d..25a6f18 100644
--- a/modules/restbase/templates/config.labs.yaml.erb
+++ b/modules/restbase/templates/config.labs.yaml.erb
@@ -86,6 +86,18 @@
# global domain
/{domain:wikimedia.org}: *wikimedia.org
+ # A robots.txt to make sure that the content isn't indexed.
+ /robots.txt:
+ get:
+ x-request-handler:
+ - static:
+ return:
+ headers:
+ content-type: text/plain
+ body: |
+ User-agent: *
+ Allow: /*/v1/?doc
+ Disallow: /
# Finally, a standard service-runner config.
diff --git a/modules/restbase/templates/config.yaml.erb
b/modules/restbase/templates/config.yaml.erb
index 0baff58..007655e 100644
--- a/modules/restbase/templates/config.yaml.erb
+++ b/modules/restbase/templates/config.yaml.erb
@@ -829,6 +829,18 @@
# global domain
/{domain:wikimedia.org}: *wikimedia.org
+ # A robots.txt to make sure that the content isn't indexed.
+ /robots.txt:
+ get:
+ x-request-handler:
+ - static:
+ return:
+ headers:
+ content-type: text/plain
+ body: |
+ User-agent: *
+ Allow: /*/v1/?doc
+ Disallow: /
# Finally, a standard service-runner config.
--
To view, visit https://gerrit.wikimedia.org/r/257922
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I338a15398cb6583d417e6869acfe428f9fa3aa5e
Gerrit-PatchSet: 2
Gerrit-Project: operations/puppet
Gerrit-Branch: production
Gerrit-Owner: GWicke <[email protected]>
Gerrit-Reviewer: Alexandros Kosiaris <[email protected]>
Gerrit-Reviewer: Dzahn <[email protected]>
Gerrit-Reviewer: Eevans <[email protected]>
Gerrit-Reviewer: Filippo Giunchedi <[email protected]>
Gerrit-Reviewer: Mobrovac <[email protected]>
Gerrit-Reviewer: Ori.livneh <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
