20after4 has submitted this change and it was merged.
Change subject: add /files/ to robots.txt exclusion
......................................................................
add /files/ to robots.txt exclusion
Search engine spiders are stressing phabricator. See also T109247
Bug: T109279
Change-Id: Idcc93f99b0ea3ac21aabbd3c6ecd612c925ef4bc
---
M src/applications/system/controller/PhabricatorRobotsController.php
1 file changed, 2 insertions(+), 1 deletion(-)
Approvals:
Rush: Looks good to me, but someone else must approve
20after4: Looks good to me, approved
jenkins-bot: Verified
diff --git a/src/applications/system/controller/PhabricatorRobotsController.php
b/src/applications/system/controller/PhabricatorRobotsController.php
index b7767d7..11177f4 100644
--- a/src/applications/system/controller/PhabricatorRobotsController.php
+++ b/src/applications/system/controller/PhabricatorRobotsController.php
@@ -21,13 +21,14 @@
$out[] = 'User-Agent: *';
$out[] = 'Disallow: /diffusion/';
$out[] = 'Disallow: /multimeter/';
+ $out[] = 'Disallow: /file/';
// Add a small crawl delay (number of seconds between requests) for spiders
// which respect it. The intent here is to prevent spiders from affecting
// performance for users. The possible cost is slower indexing, but that
// seems like a reasonable tradeoff, since most Phabricator installs are
// probably not hugely concerned about cutting-edge SEO.
- $out[] = 'Crawl-delay: 1';
+ $out[] = 'Crawl-delay: 2';
$content = implode("\n", $out)."\n";
--
To view, visit https://gerrit.wikimedia.org/r/232061
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Idcc93f99b0ea3ac21aabbd3c6ecd612c925ef4bc
Gerrit-PatchSet: 1
Gerrit-Project: phabricator/phabricator
Gerrit-Branch: production
Gerrit-Owner: 20after4 <[email protected]>
Gerrit-Reviewer: 20after4 <[email protected]>
Gerrit-Reviewer: Rush <[email protected]>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
