https://bugzilla.wikimedia.org/show_bug.cgi?id=62494
Bug ID: 62494
Summary: Add exception in robots.txt to allow the Internet
Archiver to index action=raw
Product: Wikimedia
Version: wmf-deployment
Hardware: All
OS: All
Status: NEW
Severity: normal
Priority: Unprioritized
Component: Site requests
Assignee: wikibugs-l@lists.wikimedia.org
Reporter: nathanlarson3...@gmail.com
CC: benap...@gmail.com,
bugzilla+org.wikime...@tuxmachine.com,
dereck...@espace-win.org, g...@wikimedia.org,
tom...@twkozlowski.net, wikimedia.b...@snowolf.eu
Web browser: ---
Mobile Platform: ---
Presently, robots.txt has Disallow: /w/ Thus, the raw wikitext of pages isn't
accessible via the Internet Archive; see e.g.
https://web.archive.org/web/20140307111730/http://en.wikipedia.org/w/index.php?title=Main_Page&action=raw
This is in contrast to sites like WikiIndex, which allow it:
https://web.archive.org/web/20131021230044/http://wikiindex.org/index.php?title=Welcome&action=raw
We should allow the Internet Archiver to index these pages so that the raw
wikitext will be available for future generations, even if the page goes away.
See
[[mw:Manual:Robots.txt#Allow_indexing_of_raw_pages_by_the_Internet_Archiver]].
--
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.
_______________________________________________
Wikibugs-l mailing list
Wikibugs-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikibugs-l
