Brad Jorsch (Anomie), 07/07/2014 17:37: > And the robots.txt for the new > version of the site denies everything, likely because the new owners don't > want the redirects or other old content showing up in Google searches. But > this has the unfortunate side effect that IA removes all the old content > from public access.
This is not correct. If you can reproduce it, please file a bug, but it's not how it's supposed or said to work. I've pasted some links where you can find additional information like this at https://archive.org/post/1019415/retroactive-robotstxt-removal-of-past-crawls-aka-oakland-archive-policy (also reposting an elaborated version of my message of this morning). Nemo _______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines [email protected] Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:[email protected]?subject=unsubscribe>
