On Wed, Jun 25, 2014 at 11:13 AM, M.-A. Lemburg <[email protected]> wrote: > On 25.06.2014 06:16, anatoly techtonik wrote: >> http://web.archive.org/web/*/http://docs.python.org/release/2.2.3/whatsnew/node5.html > > Can you propose an alternative robots.txt file ? > > """ > # Prevent development and old documentation from showing up in search results. > User-agent: * > # Disallow: /dev > Disallow: /release > """
Try this: """ User-agent: ia_archiver Disallow: User-agent: * Disallow: /release """ _______________________________________________ pydotorg-www mailing list [email protected] https://mail.python.org/mailman/listinfo/pydotorg-www
