Barry A. Warsaw added the comment:
Two things: is it worth fixing this bug given the impending move to github?
Also, why is this reported here and not the pydotorg tracker?
https://github.com/python/pythondotorg/issues
Given that the last comment was 2014, I'm going to go ahead and close
Antoine Pitrou added the comment:
Yes, I think we should whitelist rather than blacklist. The problem with
letting engines index the repositories is the sheer resource cost when they
fetch many heavy pages (such as annotate, etc.).
--
___
Python
Emily Zhao added the comment:
I don't know too much about robots.txt but how about
Disallow: */rev/*
Disallow: */shortlog/*
Allow:
Are there any other directories we'd like to exclude?
--
nosy: +emily.zhao
___
Python tracker rep...@bugs.python.org
Benjamin Peterson added the comment:
Unfortunately, I don't think it will be that easy because I don't think
robots.txt supports wildcard paths like that. Possibly, we should just
whitelist a few important repositories.
--
nosy: +benjamin.peterson
Changes by Ezio Melotti ezio.melo...@gmail.com:
--
keywords: +easy
stage: - needs patch
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13924
___
Antoine Pitrou pit...@free.fr added the comment:
Can you propose a robots.txt file?
--
nosy: +georg.brandl, pitrou
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13924
___
Changes by Ezio Melotti ezio.melo...@gmail.com:
--
nosy: +ezio.melotti
___
Python tracker rep...@bugs.python.org
http://bugs.python.org/issue13924
___
___
New submission from Ivaylo Popov popov@gmail.com:
http://hg.python.org/robots.txt currently disallows all robots from all paths.
This means that the site doesn't show up in Google search results seeking, for
instance, browsing access to the python source
