On 2006-10-03 20:10:14 +0200, A.M. Kuchling wrote: > I've added a robots.txt to keep crawlers out of /dev/.
Isn't there a lot of useful, search-engine worthy stuff in /dev? I search for peps with google, and I suppose the 'explanation' section, as well as the developer faq and subversion instructions, are good pages that deserve to be in the google index. Should /dev really be Disallow:'ed entirely in robots.txt? kind regards, Gerrit Holl. _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
