Bug#67637: www.debian.org: robots.txt has too many entries

[Nicolás Lichtmaier]

> >  Current /robots.txt prohibits indexing of many resources that should be
> > indexed.
> Probably because search engines would overload www.debian.org otherwise.

 This can be easily checked.. are there any log analisis that has shown
this?