On Fri, Jan 23, 2009 at 9:09 AM, amit sethi <[email protected]> wrote: > Well that is interesting but why should that happen in case I am using a > different User Agent because I tried doing > status=rp.can_fetch('Mozilla/5.0', > "http://en.wikipedia.org/wiki/Sachin_Tendulkar";) > but even that returns false > Is there something wrong with the syntax , Is there a catch that i don't > understand.
The problem is that you are using the standard Python user agent when getting the robots.txt. Because the user agent is refused, it cannot get the robots.txt file itself to look at. -- André Engels, [email protected] _______________________________________________ Tutor maillist - [email protected] http://mail.python.org/mailman/listinfo/tutor
