This bug, "[ 813986 ] robotparser interactively prompts for username and password", has been open since 2003. It killed a big batch job of ours last night.
Module "robotparser" naively uses "urlopen" to read "robots.txt" URLs. If the server asks for basic authentication on that file, "robotparser" prompts for the password on standard input. Which is rarely what you want. You can demonstrate this with: import robotparser url = 'http://mueblesmoraleda.com' # this site is password-protected. parser = robotparser.RobotFileParser() parser.set_url(url) parser.read() # Prompts for password That's the tandard, although silly, "urllib" behavior. This was reported in 2003, and a patch was uploaded in 2005, but the patch never made it into Python 2.4 or 2.5. A temporary workaround is this: import robotparser def prompt_user_passwd(self, host, realm): return None, None robotparser.URLopener.prompt_user_passwd = prompt_user_passwd # temp patch John Nagle -- http://mail.python.org/mailman/listinfo/python-list
