Berker Peksag added the comment:
Thanks for the report. This is a duplicate of issue 15851.
--
nosy: +berker.peksag
resolution: -> duplicate
stage: -> resolved
status: open -> closed
superseder: -> Lib/robotparser.py doesn't accept setting a user agent string,
instead it uses the
New submission from John Nagle:
"robotparser" uses the default Python user agent when reading the "robots.txt"
file, and there's no parameter for changing that.
Unfortunately, the "mod_security" add-on for Apache web server, when used with
the standard OWASP rule set, blacklists the default
