[issue25400] robotparser doesn't return crawl delay for default entry

Changes by Donald Stufft : -- pull_requests: +1034 ___ Python tracker ___ ___

[issue25400] robotparser doesn't return crawl delay for default entry

Changes by Berker Peksag : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue25400] robotparser doesn't return crawl delay for default entry

Roundup Robot added the comment: New changeset d5d910cfd288 by Berker Peksag in branch '3.6': Issue #25400: RobotFileParser now correctly returns default values for crawl_delay and request_rate https://hg.python.org/cpython/rev/d5d910cfd288 New changeset 911070065e38 by Berker Peksag in branch

[issue25400] robotparser doesn't return crawl delay for default entry

Changes by Berker Peksag : Added file: http://bugs.python.org/file44740/issue25400_v3.diff ___ Python tracker ___

[issue25400] robotparser doesn't return crawl delay for default entry

Berker Peksag added the comment: Here's an updated patch. -- versions: +Python 3.7 Added file: http://bugs.python.org/file44739/issue25400_v2.diff ___ Python tracker

[issue25400] robotparser doesn't return crawl delay for default entry

Berker Peksag added the comment: I've now updated Lib/test/test_robotparser.py (issue 25497) Peter, do you have time to update your patch? Thanks! -- ___ Python tracker

[issue25400] robotparser doesn't return crawl delay for default entry

Peter Wirtz added the comment: On further inspection of the tests, it appears that the way in which the tests are written, a test case can only be tested for one useragent at a time. I will attempt to work on the tests so work correctly. Any advice would be much appreciated. --

[issue25400] robotparser doesn't return crawl delay for default entry

Berker Peksag added the comment: Thanks for the patch Peter(and welcome to Python and open source development). I have a WIP patch to rewrite test_robotparser in a less magic way. So we can ignore test failures for now. I'll take a closer look to your patch. -- stage: -> patch review

[issue25400] robotparser doesn't return crawl delay for default entry

Changes by Berker Peksag : -- nosy: +berker.peksag ___ Python tracker ___ ___

[issue25400] robotparser doesn't return crawl delay for default entry

Peter Wirtz added the comment: Ok, for the mean time, I reworked the test so it appears to test correctly and tests passes. There does seem to be some magic, so I do hope I did not overlook anything. Here is the new patch. -- Added file:

[issue25400] robotparser doesn't return crawl delay for default entry

New submission from Peter Wirtz: After changeset http://hg.python.org/lookup/dbed7cacfb7e, calling the crawl_delay method for a robots.txt files that has a crawl-delay for * useragents always returns None. Ex: Python 3.6.0a0 (default:1aae9b6a6929+, Oct 9 2015, 22:08:05) [GCC 4.2.1

[issue25400] robotparser doesn't return crawl delay for default entry

Peter Wirtz added the comment: This fix breaks the unit tests though. I am not sure how to go about checking those as this would be my first contribution to python and an open source project in general. -- ___ Python tracker