| valhallasw created this task. valhallasw added projects: Pywikibot-weblinkchecker.py, Tool-Labs. Herald added subscribers: pywikibot-bugs-list, Aklapper. Herald added a project: Labs. |
TASK DESCRIPTION
(and probably listen to robots.txt in most cases as well)
In November 2016, a weblinkchecker bot on Tool Labs requested many pages from http://www.minorplanetcenter.net/, at a rate of > 100 req/minute. This caused excessive load on the web server, and resulted on Tool Labs being blocked from accessing the site.
Weblinkchecker should reduce the number of requests to a single site, preferrably to something along the lines of one every 10s or so. There are a large number of links to check, and pages different websites can reasonably be requested in parallel. The overall slowdown shouldn't be too significant.
Access log from external site: F4978348: Access log from Harvard, reason for banning labs. (only visible to selected users)
TASK DETAIL
EMAIL PREFERENCES
To: valhallasw
Cc: Aklapper, valhallasw, pywikibot-bugs-list, Freddy2001, JJMC89, Luke081515, yuvipanda, Gryllida, jayvdb, Wesalius, scfc, coren, chasemp
Cc: Aklapper, valhallasw, pywikibot-bugs-list, Freddy2001, JJMC89, Luke081515, yuvipanda, Gryllida, jayvdb, Wesalius, scfc, coren, chasemp
_______________________________________________ pywikibot-bugs mailing list [email protected] https://lists.wikimedia.org/mailman/listinfo/pywikibot-bugs
