On Sat, Mar 1, 2014 at 10:33 PM, Sam Gleske <sam.mxra...@gmail.com> wrote:
> @gimp-dev, @gimp-web
> My crawler has a stable release now. I'd like to request permission to
> crawl www.gimp.org. It will be restricted to that domain and have
> request delays implemented to minimize impact.
FYI most of the requests will only be HTTP method HEAD -
It only requests headers and not the response body so it would actually be
less of an impact than most crawlers when doing the actual link testing.
For the crawling itself the response body is received so that it can find
gimp-web-list mailing list