In [EMAIL PROTECTED], Sean M. Burke
[EMAIL PROTECTED] writes:
User-agent: *
Disallow: /cgi-bin/
Disallow: /~mojojojo/misc/
So I've changed it to this, and was about to submit it as a patch for the
next LWP release:
/^\s*Disallow:\s*(.*)/i
# Silently forgive leading
I've been hitting problems with a Python-based robot I'm working on and just
found out that there's a timeout module that will make it easy to implement
the kind of functionality that Tim Bray was suggesting here earlier. It
apparently works for any TCP connection. Here's the link:
