Recently I saw LWP's WWW::RobotRules seeing a robots.txt file that looked
like this:
#
User-agent: *
Disallow: /cgi-bin/
Disallow: /~mojojojo/misc/
It complained about the Disallow lines being "unexpected".
The regexp it was using for these things is:
/^Disallow:\s*(.*)/i
So I've changed it to this, and was about to submit it as a patch for the
next LWP release:
/^\s*Disallow:\s*(.*)/i
# Silently forgive leading whitespace.
But first, I thought I'd ask the list here: does anyone thing this'd break
anything? I sure hope no-one out there is using leading-whitespace lines
as comments, or as RFC-822-style continuation lines!
Thoughts, anyone?
--
Sean M. Burke [EMAIL PROTECTED] http://www.spinn.net/~sburke/
