At 1:19 PM -0500 1/28/02, Geoff Hutchison wrote:
>On Mon, 28 Jan 2002 [EMAIL PROTECTED] wrote:
>
>> There are a very few indiscriminate robots that follow all links:
>> comments, alt tags, javascript, even text, everything. The
>> PicoSearch service tends do to this. Most others are much more
>> reasonable.
>
>By this, Avi, I'm assuming you mean the robots follow anything that looks
>like a URL in addition to actual URL tags like <a href=...>, <link ...>,
>etc.?
Yep, the indiscriminate ones send everything vaguely URL-like as a
request. It's weird to watch.
Avi
--
Complete Guide to Search Engines for Web Sites and Intranets
<http://www.searchtools.com>
_______________________________________________
htdig-general mailing list <[EMAIL PROTECTED]>
To unsubscribe, send a message to <[EMAIL PROTECTED]> with a
subject of unsubscribe
FAQ: http://htdig.sourceforge.net/FAQ.html
