Randal,
this is definitely Internet Explorer 5. I've seen the same several times on
my site and here's how it works: In IE5 you can make web pages available
for offline reading, you just need to go to Favorites/Add to Favorites ...
and select the number of levels you want to download. IE5 will hammer your
site and download as many levels as specified in the dialog until it has
seen all the links. It doesn't care at all about robots.txt and the
contents in there.
You may want to check out Apache::SpeedLimit or Apache::RobotLimit for your
site.
Sorry
Tobias
At 07:50 AM 10/28/99 -0700, Randal L. Schwartz wrote:
>
>In the past week or so, I've been seeing many many portions of my site
>sucked down in rapid fire. These sucks followed patterns of being a
>spider -- some autogenerated URLs from pod2html are definitely bad,
>and I saw these bad hits, indicating that someone is following every
>link in every one of my WT columns. robots.txt is *occasionally*
>being fetched, but not always.
>
>The only thing in common with these rude intrusions is a windows-IE
>user agent, along with a new string "DigExt".
>
>Has anyone else seen this? Can we trace this back to some lousy user
>interface somewhere? I tried blocking them, and I got back some users
>that wondered why I was blocking them.
>
>Is it a new version of IE that permits heavy rapid download?
>
>--
>Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
><[EMAIL PROTECTED]> <URL:http://www.stonehenge.com/merlyn/>
>Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
>See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
