Randal,
This is IE5. It has to do with the offline reading feature, it will try and
download as much of your site as it can depending on what the user has specified
in number of levels to download. It totaly ignores robots.txt.
Frank
"Randal L. Schwartz" wrote:
> In the past week or so, I've been seeing many many portions of my site
> sucked down in rapid fire. These sucks followed patterns of being a
> spider -- some autogenerated URLs from pod2html are definitely bad,
> and I saw these bad hits, indicating that someone is following every
> link in every one of my WT columns. robots.txt is *occasionally*
> being fetched, but not always.
>
> The only thing in common with these rude intrusions is a windows-IE
> user agent, along with a new string "DigExt".
>
> Has anyone else seen this? Can we trace this back to some lousy user
> interface somewhere? I tried blocking them, and I got back some users
> that wondered why I was blocking them.
>
> Is it a new version of IE that permits heavy rapid download?
>
> --
> Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
> <[EMAIL PROTECTED]> <URL:http://www.stonehenge.com/merlyn/>
> Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
> See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
