>>>>> "Justin" == Justin <[EMAIL PROTECTED]> writes:
Justin> So I also simply outright ban these user agents:
Justin> (EmailSiphon)|(LinkWalker)|(WebCapture)|(w3mir)|
Justin> (WebZIP)|(Teleport Pro)|(PortalBSpider)|(Extractor)|
Justin> (Offline Explorer)|(WebCopier)|(NetAttache)|(iSiloWeb)|
Justin> (eCatch)|(ecila)|(WebStripper)|(Oxxbot)|(MuscatFerret)|
Justin> (AVSearch)|(MSIECrawler)|(SuperBot 2.4)
Here's my list, after running Stonehenge::Throttle for probably
longer than you have... :)
or m{Offline Explorer/} # bad robot!
or m{www\.gozilla\.com} # bad robot!
or m{pavuk-} # bad robot!
or m{ExtractorPro} # bad robot!
or m{WebCopier} # bad robot!
or m{MSIECrawler} # bad robot!
or m{WebZIP} # bad robot!
or m{Teleport Pro} # bad robot!
or m{NetAttache/} # bad robot!
or m{gazz/} # bad robot!
or m{geckobot} # bad robot!
or m{nttdirectory} # bad robot!
or m{Mister PiX} # bad robot!
or m{ia_archiver} # bad robot!
or m{DIIbot/} # bad robot!
or m{WhizBang!} # bad robot!
or m{WebCopy/} # bad robot!
or m{WebStripper/} # bad robot!
or m{EmailSiphon} # bad robot!
or m{AlkalineBOT} # bad robot! (in Perl!)
That last one is nasty. A Perl bot that basically sucks full speed.
Evil.
--
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<[EMAIL PROTECTED]> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
See PerlTraining.Stonehenge.com for onsite and open-enrollment Perl training!
