Ron Woodall <[EMAIL PROTECTED]> wrote: > Is there anyway that a search engine spyder can be identified with > analog? I tried pulling out robots.txt but I get 600+ redirects on it > but only two hits on it, one from me? (go figure) Is there some sort > of tell-tale behaviour? Don't say that any host that retrieves a lot > of files, that could be any proxy server. I have some that swallow > hundreds of files at a time.
Define a spider! A well behaved spider will ask for /robots.txt and identify itself in the Browser report (if you're logging Browser information). Any other kind of spider is obviously trying to avoid detection (or at least not interested in making itself obvious), so what characteristics do you think Analog should be looking for? You can probably think of ways to use Analog to help you track specific behaviour, but it's not going to do it automatically. Aengus +------------------------------------------------------------------------ | This is the analog-help mailing list. To unsubscribe from this | mailing list, go to | http://lists.isite.net/listgate/analog-help/unsubscribe.html | | List archives are available at | http://www.mail-archive.com/analog-help@;lists.isite.net/ | http://lists.isite.net/listgate/analog-help/archives/ | http://www.tallylist.com/archives/index.cfm/mlist.7 +------------------------------------------------------------------------
