I'm responsible for assessing the use of our library website by
pondering the web server logs. The log I get from our system
administrator doesn't include a translation of the numerical addresses
of what Analog calls Hosts, nor does it have anything about User Agent.
<snip...>
Out of curiosity I have been translating the numerical addresses of the
Hosts from which we get a lot of requests, and I notice that a lot of
them are from googlebot and the like. I'm thinking this doesn't really
count as the sort of "use" of our website I want to keep track of. On
the other hand, disregarding requests from these sites takes some
effort.
<snip...>
So I'm looking for advice about what to do. I'm thinking about looking
at the hosts from which I get more than x requests in a month, figuring
out which of those are search engines, and throwing them out. I might
use a percentage of requests rather than a specific number of requests
as a threshhold.
Thanks for your attention.
Tom Zimoski
Reference Dept/Fresno County Library
Tom,
What you are looking to do is called DNS Lookup. analog has documentation at http://analog.cx/docs/dns.html. The first time you have analog do DNS Lookup, it will take a long time as analog goes and translates each IP address to its host name. You can configure analog so that subsequent runs of analog go much faster by using the DNS cache (see DNSFILE).
Robots (search engines and the like) are handled through the ROBOT* commands. See http://analog.cx/docs/include.html#ROBOTINCLUDE for a start. The pre-configured .cfg file contains a good starting point.
| Leonard Daly <[EMAIL PROTECTED]>
| Internet Development http://realism.com/
| e3D News Technical Editor http://e3dNews.com/
| SIGGRAPH 2002&2003 X3D Course Organizer
| Member, Web3D Board of Directors
+------------------------------
+------------------------------------------------------------------------ | TO UNSUBSCRIBE from this list: | http://lists.meer.net/mailman/listinfo/analog-help | | Usenet version: news://news.gmane.org/gmane.comp.web.analog.general | List archives: http://www.analog.cx/docs/mailing.html#listarchives +------------------------------------------------------------------------
