Hi Corey, hi list, perhaps you already know, but http://www.w3.org/Robot/ is worth a visit: it is a free robot available in portable source code. It does database logging with MySQL. Another software available as source code I recommend is "wget": look out for wget-1.5.2.tar.gz, which does the things you described.
Regards Christian Corey Wineman wrote: [snip] > files, storing the results( things like, if the file meets a certain > criteria, what is the IP of the site, when did I visit the site) to a > database. I would like to be able to configure the spider. Limiting the > depth from a source URL, limiting the depth it will search onto external > sites, and setting the defaults on various timeouts. > > Does anyone know of a webspider that does some of these things and is > available along with the source code? > > Thanks, > Corey -- CSK Computer Services Kaisha Deutschland GmbH Christian Brunner | Tel: (+49) 69 / 50952-234 Holzhausenstrasse 44 | Fax: (+49) 69 / 50952-299 60322 Frankfurt a. M. | E-Mail: [EMAIL PROTECTED]
