On Tue, 25 Jan 2005 17:19, John Carter wrote: > I'm peeved. > > Too many web sites are hiding behind %$#% "free registrations". They > invite me in by letting Google spider them, and then keep me out with a > [EMAIL PROTECTED] registration window. > > I'm sick of it. > > Has anyone been spidered by Google recently, if so, can you have a look in > your /var/log/apache/access.log to see what the google spider reports its > useragent as? > > I want to build me a spiderman suit for my browser...
66.249.64.68 - - [11/Jan/2005:23:49:14 +1300] "GET /robots.txt HTTP/1.0" 404 300 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)" 66.249.64.68 - - [11/Jan/2005:23:52:19 +1300] "GET /KdeApps.pdf HTTP/1.0" 200 2826487 "-" "Googlebot/2.1(+http://www.google.com/bot.html) Being decent corporate citizens the do obey your wishes in robots.txt That's what they claim anyway. http://www.robotstxt.org/wc/robots.html http://www.kuznetsov.uklinux.net/robots-tutorial.php -- Sincerely etc., Christopher Sawtell
