Philip Arndt wrote: > What's the IP address of the spider? :-D For me it has exclusively been 149.20.55.4.
The spider has been sucking down content for the past 7 days. It's been my biggest user (in bytes) for half of those days. It's just a pity that it isn't following robots.txt as it has gotten stuck in one of my calendar apps and indexed my room and equipment bookings for the past 8 years. Oh well, it's going to be an expensive crawl (for me). I won't be blocking it as I have found archive.org useful, but I will implement some things to restrict the depth of crawling of runaway bots. Paul --~--~---------~--~----~------------~-------~--~----~ NZ PHP Users Group: http://groups.google.com/group/nzphpug To post, send email to [email protected] To unsubscribe, send email to [EMAIL PROTECTED] -~----------~----~----~----~------~----~------~--~---
