National Archive's this month are attempting to make a copy of all NZ internet sites with a .nz on the domain.
More information can be seen here: http://www.natlib.govt.nz/about-us/current-initiatives/web-harvest-2008 We first noticed this when one of our clients had a spike in international traffic, and only in one day reached 2GB of international traffic, all coming from the Harvest user agent. And to make this even better they are ignoring robots.txt protocol. Has anyone else noticed this or concerned by it? Apparently you can contact the National archives and request this to be disable for a single site. However this seems to be only noticed after the fact. Especially since most businesses will get their hosting bill at the end of the month anyway. --~--~---------~--~----~------------~-------~--~----~ NZ PHP Users Group: http://groups.google.com/group/nzphpug To post, send email to [email protected] To unsubscribe, send email to [EMAIL PROTECTED] -~----------~----~----~----~------~----~------~--~---
