On Sun, 15 Aug 2010 12:39:34 -0400 Anthony <[email protected]> wrote: > I see http://planet.openstreetmap.org/robots.txt now has User-agent: * > and Disallow: / > > Are we allowed to download the minute-replicate files as they become > available? If not, what's the point of having them?
I think the sysadmins just want to block search engine spiders which download bz2 files. Your own scripts are robots, but I think it isn't considered abusive if they don't obey robots.txt in case they only download the diffs (once) and apply them to a database. If you want an explicit permission, ask someone from the admin team to create a robots.txt entry in this form: ####### User-agent: AnthonyBot Disallow: User-agent: * Disallow: / ####### Regards, Roland _______________________________________________ dev mailing list [email protected] http://lists.openstreetmap.org/listinfo/dev
