I don't see this as a problem is you get a dozen mirros. This would be ~200mb/day per site. I have a couple hosted sites alone that pull 2gb data per day. If the sites are load balanced in some way then it shouldn't be a problem.
Also, here is an idea. The page that lists the time for the cronjob sample. Put that into a dynamic html page that will generate a random time so these people will not just cut and paste. I know that it says not to but people are people. Also, I'm not saying stop the rate limiter. I have said rethink it. Each response talks about people who are hitting it every couple minutes. Let's change it to say something like 4 hours. Something else we can do to help would be to modify RDJ to not download the files but to download a control file that contains all of the versions in it. It would probably be 1k in size. If the version of a single one has changed then pull it. As for the mirror, since this is a viable project that my clients, myself and the community benefits from I will put up 1/2gb bandwidth per day to the cause... Gary > > One of the more recent examples cited to us by the fine folks hosting > rulesemporium.com was the usage for July 14th, over 2.5 GB of data was > pulled from > the site. As far as I can see, at most there was only one file that may > have been > updated that day. Thats a whole lot of people pulling the same data over > and over > and over....... > > If we decide to mirror the site, thats a sample of the bandwidth we'll > need unless > we come up with a method of controlling those who setup cron jobs to > download the > data every few minutes. > > If you have not already made sure that you are pulling the rulesets once > per 24 hour > period, please do so. > > For those that are already configured to only pull once a day, let me say > thank > you! We're working on fixing the rate limiter and discussing how to > handle this > long term, so please be patient. :) > > -matt
