Hi Peter, Am 28.10.2013 um 13:34 schrieb Peter Bonivart <[email protected]>: > On Mon, Oct 28, 2013 at 11:05 AM, Dagobert Michelsen <[email protected]> wrote: >> Am 28.10.2013 um 10:37 schrieb Maciej (Matchek) Bliziński >> <[email protected]>: >>> Hey Peter (B) and maintainers, >>> >>> I spoke to Dago a few days ago, and we had a chat about a large portion of >>> traffic from our main mirror being just the catalog files, that is, the >>> files named 'catalog' that are downloaded and re-downloaded a countless >>> number of times. The mirror can withstand it, but it's a constant stream of >>> a few megabytes per second, day and night. >> >> Some numbers: we have constantly 3-4 MB per second. This is not a problem >> ATM as we >> have a direct gigabit uplink to the internet, but summing this up it is >> roughly >> 10 TB. Just as a comparison: Amazon would charge $0,120 per GB resulting in >> 1200$ !! >> So I would like to take the initiative and see that we save bandwidth now >> that we still >> have the cheap mirror. >> >>> Perhaps this can be helped by using the conditional GET with the possible >>> HTTP 304 Not Modified response, or timestamping. wget has an option to >>> timestamp files, and it can issue just a HEAD request to skip downloading >>> the whole file. Here's some information I found: >>> >>> http://www.gnu.org/software/wget/manual/wget.html#Time_002dStamping >>> >>> Have we considered this in the past? I don't recall it. Maybe we should >>> take a look, it could be simple to implement, and we would save some >>> bandwidth on our main mirror and on other mirrors worldwide. >> >> Just adding --timestamping would already be a great benefit. >> >> Peter, what do you think? > > I could do some tests I guess. What I did was to make the default for > expired catalogs 14 days but I think most people add -U to their > command line all the time.
I think the main problem is with our puppet provider, maybe it should be changed there. When I talked to some downstream user they used 10 minute updates with -U for each client, 10 servers with 20 zones each = 200 downloads every 10 minutes = every 3 seconds one download for just one "customer"! > Is timestamping available in our old static wget binaries (those I > distribute with pkgutil as a last resort)? Probably, timestamping is a pretty basic feature based on HTTP header. Best regards -- Dago -- "You don't become great by trying to be great, you become great by wanting to do something, and then doing it so hard that you become great in the process." - xkcd #896
smime.p7s
Description: S/MIME cryptographic signature
