What would it take to establish a mirror of cdimage.debian.org in North America?
Fetching a DVD image across the North Atlantic takes a long time -- 5-7 hours. And the bit-torrent seeders don't provide the beta images, let alone the daily or weekly images. So... are there any volunteers out there to provide a mirror of cdimage.debian.org/cdimage on this side of the pond? How big a machine would it require? How much bandwidth? Thanks! Rick -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/caonepfb8qz2rbjjiwics0har-rj0d2xzabgh9obmwxr56n8...@mail.gmail.com
Re: What would it take to establish a mirror of cdimage.debian.org in North America?
Hi Rick, Rick Thomas rbtf...@gmail.com writes: Fetching a DVD image across the North Atlantic takes a long time -- 5-7 hours. And the bit-torrent seeders don't provide the beta images, let alone the daily or weekly images. This is not the answer to the question asked, but if you're downloading a daily image over HTTP or FTP then you're almost certainly doing it wrong. If you regularly play with these images then you've probably got a fairly recent one laying around, in which case you could use rsync for a much faster download, but if you want to be kind to cdimage.debian.org the right thing to do is use jigdo, which will pull almost all of the image from your nearest mirror, in form of packages, and then assemble them into the image locally. If you have an old DVD image, you can mount that and offer it to jigdo as a source of packages, saving even more bandwidth. So... are there any volunteers out there to provide a mirror of cdimage.debian.org/cdimage on this side of the pond? How big a machine would it require? How much bandwidth? That said, I'd imagine that a US-ian mirror would be helpful to people that cannot be bothered to work out how to do the right thing. It is possible that one could be smarter than simply mirroring the whole lot, since for some of the images it may be that you find that nobody ever tries to download them from you, so the act of mirroring them would simply add load to cdimage.debian.org. Steve McIntyre proposed a while ago (and I think did some work on) a FUSE file system that would take jigdo files and a local mirror, and provide the illusion of a cdimage mirror without needing the images locally. If you could implement that then any mirror willing to run that FUSE filesystem could provide the content you're looking to mirror, without consuming the bandwidth or disk space required to really do that. Cheers, Phil. -- |)| Philip Hands [+44 (0)20 8530 9560]http://www.hands.com/ |-| HANDS.COM Ltd.http://www.uk.debian.org/ |(| 10 Onslow Gardens, South Woodford, London E18 1NE ENGLAND pgpIQtneuYnax.pgp Description: PGP signature
Re: What would it take to establish a mirror of cdimage.debian.org in North America?
On Sat, 27 Oct 2012, Rick Thomas wrote: Fetching a DVD image across the North Atlantic takes a long time -- 5-7 hours. That's not universal, you should look into why. Either you don't have big enough tcp buffers or you have packet loss somewhere along the path. There is no reason a transatlantic download should take more than half an hour if you have the bandwith to support it locally. And the bit-torrent seeders don't provide the beta images, let alone the daily or weekly images. Bittorrent only gets faster than http when there are plenty of other users on the same torrent (or the http source is bandwidth starved). For weekly builds I'm not convinced that's the case - also the extra day or two latency to sync the images to the torrent seeder hurts on something that updates weekly. So... are there any volunteers out there to provide a mirror of cdimage.debian.org/cdimage on this side of the pond? How big a machine would it require? How much bandwidth? Weekly-builds are a few TB/week. Assuming the mirror builds them from jigdos it should be a reasonable idea. /Mattias Wadenstein -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/alpine.gso.2.02.1210271225450.29...@montezuma.acc.umu.se
Re: What would it take to establish a mirror of cdimage.debian.org in North America?
Thanks Philip, for the thoughtful reply. Of course, jigdo is the correct way to do it. But jigdo is complex to use (I've done it, but I couldn't explain it to my nephew when he wanted to do it.) And jigdo outputs to standard-out lots of minimally informative stuff while it's working, unless you shut it up, then it's *too* quiet... Perhaps the right thing to do here is to go to work on Steve McIntyre's fuse thingie and get it ready for prime-time. Steve? Is it in a shape that somebody could look at it? Rick On Oct 27, 2012, at 2:49 AM, Philip Hands wrote: Hi Rick, Rick Thomas rbtf...@gmail.com writes: Fetching a DVD image across the North Atlantic takes a long time -- 5-7 hours. And the bit-torrent seeders don't provide the beta images, let alone the daily or weekly images. This is not the answer to the question asked, but if you're downloading a daily image over HTTP or FTP then you're almost certainly doing it wrong. If you regularly play with these images then you've probably got a fairly recent one laying around, in which case you could use rsync for a much faster download, but if you want to be kind to cdimage.debian.org the right thing to do is use jigdo, which will pull almost all of the image from your nearest mirror, in form of packages, and then assemble them into the image locally. If you have an old DVD image, you can mount that and offer it to jigdo as a source of packages, saving even more bandwidth. So... are there any volunteers out there to provide a mirror of cdimage.debian.org/cdimage on this side of the pond? How big a machine would it require? How much bandwidth? That said, I'd imagine that a US-ian mirror would be helpful to people that cannot be bothered to work out how to do the right thing. It is possible that one could be smarter than simply mirroring the whole lot, since for some of the images it may be that you find that nobody ever tries to download them from you, so the act of mirroring them would simply add load to cdimage.debian.org. Steve McIntyre proposed a while ago (and I think did some work on) a FUSE file system that would take jigdo files and a local mirror, and provide the illusion of a cdimage mirror without needing the images locally. If you could implement that then any mirror willing to run that FUSE filesystem could provide the content you're looking to mirror, without consuming the bandwidth or disk space required to really do that. Cheers, Phil. -- |)| Philip Hands [+44 (0)20 8530 9560]http://www.hands.com/ |-| HANDS.COM Ltd.http://www.uk.debian.org/ |(| 10 Onslow Gardens, South Woodford, London E18 1NE ENGLAND -- To UNSUBSCRIBE, email to debian-cd-requ...@lists.debian.org with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org Archive: http://lists.debian.org/80c7f7eb-2561-4b43-b158-2a7c23621...@pobox.com
