How to mirror Debian across a firewall
Hello, I want to have a local Debian mirror but to do this I have to go through our firewall. I'm looking at rsync and mirror. The documentation of the former doesn't mention proxies at all and that of the latter suggests defining the variables proxy=true, proxy_ftp_port= and proxy_gateway=gateway_name. However I've been unable to get mirror through the firewall here. The environment variable ftp_proxy and http_proxy work with other packages like lynx and apt but neither rsync or mirror seem to care about them. I'd appreciate any suggestions. -- Pedro
Re: How to mirror Debian across a firewall
Pedro Sanchez [EMAIL PROTECTED] writes: I want to have a local Debian mirror but to do this I have to go through our firewall. I'm looking at rsync and mirror. The documentation of the former doesn't mention proxies at all and that of the latter suggests defining the variables proxy=true, proxy_ftp_port= and proxy_gateway=gateway_name. However I've been unable to get mirror through the firewall here. The environment variable ftp_proxy and http_proxy work with other packages like lynx and apt but neither rsync or mirror seem to care about them. I'd appreciate any suggestions. Are you planning on mirroring all the debian distributions (slink, potato, sid) for every architecture, eg., alpha, sparc, i386, etc.? If so then, once you figure out your ftp proxy, it'll work fine. If, on the other hand, you're going to try and mirror only one distribution, and only one architecture within that distribution your going to be in for a wild ride trying to do it with ftp! Most of the Debian mirrors use the proftpd server and it refuses to flatten symlinks. This is a problem, for example, if you want the potato distribution, because there are quite a few packages in potato that are merely symlinks to packages in slink. If you were just doing slink (don't know about sid) this wouldn't be a problem, but for potato it's a headache. Anyway, I gave up on the ftp solution and just started using w3mir. Other than a small bug that causes files with a + in their names to always be deleted it works fine and there aren't symlinks, at least visible ones, on the http mirrors, so it's easy to get all the files needed for a particular architecture in a particular distribution. It also has handled our http proxy without a hitch. A simple command-line argument was all that was needed. I think it's a bit more inefficient than an ftp mirror, but for partial mirrors it's about the only solution. Good luck, Gary
Re: How to mirror Debian across a firewall
Take a look at the apt-move package in potato... it should do just about what you want using rsync (you'll need to upgrade rsync if you're running slink). The current version (3.0-7) has a few problems, but a fixed version should be uploaded either tomorrow or Saturday (I'm the maintainer :-). Information on proxies will be included in the upcoming version's README.Debian file. On Thu, Nov 18, 1999 at 09:21:46PM +, Pedro Sanchez wrote: Hello, I want to have a local Debian mirror but to do this I have to go through our firewall. I'm looking at rsync and mirror. The documentation of the former doesn't mention proxies at all and that of the latter suggests defining the variables proxy=true, proxy_ftp_port= and proxy_gateway=gateway_name. However I've been unable to get mirror through the firewall here. The environment variable ftp_proxy and http_proxy work with other packages like lynx and apt but neither rsync or mirror seem to care about them.
Re: How to mirror Debian across a firewall
Gary Hennigan wrote: Pedro Sanchez [EMAIL PROTECTED] writes: I want to have a local Debian mirror but to do this I have to go through our firewall. I'm looking at rsync and mirror. The documentation of Are you planning on mirroring all the debian distributions (slink, potato, sid) for every architecture, eg., alpha, sparc, i386, etc.? If so then, once you figure out your ftp proxy, it'll work fine. If, on the other hand, you're going to try and mirror only one distribution, and only one architecture within that distribution your going to be in for a wild ride trying to do it with ftp! Most of the Debian mirrors use the proftpd server and it refuses to flatten symlinks. This is a problem, for example, if you want the potato distribution, because there are quite a few packages in potato that are merely symlinks to packages in slink. If you were just doing slink (don't know about sid) this wouldn't be a problem, but for potato it's a headache. Anyway, I gave up on the ftp solution and just started using w3mir. Other than a small bug that causes files with a + in their names to always be deleted it works fine and there aren't symlinks, at least visible ones, on the http mirrors, so it's easy to get all the files needed for a particular architecture in a particular distribution. It also has handled our http proxy without a hitch. A simple command-line argument was all that was needed. I think it's a bit more inefficient than an ftp mirror, but for partial mirrors it's about the only solution. I'm using wget to maintain a local mirror for my servers to update off and do local installs for students faculty. --cut here-- #!/bin/sh cd /where/you/want/the/mirror echo debian.log /usr/bin/wget --mirror -nH -c --cut-dirs=1 -b -o debian.log --dot-style=binary --tries=10 ftp://ftp.us.debian.org/debian/dists/stable/main/disks-i386 ftp://ftp.us.debian.org/debian/dists/stable/main/binary-i386/ ftp://ftp.us.debian.org/debian/dists/stable/main/binary-all/ ftp://ftp.us.debian.org/debian/dists/stable/contrib/binary-i386/ ftp://ftp.us.debian.org/debian/dists/stable/contrib/binary-all/ ftp://ftp.us.debian.org/debian/dists/stable/non-free/binary-i386/ ftp://ftp.us.debian.org/debian/dists/stable/non-free/binary-all/ --cut here-- All the ftp urls go on the same line as wget if your mailer wrapped them to new lines. You can set your proxy in /etc/wgetrc jpb -- Joe Block [EMAIL PROTECTED] CREOL System Administrator Social graces are the packet headers of everyday life.
