Package: debmirror Version: 1:2.14~bpo60+1 Severity: normal Hi,
At one particular customer, where there is a firewall which blocks just about everything, I need to go through a somewhat paranoid proxy. Because of the way this proxy is set up, the chain has to be something like this: putty SSH tunnel to server running Debian -> cntlm running on Windows machine -> active directory-authenticated proxy server running in another country -> (possibly more things) -> public Debian mirror I run debmirror like so: debmirror --section=main,contrib --ignore-release-gpg -p --host=ftp.uk.debian.org --method=http --proxy=http://localhost:3128/ --dist=squeeze --arch=amd64 --nosource --nocleanup --progress --ignore-small-errors --exclude=clamav-testfiles --rsync-extra=none <directory> (later on it's also run without the nocleanup, and with some other options added, but that's irrelevant for this bug report) This works for the most part, except for one detail: Something in the above chain (likely the proxy server on which I have no access) seems to be configured to abort large downloads (starting at files of somewhere between 50 and 60MiB). Repeating the download as a simple HTTP GET at that point doesn't fix it; the proxy just notices it has an object from the requested URL already in its cache, and will happily produce it -- truncating the download at the exact same place. However, if the download is then repeated with a Range: header so that it only requests the missing parts of the file, the proxy will behave correctly and produce the missing parts. Unfortunately, debmirror doesn't even try to do that. If a download is truncated, the already-downloaded file range is removed(!) and the mirror run is aborted. When this happens, I have to use wget to download the problematic file manually (which will get the truncated part with a normal GET, notice the file isn't complete, and send another GET request with a Range: header so that the file is completed, which then (usually) succeeds. I then need to restart debmirror, where it will continue on for a few more files, until it encounters another file over 50-60MiB where it will fail again. It would be great if debmirror could retry downloading if files are truncated. Alternatively, if that's too complex to implement, it would be okay if there was an option for me to have it soldier on downloading even in the face of truncated downloads, possibly with a file containing errors that I can then inspect (and possibly retry downloading them manually with wget or something). Thanks, -- This end should point toward the ground if you want to go to space. If it starts pointing toward space you are having a bad problem and you will not go to space today. -- http://xkcd.com/1133/ -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
