wget 1.8x does not mirror FTP servers using an HTTP proxy

[Bernhard Erdmann]

Hi,

I discovered wget starting with version 1.8 does not mirror FTP servers properly using Squid as an HTTP proxy.

wget -m -np ftp://ftp.gnu.org/ using no proxy works as expected.

wget starting with version 1.8 just saves the index.html generated by Squid but no further files and no directory hierarchy when directed to use Squid for ftp_proxy (speaking HTTP to it).

wget version 1.5.3 up to and including 1.7.1 is ok.

It happens as well using Squid 2.4STABLE6, Squid 2.4STABLE7 and Squid 2.5STABLE1. The OS is Linux.

I was used to mirror (parts of) FTP servers with wget using Squid. I consider this being a bug since wget 1.8.

You can review my test case

wget --delete-after -m -np ftp://ftp.gnu.org/gnu/wget/

using the seven different versions of wget available from ftp://ftp.gnu.org/gnu/wget/ viewing the URL

http://berdmann.dyndns.org/debug/wget/ .

Basically, the script test-all uses the different versions of wget to download ftp://ftp.gnu.org/gnu/wget/ and saves its output in test-wget-$versionnumber. Using or not using --delete-after does not make any difference to the downloaded URLs.

wget in the test case above did use /usr/local/etc/wgetrc. This file is available as well. A ~/.wgetrc did not exist.

The relevant lines of wgetrc are:
timestamping = on
http_proxy = http://www-cache:3128/
ftp_proxy = http://www-cache:3128/

Regards
Bernie