RE: wget 1.8x does not mirror FTP servers using an HTTP proxy

[Micah Hoffman]

Sorry for the length of this post. Some of you offered some helpful things for me to try. First, here's my ~/.wgetrc file:

$ more ~/.wgetrc
continue = on
dirstruct = off
dot_style = mega
glob = on
mirror = on
passive_ftp = on
progress = bar
tries = 3
verbose = on
use_proxy = on
ftp_proxy = webproxy
http_proxy = webproxy

-=-=-=-=-=-=-=-=-
Now, here are the results of the tests:

Jan Hnila suggested I try to do the following:

$ wget -m --no-directories ftp://ftp.gnu.org/pub/gnu/gcc/
--15:00:08-- ftp://ftp.gnu.org/pub/gnu/gcc/
=> `index.html'
Resolving webproxy... done.
Connecting to webproxy:80... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]

[ <=> ] 4,821 2.30M/s
Last-modified header missing -- time-stamps turned off.
15:00:08 (2.30 MB/s) - `index.html' saved [4821]


FINISHED --15:00:08--
Downloaded: 4,821 bytes in 1 files

So wget finishes and I do an "ls" on the directory and find:

$ ls -l
total 8
-rw-r--r-- 1 micah micah 4821 Feb 3 15:00 index.html

This index.html file contains an HTML rendering of the dir contents of ftp.gnu.org/pub/gnu/gcc.

-=-=-=-=-=-=-=-=-=-=-

Next, Harold Heiko suggested I try the following (same as above with no trailing slash or asterik):

$ wget -m --no-directories ftp://ftp.gnu.org/pub/gnu/gcc
--15:00:26-- ftp://ftp.gnu.org/pub/gnu/gcc
=> `gcc'
Resolving webproxy... done.
Connecting to webproxy:80... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]

[ <=>] 4,821 2.30M/s
Last-modified header missing -- time-stamps turned off.
15:00:37 (2.30 MB/s) - `gcc' saved [4821]


FINISHED --15:00:37--
Downloaded: 4,821 bytes in 1 files

Do an "ls" on the dir and find:

$ ls -l
total 16
-rw-r--r-- 1 micah micah 4821 Feb 3 15:00 gcc


This file "gcc" has as its content an HTML rendering of the dir contents of ftp.gnu.org/pub/gnu/gcc (same as the index.html).
Any other ideas? Remember that if I issue the "wget -m ftp://ftp.gnu.org/pub/gnu/gcc/*"; not using the Apache proxy, I get everything in that directory downloaded to the local machine.

--- Micah