wget 1.8.x proxies

2002-02-12 Thread Holger Pfaff

Hi!

I'm having trouble using wget 1.8.[01] over a (squid24-) proxy
to mirror a ftp-directory:

# setenv ftp_proxy http://139.21.68.25:
# wget181 -r -np -l0 ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates

--12:06:58--  ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates
= `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates'
Connecting to 139.21.68.25:... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]

 [ = ] 3,665  3.50M/s

12:06:58 (3.50 MB/s) - `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates' 
saved [3665]


FINISHED --12:06:58--
Downloaded: 3,665 bytes in 1 files

--
It does not seem to go into recursion for some reason ...
Version 1.7.1 however does go into recursion but does not honor
the -np switch and traverses upwards:


# wget171 -r -np -l0 ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates
--12:25:05--  ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates
= `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates.1'
Connecting to 139.21.68.25:... connected!
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]

 0K ...@   3.50 MB/s

12:25:05 (3.50 MB/s) - `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates.1' 
saved [3665]

--12:25:05--  
ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates/internal-gopher-menu
= 
`ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates/internal-gopher-menu'
Connecting to 139.21.68.25:... connected!
Proxy request sent, awaiting response... 200 OK
Length: 181 [image/gif]

 0K   100% @ 176.76 KB/s

12:25:05 (176.76 KB/s) - 
`ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates/internal-gopher-menu' 
saved [181/181]

--12:25:05--  ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/
= `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/index.html'
Connecting to 139.21.68.25:... connected!
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]

 0K ...@ 447.54 KB/s

12:25:07 (447.54 KB/s) - 
`ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/index.html' saved [3208]

--12:25:07--  ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/6.2/
= `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/6.2/index.html'
Connecting to 139.21.68.25:... connected!

.
.
.
.

Am I missing something here? Or is this just not possible via proxies?

Thanx in advance for any hints


-- Holger





Re: wget 1.8.x proxies

2002-02-12 Thread Winston Smith

 # wget181 -r -np -l0
 ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates

ummm... looks like the -l0 might be limiting your recursion level to 0
levels





Re: wget 1.8.x proxies

2002-02-12 Thread Ian Abbott

On 12 Feb 2002 at 12:30, Holger Pfaff wrote:

 I'm having trouble using wget 1.8.[01] over a (squid24-) proxy
 to mirror a ftp-directory:
 
 # setenv ftp_proxy http://139.21.68.25:
 # wget181 -r -np -l0 ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates
 
 --12:06:58--  ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates
 = `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates'
 Connecting to 139.21.68.25:... connected.
 Proxy request sent, awaiting response... 200 OK
 Length: unspecified [text/html]
 
  [ = ] 3,665  3.50M/s
 
 12:06:58 (3.50 MB/s) - `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates' 
saved [3665]

I've never tried wget through an http-based ftp proxy.

Are there any clues in the file it wrote (presumably a html-format
directory listing)?

Are there any more clues if you use the -d (--debug) option?




Re: wget 1.8.x proxies

2002-02-12 Thread Ian Abbott

On 12 Feb 2002 at 7:54, Winston Smith wrote:

  # wget181 -r -np -l0
  ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates
 
 ummm... looks like the -l0 might be limiting your recursion level to 0
 levels

No. '-l0' is the same as '-l inf'.




Re: wget 1.8.x proxies

2002-02-12 Thread Holger Pfaff

 I've never tried wget through an http-based ftp proxy.
 
 Are there any clues in the file it wrote (presumably a html-format
 directory listing)?
 
 Are there any more clues if you use the -d (--debug) option?



# wget181 -r -np -l8 --debug 
ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates
DEBUG output created by Wget 1.8.1 on solaris2.8.

--07:28:38--  ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates
= `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates'
Connecting to 139.21.68.25:... connected.
Created socket 3.
Releasing 4d0f8 (new refcount 0).
Deleting unused 4d0f8.
---request begin---
GET ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates HTTP/1.0
User-Agent: Wget/1.8.1
Host: ftp.funet.fi
Accept: */*

---request end---
Proxy request sent, awaiting response... HTTP/1.0 200 OK
Mime-Version: 1.0
Server: NetCache NetApp Release 3.4R1: Tue Nov 9 13:21:00 PST 1999-Solaris
Content-Type: text/html
X-Cache: MISS from s1aa56a
Proxy-Connection: close


Length: unspecified [text/html]

 [ = 
   ] 3,665  3.50M/s

Closing fd 3
07:28:38 (3.50 MB/s) - `ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates' 
saved [3665]


FINISHED --07:28:38--
Downloaded: 3,665 bytes in 1 files





# cat ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates
HTMLBASE HREF=ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates/;
H4Message from ftp.funet.fi:/H4
PRE
Guest `[EMAIL PROTECTED]' login ok.

Finnish University and Research network FUNET
Archive FTP.FUNET.FI

Most important file name suffixes are described at  /README.FILETYPES

Welcome, you are 114th archive user in your class (max 250).
Your class is named: The known world outside NORDUnet region
There are 149 users in all classes (max 687)
Your data-transfer rate has no limitations.

Local time is Mon Feb 11 17:05:32 2002 EET

We have special access features, see file README
   It was last updated Thu Oct 11 19:48:59 2001 - 122.9 days ago
/PRE
HR
!-- HTML listing generated by NetCache NetApp Release 3.4R1: Tue Nov 9 13:21:00 PST 
1999-Solaris --
!-- Mon, 11 Feb 2002 15:02:29 GMT --
HEADTITLE
FTP Directory: ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates/
/TITLE/HEAD
H2
FTP Directory: ftp://ftp.funet.fi/pub/Linux/mirrors/redhat/redhat/linux/updates/
/H2
PRE
IMG BORDER=0 SRC=internal-gopher-menu ALT=[DIR]   A HREF=../Parent 
Directory/A
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=1.0/1.0/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=1.1/1.1/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=2.0/2.0/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=2.1/2.1/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=3.0.3/3.0.3/A . . . . . . 
. . . . . . . 
   [Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=4.0/4.0/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=4.1/4.1/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=4.2/4.2/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=5.0/5.0/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=5.1/5.1/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=5.2/5.2/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=6.0/6.0/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=6.1/6.1/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=6.2/6.2/A . . . . . . . . 
. . . . . . 
[Oct 22 01:56]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=6.2EE/6.2EE/A . . . . . . 
. . . . . . . 
   [Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=7.0/7.0/A . . . . . . . . 
. . . . . . 
[Mar 29  2001]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=7.1/7.1/A . . . . . . . . 
. . . . . . 
[Nov 13 01:32]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=7.2/7.2/A . . . . . . . . 
. . . . . . 
[Jan 22 22:07]
IMG SRC=internal-gopher-binary ALT=[LINK]  A HREF=currentcurrent/A . . . . 
. . . . . . . 
.   [Oct 23 00:47]
IMG SRC=internal-gopher-menu ALT=[DIR]   A HREF=other_prod/other_prod/A. . 
. . . . . . . 
. .   [Mar 29  2001]
/PRE
HR
ADDRESS
Generated Mon, 11 Feb 2002 15:02:29 GMT, by NetCache NetApp Release 3.4R1: Tue Nov 9 
13:21:00 PST 
[EMAIL PROTECTED]
/ADDRESS
/BODY/HTML




Looks all pretty normal to me. Just to clarify: We are using a proxy cascade here.
The first proxy is a squid24 the last one is a NetCache, but since it is working
with older wget versions ...


-- Holger