Re: AW: AW: AW: Problem mirroring a site using ftp over proxy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Juon, Stefan wrote: > Well, here is the index.html (I'm not sure wheter is also accessible in the > maillist as I send it as attachement?) Sorry, I somehow failed to notice this post. :\ The index.html file that the proxy generated is invalid. Apparently it wants to tack on ^M (carriage return, \r) after every filename, as a literal part of the link. It looks like Wget doesn't even acknowledge links like that; but even if it did, it'd send a request to the proxy like: GET /CommonUpdater/avvdat-.zip%0D rather than GET /CommonUpdater/avvdat-.zip so it would still most likely fail to get a real file (though it _might_ work, if the proxy and/or the FTP server are a little sloppy). One likely explanation for this, seems to me, is that the proxy gets back the LIST response like: foo CR LF bar CR LF and removes the LFs while leaving in the CR, and spitting them out as part of the link. That's really poor behavior, considering that FTP servers _ought_ to send CR LF (and not bare LF), as it's supposed to use "telnet conventions". - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIohiL7M8hyUobTrERApkmAJ9Ia9yvahBPtp0aJDZehKciEMc3vQCgjXSC T9DYFPDUxtBEx6HvOnwBzos= =MAXZ -END PGP SIGNATURE-
Re: AW: AW: Problem mirroring a site using ftp over proxy
"Juon, Stefan" <[EMAIL PROTECTED]> writes: > I just noticed these debug messages: > > ** > DEBUG output created by Wget 1.10.2 on cygwin. You are of course aware that this is not the latest Wget (1.11.4)? As mentioned before, recursive download over FTP proxy was broken prior to Wget 1.11. > The point is that wget sends rather a http request than a pure ftp > command That's how proxying FTP normally works.
AW: AW: AW: Problem mirroring a site using ftp over proxy
Well, here is the index.html (I'm not sure wheter is also accessible in the maillist as I send it as attachement?) -Ursprüngliche Nachricht- Von: Micah Cowan [mailto:[EMAIL PROTECTED] Gesendet: Freitag, 8. August 2008 21:11 An: Juon, Stefan Cc: wget@sunsite.dk Betreff: Re: AW: AW: Problem mirroring a site using ftp over proxy -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Juon, Stefan wrote: > The point is that wget sends rather a http request than a pure ftp > command (GET ftp://ftpde.nai.com/CommonUpdater/ HTTP/1.0) which causes > the proxy to send back a index.html. Do u agree? Well of course it does: it's using an HTTP proxy. How do you send FTP commands over HTTP? The problem isn't that the result is an HTML file; the problem is that the proxy sends an HTML file that Wget apparently can't parse. Perhaps the proxy's not really sending an HTML file at all, which would be unusual (but I'm not sure there are standards governing how FTP gets proxied across HTTP), in which case Wget would need to be modified to check whether the proxied results are a listing file. But until you show us what index.html file Wget is getting, I don't see how we can help. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFInJpC7M8hyUobTrERAhGtAJ9/cY3nJk8xf1oWb+KCH8mQ54nXNACgg/is xD3eHrajIfnUDaRhnFI+X+s= =g1QP -END PGP SIGNATURE- Title: Directory of /CommonUpdater/ Current directory is /CommonUpdater/ ..53425343.upd 488132Fri Aug 8 06:20:00 2008 53425343avv.gem 249132Fri Aug 8 06:20:00 2008 53435344.upd 1007181Fri Aug 8 06:20:00 2008 53435344avv.gem 719148Fri Aug 8 06:20:00 2008 53445345.upd 647667Fri Aug 8 06:20:00 2008 53445345avv.gem 395556Fri Aug 8 06:20:00 2008 53455346.upd 435721Fri Aug 8 06:20:00 2008 53455346avv.gem 205412Fri Aug 8 06:20:00 2008 53465347.upd 416621Fri Aug 8 06:20:00 2008 53465347avv.gem 190396Fri Aug 8 06:20:00 2008 53475348.upd 379599Fri Aug 8 06:20:00 2008 53475348avv.gem 145164Fri Aug 8 06:20:00 2008 53485349.upd 591065Fri Aug 8 06:20:00 2008 53485349avv.gem 346116Fri Aug 8 06:20:00 2008 53495350.upd 529113Fri Aug 8 06:20:00 2008 53495350avv.gem 291500Fri Aug 8 06:20:00 2008 53505351.upd 573233Fri Aug 8 06:20:00 2008 53505351avv.gem 341996Fri Aug 8 06:20:00 2008 53515352.upd 492563Fri Aug 8 06:20:00 2008 53515352avv.gem 276548Fri Aug 8 06:20:00 2008 53525353.upd 786499Fri Aug 8 06:20:00 2008 53525353avv.gem 1005388Fri Aug 8 06:20:00 2008 53535354.upd 516762Fri Aug 8 06:20:00 2008 53535354avv.gem 327140Fri Aug 8 06:20:00 2008 53545355.upd 340941Fri Aug 8 06:20:00 2008 53545355avv.gem 146724Fri Aug 8 06:20:00 2008 53555356.upd 284106Fri Aug 8 06:20:00 2008 53555356avv.gem 97148Fri Aug 8 06:20:00 2008 53565357.upd 231339Fri Aug 8 06:20:00 2008 53565357avv.gem 70244Fri Aug 8 06:20:00 2008 avvdat-5357.zip 29893719Fri Aug 8 06:20:00 2008 catalog.z 3052Fri Aug 8 06:23:00 2008 ceu.ini 54Fri Aug 8 06:21:00 2008 Current Tue Jul 1 11:29:00 2008 dat-5357.zip 36336239Fri Aug 8 06:20:00 2008 DATInstall.mcs 49772Fri Aug 8 06:20:00 2008 delta.ini 1342Fri Aug 8 06:21:00 2008 gdeltaavv.ini 1062Fri Aug 8 06:21:00 2008 oem.ini 1031Fri Aug 8 07:14:00 2008 Replica.log 4142Fri Aug 8 06:23:00 2008 SiteStat.xml 118Fri Aug 8 06:23:00 2008 update.ini 1286Fri Aug 8 06:21:00 2008 V2datdet.mcs 80500Fri Aug 8 06:21:00 2008 V2datInstall.mcs 83700Fri Aug 8 06:21:00 2008
Re: AW: AW: Problem mirroring a site using ftp over proxy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Juon, Stefan wrote: > The point is that wget sends rather a http request than a pure ftp > command (GET ftp://ftpde.nai.com/CommonUpdater/ HTTP/1.0) which > causes the proxy to send back a index.html. Do u agree? Well of course it does: it's using an HTTP proxy. How do you send FTP commands over HTTP? The problem isn't that the result is an HTML file; the problem is that the proxy sends an HTML file that Wget apparently can't parse. Perhaps the proxy's not really sending an HTML file at all, which would be unusual (but I'm not sure there are standards governing how FTP gets proxied across HTTP), in which case Wget would need to be modified to check whether the proxied results are a listing file. But until you show us what index.html file Wget is getting, I don't see how we can help. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFInJpC7M8hyUobTrERAhGtAJ9/cY3nJk8xf1oWb+KCH8mQ54nXNACgg/is xD3eHrajIfnUDaRhnFI+X+s= =g1QP -END PGP SIGNATURE-
AW: AW: Problem mirroring a site using ftp over proxy
I just noticed these debug messages: ** DEBUG output created by Wget 1.10.2 on cygwin. --15:49:45-- ftp://ftpde.nai.com/CommonUpdater/ => `ftpde.nai.com/CommonUpdater/index.html' Resolving * 192.168.182.76 Caching * => 192.168.182.76 Connecting to *|192.168.182.76|:8080... connected. Created socket 4. Releasing 0x006a1390 (new refcount 1). ---request begin--- GET ftp://ftpde.nai.com/CommonUpdater/ HTTP/1.0 User-Agent: Wget/1.10.2 Accept: */* Host: ftpde.nai.com ---request end--- Proxy request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Server: Sun-Java-System-Web-Proxy-Server/4.0 Date: Fri, 08 Aug 2008 13:49:46 GMT Content-type: text/html Proxy-agent: Sun-Java-System-Web-Proxy-Server/4.0 Via: 1.1 proxy-internet Connection: close ---response end--- 200 OK Length: unspecified [text/html] 0K 7.11 MB/s Closed fd 4 Last-modified header missing -- time-stamps turned off. 15:49:47 (7.11 MB/s) - `ftpde.nai.com/CommonUpdater/index.html' saved [8786] FINISHED --15:49:47-- Downloaded: 8,786 bytes in 1 files ** The point is that wget sends rather a http request than a pure ftp command (GET ftp://ftpde.nai.com/CommonUpdater/ HTTP/1.0) which causes the proxy to send back a index.html. Do u agree? -Ursprüngliche Nachricht- Von: Micah Cowan [mailto:[EMAIL PROTECTED] Gesendet: Donnerstag, 7. August 2008 23:23 An: Juon, Stefan Cc: wget@sunsite.dk Betreff: Re: AW: Problem mirroring a site using ftp over proxy -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Well, considering that FTP proxied over HTTP is working fine for me, it's probably more a matter of the index.html file that's generated by the proxy (since one can't do a true LIST over a proxy). Perhaps you could supply the index.html files that are being generated (be sure to clean out any sensitive info first). It might also be informative to know what server program is doing the proxying. - -Micah Juon, Stefan wrote: > ...problem exists also with version 1.11.4. So what might cause wget > not to download the files as it has performed a LIST? > > Thanks, Stefan > > Juon, Stefan wrote: >> Hi there >> I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy >> 4.0.4) > >> using this wget-command: > >> export ftp_proxy=http://proxy.company.com:8080 >> wget --follow-ftp --passive-ftp --proxy=on --mirror >> --output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater > > What version of Wget are you running? If it's not the latest, please > try the current 1.11.4 release. > > Please also try the --debug option, to see if Wget gives you more > information. > - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIm2fF7M8hyUobTrERAv/BAJ9biwIIUFaIWZ9Ds7IZxiGAKriA7wCeJtn1 lYdaP8hzodianPg1Bp6b6gk= =+HQo -END PGP SIGNATURE-
Re: AW: Problem mirroring a site using ftp over proxy
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Well, considering that FTP proxied over HTTP is working fine for me, it's probably more a matter of the index.html file that's generated by the proxy (since one can't do a true LIST over a proxy). Perhaps you could supply the index.html files that are being generated (be sure to clean out any sensitive info first). It might also be informative to know what server program is doing the proxying. - -Micah Juon, Stefan wrote: > ...problem exists also with version 1.11.4. So what might cause wget not > to download the files as it has performed a LIST? > > Thanks, Stefan > > Juon, Stefan wrote: >> Hi there >> I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy 4.0.4) > >> using this wget-command: > >> export ftp_proxy=http://proxy.company.com:8080 >> wget --follow-ftp --passive-ftp --proxy=on --mirror >> --output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater > > What version of Wget are you running? If it's not the latest, please try > the current 1.11.4 release. > > Please also try the --debug option, to see if Wget gives you more > information. > - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFIm2fF7M8hyUobTrERAv/BAJ9biwIIUFaIWZ9Ds7IZxiGAKriA7wCeJtn1 lYdaP8hzodianPg1Bp6b6gk= =+HQo -END PGP SIGNATURE-
AW: Problem mirroring a site using ftp over proxy
...problem exists also with version 1.11.4. So what might cause wget not to download the files as it has performed a LIST? Thanks, Stefan -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Juon, Stefan wrote: > Hi there > I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy 4.0.4) > using this wget-command: > > export ftp_proxy=http://proxy.company.com:8080 > wget --follow-ftp --passive-ftp --proxy=on --mirror > --output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater What version of Wget are you running? If it's not the latest, please try the current 1.11.4 release. Please also try the --debug option, to see if Wget gives you more information. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFImVZ77M8hyUobTrERAgS7AJ4lWgDuBJonnms+gkriGTZ7LlA4TwCfeNqo jOtcPq60sVWXb9CA1n6FSnI= =Z/D4 -END PGP SIGNATURE-
AW: Problem mirroring a site using ftp over proxy
Version is 1.10.2 which comes with cygwin. Option --debug does not give any more information. I will go on compiling the latest version and run it again. -Ursprüngliche Nachricht- Von: Micah Cowan [mailto:[EMAIL PROTECTED] Gesendet: Mittwoch, 6. August 2008 09:45 An: Juon, Stefan Cc: wget@sunsite.dk Betreff: Re: Problem mirroring a site using ftp over proxy -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Juon, Stefan wrote: > Hi there > I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy 4.0.4) > using this wget-command: > > export ftp_proxy=http://proxy.company.com:8080 > wget --follow-ftp --passive-ftp --proxy=on --mirror > --output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater What version of Wget are you running? If it's not the latest, please try the current 1.11.4 release. Please also try the --debug option, to see if Wget gives you more information. - -- Micah J. Cowan Programmer, musician, typesetting enthusiast, gamer. GNU Maintainer: wget, screen, teseq http://micah.cowan.name/ -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFImVZ77M8hyUobTrERAgS7AJ4lWgDuBJonnms+gkriGTZ7LlA4TwCfeNqo jOtcPq60sVWXb9CA1n6FSnI= =Z/D4 -END PGP SIGNATURE-
