Re: Problem with libeay32.dll, ordinal 2253

[Charles]

On Wed, Sep 17, 2008 at 11:02 PM, Tobias Opialla
<[EMAIL PROTECTED]> wrote:
> Hey all,
>
> I hope this is the right adress, and you can help me.
> I'm currently trying to run a perlscript including some wget commands, but if 
> I try to run it, it says:
> "The ordinal 2253 could not be located in the dynamic link library 
> LIBEAY32.dll."

Probably because of dll conflict between the version used by wget and
the version supplied by perl.
You could try renaming libeay32.dll found in perl/bin directory.

Problem with libeay32.dll, ordinal 2253

[Tobias Opialla]

Hey all,

I hope this is the right adress, and you can help me.
I'm currently trying to run a perlscript including some wget commands, but if I 
try to run it, it says:
"The ordinal 2253 could not be located in the dynamic link library 
LIBEAY32.dll."

Any Ideas on that one? I couldn't find anythin on the web.

Regards, Tobias Opialla

Re: AW: AW: AW: Problem mirroring a site using ftp over proxy

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Juon, Stefan wrote:
> Well, here is the index.html (I'm not sure wheter is also accessible in the 
> maillist as I send it as attachement?) 

Sorry, I somehow failed to notice this post. :\

The index.html file that the proxy generated is invalid. Apparently it
wants to tack on ^M (carriage return, \r) after every filename, as a
literal part of the link. It looks like Wget doesn't even acknowledge
links like that; but even if it did, it'd send a request to the proxy like:

  GET /CommonUpdater/avvdat-.zip%0D

rather than

  GET /CommonUpdater/avvdat-.zip

so it would still most likely fail to get a real file (though it _might_
work, if the proxy and/or the FTP server are a little sloppy).

One likely explanation for this, seems to me, is that the proxy gets
back the LIST response like:

  foo CR LF
  bar CR LF

and removes the LFs while leaving in the CR, and spitting them out as
part of the link. That's really poor behavior, considering that FTP
servers _ought_ to send CR LF (and not bare LF), as it's supposed to use
"telnet conventions".

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIohiL7M8hyUobTrERApkmAJ9Ia9yvahBPtp0aJDZehKciEMc3vQCgjXSC
T9DYFPDUxtBEx6HvOnwBzos=
=MAXZ
-END PGP SIGNATURE-

Re: AW: AW: Problem mirroring a site using ftp over proxy

[Hrvoje Niksic]

"Juon, Stefan" <[EMAIL PROTECTED]> writes:

> I just noticed these debug messages:
>
> **
> DEBUG output created by Wget 1.10.2 on cygwin.

You are of course aware that this is not the latest Wget (1.11.4)?
As mentioned before, recursive download over FTP proxy was broken
prior to Wget 1.11.

> The point is that wget sends rather a http request than a pure ftp
> command

That's how proxying FTP normally works.

AW: AW: AW: Problem mirroring a site using ftp over proxy

[Juon, Stefan]

Well, here is the index.html (I'm not sure wheter is also accessible in the 
maillist as I send it as attachement?) 

-Ursprüngliche Nachricht-
Von: Micah Cowan [mailto:[EMAIL PROTECTED] 
Gesendet: Freitag, 8. August 2008 21:11
An: Juon, Stefan
Cc: wget@sunsite.dk
Betreff: Re: AW: AW: Problem mirroring a site using ftp over proxy

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Juon, Stefan wrote:

> The point is that wget sends rather a http request than a pure ftp 
> command (GET ftp://ftpde.nai.com/CommonUpdater/ HTTP/1.0) which causes 
> the proxy to send back a index.html. Do u agree?

Well of course it does: it's using an HTTP proxy. How do you send FTP commands 
over HTTP?

The problem isn't that the result is an HTML file; the problem is that the 
proxy sends an HTML file that Wget apparently can't parse. Perhaps the proxy's 
not really sending an HTML file at all, which would be unusual (but I'm not 
sure there are standards governing how FTP gets proxied across HTTP), in which 
case Wget would need to be modified to check whether the proxied results are a 
listing file. But until you show us what index.html file Wget is getting, I 
don't see how we can help.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFInJpC7M8hyUobTrERAhGtAJ9/cY3nJk8xf1oWb+KCH8mQ54nXNACgg/is
xD3eHrajIfnUDaRhnFI+X+s=
=g1QP
-END PGP SIGNATURE-
Title: Directory of /CommonUpdater/ 

	
	 
		Current directory is /CommonUpdater/
		 ..53425343.upd
  488132Fri Aug  8 06:20:00 2008 
53425343avv.gem
  249132Fri Aug  8 06:20:00 2008 
53435344.upd
  1007181Fri Aug  8 06:20:00 2008 
53435344avv.gem
  719148Fri Aug  8 06:20:00 2008 
53445345.upd
  647667Fri Aug  8 06:20:00 2008 
53445345avv.gem
  395556Fri Aug  8 06:20:00 2008 
53455346.upd
  435721Fri Aug  8 06:20:00 2008 
53455346avv.gem
  205412Fri Aug  8 06:20:00 2008 
53465347.upd
  416621Fri Aug  8 06:20:00 2008 
53465347avv.gem
  190396Fri Aug  8 06:20:00 2008 
53475348.upd
  379599Fri Aug  8 06:20:00 2008 
53475348avv.gem
  145164Fri Aug  8 06:20:00 2008 
53485349.upd
  591065Fri Aug  8 06:20:00 2008 
53485349avv.gem
  346116Fri Aug  8 06:20:00 2008 
53495350.upd
  529113Fri Aug  8 06:20:00 2008 
53495350avv.gem
  291500Fri Aug  8 06:20:00 2008 
53505351.upd
  573233Fri Aug  8 06:20:00 2008 
53505351avv.gem
  341996Fri Aug  8 06:20:00 2008 
53515352.upd
  492563Fri Aug  8 06:20:00 2008 
53515352avv.gem
  276548Fri Aug  8 06:20:00 2008 
53525353.upd
  786499Fri Aug  8 06:20:00 2008 
53525353avv.gem
  1005388Fri Aug  8 06:20:00 2008 
53535354.upd
  516762Fri Aug  8 06:20:00 2008 
53535354avv.gem
  327140Fri Aug  8 06:20:00 2008 
53545355.upd
  340941Fri Aug  8 06:20:00 2008 
53545355avv.gem
  146724Fri Aug  8 06:20:00 2008 
53555356.upd
  284106Fri Aug  8 06:20:00 2008 
53555356avv.gem
  97148Fri Aug  8 06:20:00 2008 
53565357.upd
  231339Fri Aug  8 06:20:00 2008 
53565357avv.gem
  70244Fri Aug  8 06:20:00 2008 
avvdat-5357.zip
  29893719Fri Aug  8 06:20:00 2008 
catalog.z
  3052Fri Aug  8 06:23:00 2008 
ceu.ini
  54Fri Aug  8 06:21:00 2008 
Current
  Tue Jul  1 11:29:00 2008 
dat-5357.zip
  36336239Fri Aug  8 06:20:00 2008 
DATInstall.mcs
  49772Fri Aug  8 06:20:00 2008 
delta.ini
  1342Fri Aug  8 06:21:00 2008 
gdeltaavv.ini
  1062Fri Aug  8 06:21:00 2008 
oem.ini
  1031Fri Aug  8 07:14:00 2008 
Replica.log
  4142Fri Aug  8 06:23:00 2008 
SiteStat.xml
  118Fri Aug  8 06:23:00 2008 
update.ini
  1286Fri Aug  8 06:21:00 2008 
V2datdet.mcs
  80500Fri Aug  8 06:21:00 2008 
V2datInstall.mcs
  83700Fri Aug  8 06:21:00 2008 

		
	

Re: AW: AW: Problem mirroring a site using ftp over proxy

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Juon, Stefan wrote:

> The point is that wget sends rather a http request than a pure ftp
> command (GET ftp://ftpde.nai.com/CommonUpdater/ HTTP/1.0) which
> causes the proxy to send back a index.html. Do u agree?

Well of course it does: it's using an HTTP proxy. How do you send FTP
commands over HTTP?

The problem isn't that the result is an HTML file; the problem is that
the proxy sends an HTML file that Wget apparently can't parse. Perhaps
the proxy's not really sending an HTML file at all, which would be
unusual (but I'm not sure there are standards governing how FTP gets
proxied across HTTP), in which case Wget would need to be modified to
check whether the proxied results are a listing file. But until you show
us what index.html file Wget is getting, I don't see how we can help.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFInJpC7M8hyUobTrERAhGtAJ9/cY3nJk8xf1oWb+KCH8mQ54nXNACgg/is
xD3eHrajIfnUDaRhnFI+X+s=
=g1QP
-END PGP SIGNATURE-

AW: AW: Problem mirroring a site using ftp over proxy

[Juon, Stefan]

I just noticed these debug messages:

**
DEBUG output created by Wget 1.10.2 on cygwin.

--15:49:45--  ftp://ftpde.nai.com/CommonUpdater/
   => `ftpde.nai.com/CommonUpdater/index.html'
Resolving * 192.168.182.76
Caching * => 192.168.182.76
Connecting to *|192.168.182.76|:8080... connected.
Created socket 4.
Releasing 0x006a1390 (new refcount 1).

---request begin---
GET ftp://ftpde.nai.com/CommonUpdater/ HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: ftpde.nai.com

---request end---
Proxy request sent, awaiting response... 
---response begin---
HTTP/1.1 200 OK
Server: Sun-Java-System-Web-Proxy-Server/4.0
Date: Fri, 08 Aug 2008 13:49:46 GMT
Content-type: text/html
Proxy-agent: Sun-Java-System-Web-Proxy-Server/4.0
Via: 1.1 proxy-internet
Connection: close

---response end---
200 OK
Length: unspecified [text/html]

0K   7.11 MB/s

Closed fd 4
Last-modified header missing -- time-stamps turned off.
15:49:47 (7.11 MB/s) - `ftpde.nai.com/CommonUpdater/index.html' saved [8786]


FINISHED --15:49:47--
Downloaded: 8,786 bytes in 1 files
**

The point is that wget sends rather a http request than a pure ftp command (GET 
ftp://ftpde.nai.com/CommonUpdater/ HTTP/1.0) which causes the proxy to send 
back a index.html. Do u agree? 

-Ursprüngliche Nachricht-
Von: Micah Cowan [mailto:[EMAIL PROTECTED] 
Gesendet: Donnerstag, 7. August 2008 23:23
An: Juon, Stefan
Cc: wget@sunsite.dk
Betreff: Re: AW: Problem mirroring a site using ftp over proxy

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Well, considering that FTP proxied over HTTP is working fine for me, it's 
probably more a matter of the index.html file that's generated by the proxy 
(since one can't do a true LIST over a proxy). Perhaps you could supply the 
index.html files that are being generated (be sure to clean out any sensitive 
info first).

It might also be informative to know what server program is doing the proxying.

- -Micah

Juon, Stefan wrote:
> ...problem exists also with version 1.11.4. So what might cause wget 
> not to download the files as it has performed a LIST?
> 
> Thanks, Stefan
> 
> Juon, Stefan wrote:
>> Hi there
>> I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy 
>> 4.0.4)
> 
>> using this wget-command:
> 
>> export ftp_proxy=http://proxy.company.com:8080
>> wget --follow-ftp --passive-ftp --proxy=on --mirror 
>> --output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater
> 
> What version of Wget are you running? If it's not the latest, please 
> try the current 1.11.4 release.
> 
> Please also try the --debug option, to see if Wget gives you more 
> information.
> 

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIm2fF7M8hyUobTrERAv/BAJ9biwIIUFaIWZ9Ds7IZxiGAKriA7wCeJtn1
lYdaP8hzodianPg1Bp6b6gk=
=+HQo
-END PGP SIGNATURE-

Re: AW: Problem mirroring a site using ftp over proxy

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Well, considering that FTP proxied over HTTP is working fine for me,
it's probably more a matter of the index.html file that's generated by
the proxy (since one can't do a true LIST over a proxy). Perhaps you
could supply the index.html files that are being generated (be sure to
clean out any sensitive info first).

It might also be informative to know what server program is doing the
proxying.

- -Micah

Juon, Stefan wrote:
> ...problem exists also with version 1.11.4. So what might cause wget not
> to download the files as it has performed a LIST?
> 
> Thanks, Stefan
> 
> Juon, Stefan wrote:
>> Hi there
>> I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy 4.0.4)
> 
>> using this wget-command:
> 
>> export ftp_proxy=http://proxy.company.com:8080
>> wget --follow-ftp --passive-ftp --proxy=on --mirror 
>> --output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater
> 
> What version of Wget are you running? If it's not the latest, please try
> the current 1.11.4 release.
> 
> Please also try the --debug option, to see if Wget gives you more
> information.
> 

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIm2fF7M8hyUobTrERAv/BAJ9biwIIUFaIWZ9Ds7IZxiGAKriA7wCeJtn1
lYdaP8hzodianPg1Bp6b6gk=
=+HQo
-END PGP SIGNATURE-

AW: Problem mirroring a site using ftp over proxy

[Juon, Stefan]

...problem exists also with version 1.11.4. So what might cause wget not
to download the files as it has performed a LIST?

Thanks, Stefan

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Juon, Stefan wrote:
> Hi there
> I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy 4.0.4)

> using this wget-command:
>  
> export ftp_proxy=http://proxy.company.com:8080
> wget --follow-ftp --passive-ftp --proxy=on --mirror 
> --output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater

What version of Wget are you running? If it's not the latest, please try
the current 1.11.4 release.

Please also try the --debug option, to see if Wget gives you more
information.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFImVZ77M8hyUobTrERAgS7AJ4lWgDuBJonnms+gkriGTZ7LlA4TwCfeNqo
jOtcPq60sVWXb9CA1n6FSnI=
=Z/D4
-END PGP SIGNATURE-

AW: Problem mirroring a site using ftp over proxy

[Juon, Stefan]

Version is 1.10.2 which comes with cygwin. Option --debug does not give any 
more information.
I will go on compiling the latest version and run it again. 

-Ursprüngliche Nachricht-
Von: Micah Cowan [mailto:[EMAIL PROTECTED] 
Gesendet: Mittwoch, 6. August 2008 09:45
An: Juon, Stefan
Cc: wget@sunsite.dk
Betreff: Re: Problem mirroring a site using ftp over proxy

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Juon, Stefan wrote:
> Hi there
> I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy 4.0.4) 
> using this wget-command:
>  
> export ftp_proxy=http://proxy.company.com:8080
> wget --follow-ftp --passive-ftp --proxy=on --mirror 
> --output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater

What version of Wget are you running? If it's not the latest, please try the 
current 1.11.4 release.

Please also try the --debug option, to see if Wget gives you more information.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFImVZ77M8hyUobTrERAgS7AJ4lWgDuBJonnms+gkriGTZ7LlA4TwCfeNqo
jOtcPq60sVWXb9CA1n6FSnI=
=Z/D4
-END PGP SIGNATURE-

Re: Problem mirroring a site using ftp over proxy

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Juon, Stefan wrote:
> Hi there
> I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy 4.0.4)
> using this wget-command:
>  
> export ftp_proxy=http://proxy.company.com:8080
> wget --follow-ftp --passive-ftp --proxy=on --mirror
> --output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater

What version of Wget are you running? If it's not the latest, please try
the current 1.11.4 release.

Please also try the --debug option, to see if Wget gives you more
information.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
GNU Maintainer: wget, screen, teseq
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFImVZ77M8hyUobTrERAgS7AJ4lWgDuBJonnms+gkriGTZ7LlA4TwCfeNqo
jOtcPq60sVWXb9CA1n6FSnI=
=Z/D4
-END PGP SIGNATURE-

Problem mirroring a site using ftp over proxy

[Juon, Stefan]

Hi there
I'm trying to mirror a ftp site over a proxy (Sun Java Webproxy 4.0.4)
using this wget-command:
 
export ftp_proxy=http://proxy.company.com:8080
wget --follow-ftp --passive-ftp --proxy=on --mirror
--output-file=./logfile.wget ftp://ftpde.nai.com/CommonUpdater
 
The result is that I get a file ftpde.nai.com/CommonUpdater/index.html
instead of the mirrored data. The index.html is the same as I get if I
connect using a browser.
 
Tracing the traffic on the proxy I see this (I snipped just the
payload):
 
220 spftp/1.0. Server [213.200.104.32]
USER anonymous
331 Password required for USER.
PASS wget@
230- 
230-

---
230- WARNING:  This is a restricted access system.  If you do not have
explicit
230-   permission to access this system, please disconnect
immediately!
230


SYST
215 UNIX 
PASV
227 Entering Passive Mode. (213,200,104,32,193,166)
TYPE I
200 TYPE set to I.
CWD /CommonUpdater/
250 CWD command successful.
LIST
150 Opening ASCII mode data connection for /commonupdater.
226 Transfer Complete
 
and nothing else. Doing the wget without a proxy I see this traffic:
 
220 spftp/1.0. Server [213.200.104.29]
USER anonymous
331 Password required for USER.
PASS -wget@
230- 
230-

---
230- WARNING:  This is a restricted access system.  If you do not have
explicit
230-   permission to access this system, please disconnect
immediately!
230


SYST
215 UNIX 
PWD
257 "/" is current directory.
TYPE I
200 TYPE set to I.
CWD /CommonUpdater
250 CWD command successful.
PASV
227 Entering Passive Mode. (213,200,104,29,203,13)
LIST
150 Opening ASCII mode data connection for /commonupdater.
226 Transfer Complete
CWD /CommonUpdater
250 CWD command successful.
PASV
227 Entering Passive Mode. (213,200,104,29,203,34)
RETR 53395340.upd
 
...and so on. So the difference is that the transfer over the proxy
stops after the listing, the listing is somehow stored into a index.html
and proxy behaves rather than a webbrowser. Proxy is Sun Java Webproxy
4.0.3.
 
Are there any options of wget I have to set?
 
Thanks for any help.

Re: rapidshare download problem

[Doruk Fisek]

Mon, 21 Jul 2008 12:27:57 -0700, Micah Cowan <[EMAIL PROTECTED]> :

> I'll file a report for this issue; but if this is the only thing that
> crops up, I probably won't roll a new 1.11.x maintenance release just
> for that, so it'll have to wait until 1.12.
Thank you.

   Doruk

--
FISEK INSTITUTE - http://www.fisek.org.tr

Re: rapidshare download problem

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Doruk Fisek wrote:
> Thu, 17 Jul 2008 15:07:18 -0700, Micah Cowan <[EMAIL PROTECTED]> :
> 
>> Then, please provide the logs from both wget 1.10.2 and wget 1.11.4
>> (with --auth-no-challenge), with the --debug flag.
> I attached the logs you requested.

It looks like --auth-no-challenge works when --user/--password or
- --http-user/--http-password is used; but not when the user/pass
information is passed directly via the URL. You can use
- --auth-no-challenge together with either of those option pairs as a
work-around.

- --auth-no-challenge was intended to make wget 1.11.x behave like 1.10.2
with regard to authentication (that is, broken and insecure). It's
generally a bad idea to send authentication credits before the server
tells Wget what sort of authentication scheme should be used (it may
offer more secure alternatives to the "basic" mechanism). However, in
situations where the server uses forms-based authentication for
interactive communication, some servers accept authentication if
offered, but will not issue the challenge if it's not, instead expecting
users to authenticate through the forms; situations like that are why
- --auth-no-challenge was introduced.

I'll file a report for this issue; but if this is the only thing that
crops up, I probably won't roll a new 1.11.x maintenance release just
for that, so it'll have to wait until 1.12.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer,
and GNU Wget Project Maintainer.
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIhOM97M8hyUobTrERAu94AJ9fO6LT+mn0M/2pEG1KS+7pwpKDBACcDnMj
AfJSnm3pJeKrQkEFlOeuyFk=
=9p3c
-END PGP SIGNATURE-

Re: rapidshare download problem

[Jochen Roderburg]

Zitat von "Doruk Fisek" <[EMAIL PROTECTED]>:


Thu, 17 Jul 2008 15:07:18 -0700, Micah Cowan <[EMAIL PROTECTED]> :


Then, please provide the logs from both wget 1.10.2 and wget 1.11.4
(with --auth-no-challenge), with the --debug flag.

I attached the logs you requested.

wget 1.10.2 didn't recognize the --auth-no-challenge parameter, so I
only used it in 1.11.4



Looks like the --auth-no-challenge option does not work correctly with  
the http://username:[EMAIL PROTECTED]/syntax.


When you put username/password in separate parameters it should work:
--http-user=username --http-passwd=password http://rs60tl.rapidshare.com/
It *does* work for me in this form with other servers  ;-)

Best regards,

Jochen Roderburg
ZAIK/RRZK
University of Cologne
Robert-Koch-Str. 10Tel.:   +49-221/478-7024
D-50931 Koeln  E-Mail: [EMAIL PROTECTED]
Germany



This message was sent using IMP, the Internet Messaging Program.

Re: rapidshare download problem

[Doruk Fisek]

Thu, 17 Jul 2008 15:07:18 -0700, Micah Cowan <[EMAIL PROTECTED]> :

> Then, please provide the logs from both wget 1.10.2 and wget 1.11.4
> (with --auth-no-challenge), with the --debug flag.
I attached the logs you requested.

wget 1.10.2 didn't recognize the --auth-no-challenge parameter, so I
only used it in 1.11.4

   Doruk

--
FISEK INSTITUTE - http://www.fisek.org.tr
DEBUG output created by Wget 1.10.2 on linux-gnu.

--21:49:12--  http://username:[EMAIL PROTECTED]/files/30168760/Rapidshare_EN.txt
   => `Rapidshare_EN.txt'
Resolving rs60tl.rapidshare.com... 80.239.151.61
Caching rs60tl.rapidshare.com => 80.239.151.61
Connecting to rs60tl.rapidshare.com|80.239.151.61|:80... connected.
Created socket 3.
Releasing 0x080856e0 (new refcount 1).

---request begin---
GET /files/30168760/Rapidshare_EN.txt HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Authorization: Basic xx
Host: rs60tl.rapidshare.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Mon, 21 Jul 2008 18:51:18 GMT
Connection: close
Content-Type: application/octet-stream
Accept-Ranges: bytes
Content-Disposition: Attachment; filename=Rapidshare_EN.txt
Content-Length: 225

---response end---
200 OK
Length: 225 [application/octet-stream]

100%[=>]
 225   --.--K/s

Closed fd 3
21:49:12 (1.83 KB/s) - `Rapidshare_EN.txt' saved [225/225]DEBUG output created by Wget 1.11.4 on linux-gnu.

--2008-07-21 21:48:54--  http://username:[EMAIL 
PROTECTED]/files/30168760/Rapidshare_EN.txt
Resolving rs60tl.rapidshare.com... 80.239.151.61
Caching rs60tl.rapidshare.com => 80.239.151.61
Connecting to rs60tl.rapidshare.com|80.239.151.61|:80... connected.
Created socket 3.
Releasing 0x08b08e68 (new refcount 1).

---request begin---
GET /files/30168760/Rapidshare_EN.txt HTTP/1.0
User-Agent: Wget/1.11.4
Accept: */*
Host: rs60tl.rapidshare.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
P3P: CP="ALL DSP COR CURa ADMa DEVa TAIa PSAa PSDa IVAa IVDa CONa TELa OUR STP 
UNI NAV STA PRE"
Date: Mon, 21 Jul 2008 18:53:46 GMT
Connection: close
Accept-Ranges: bytes
Content-Type: text/html; charset=ISO-8859-1
Cache-Control: no-cache
Content-Length: 8861

---response end---
200 OK
Length: 8861 (8.7K) [text/html]
Saving to: `Rapidshare_EN.txt'

100%[===>]
 8,861   36.7K/s   in 0.2s

Closed fd 3
2008-07-21 21:48:55 (36.7 KB/s) - `Rapidshare_EN.txt' saved [8861/8861]

Re: rapidshare download problem

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Doruk Fisek wrote:
> Thu, 17 Jul 2008 03:14:26 -0700, Micah Cowan <[EMAIL PROTECTED]> :
> 
>>>  http://username:[EMAIL PROTECTED]/files/30168760/Rapidshare_EN.txt
>>>  wget 1.10.2 downloads it just fine but wget 1.11.4 brings an html
>>> page instead.

Then, please provide the logs from both wget 1.10.2 and wget 1.11.4
(with --auth-no-challenge), with the --debug flag.

You'll need to massage these logs somewhat, I'm afraid; notably removing
your username/password information: be sure not to leave any of it
around. You'll also need to replace Wget's WWW-Authenticate: basic
header in the output: it might not look like it, but it contains your
very easily recoverable password information.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer,
and GNU Wget Project Maintainer.
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIf8KW7M8hyUobTrERAjF1AJ4i5ndrgKM5zAYDhpDKPAad2yhG8ACfb3C1
NUSfZgmhXfB/y8R+2ZV7nPc=
=w2cc
-END PGP SIGNATURE-

Re: rapidshare download problem

[Doruk Fisek]

Thu, 17 Jul 2008 03:14:26 -0700, Micah Cowan <[EMAIL PROTECTED]> :

> >  http://username:[EMAIL PROTECTED]/files/30168760/Rapidshare_EN.txt
> >  wget 1.10.2 downloads it just fine but wget 1.11.4 brings an html
> > page instead.
> See if --auth-no-challenge fixes it for you.
No, it doesn't. The end result is the same.

   Doruk

--
FISEK INSTITUTE - http://www.fisek.org.tr

Re: rapidshare download problem

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Doruk Fisek wrote:
> Hi,
> 
>  I'm having trouble cookieless downloading from rapidshare with the
> latest version of wget.
> 
>  When I use a url like;
>  
>  http://username:[EMAIL PROTECTED]/files/30168760/Rapidshare_EN.txt
> 
>  wget 1.10.2 downloads it just fine but wget 1.11.4 brings an html page
> instead.

See if --auth-no-challenge fixes it for you.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer,
and GNU Wget Project Maintainer.
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIfxuC7M8hyUobTrERAt/mAJ97QRCx4mTJKEbSyrql8hsy7Vty3QCeOc5/
GI8fqQaVyLjrx9x/nMgSdNM=
=wZbY
-END PGP SIGNATURE-

rapidshare download problem

[Doruk Fisek]

Hi,

 I'm having trouble cookieless downloading from rapidshare with the
latest version of wget.

 When I use a url like;
 
 http://username:[EMAIL PROTECTED]/files/30168760/Rapidshare_EN.txt

 wget 1.10.2 downloads it just fine but wget 1.11.4 brings an html page
instead.
 
   Doruk

--
FISEK INSTITUTE - http://www.fisek.org.tr

Re: Problem with wget 1.11. Please help

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Amit Patel wrote:
> Ya. I have checked it properly. It checks certificates. If i don't
> specify /etc/ca-bundle.crt with my working version of 1.10.2, it
> provides "Self-signed certificate encountered" error and fails.

Er, I'm not sure, but I think I would expect that to be a problem,
regardless of whether /etc/ca-bundle.crt.

Could you please verify whether a stock wget-1.10.2, configured and
installed from our source tarballs (not RedHat's), available at
ftp.gnu.org/gnu/wget/wget-1.10.2.tar.gz ?

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer,
and GNU Wget Project Maintainer.
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH9isf7M8hyUobTrERApgfAJ4801HnL9/5W4QdcLMF87KNnutHJgCeP2bm
8rU8dsRxa4s76i/a3pnaeZ4=
=MLQZ
-END PGP SIGNATURE-

Re: Problem with wget 1.11. Please help

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Amit Patel wrote:
> Hi  Hrvoje Niksic / Micah Cowan
> 
> I am having strange problem with wget 1.11 version. While accessing a
> webserver which is running on Verisign issued Trial certificates, wget
> 1.11 gives following error even though that certificate is valid.
> 
> [EMAIL PROTECTED]:/root $ wget --ca-certificate=/etc/ca-bundle.crt
> https://rw01246.einfochips.com*



> However , if i try same with wget version 1.10.2  which is on another
> redhat machine, it is working perfect. I am not able to understand where
> the problem is. If you can spare some time and  help  help me to solve
> the issue , that would be great.

RedHat has been known to modify Wget fairly heavily. Recent versions of
RedHat's "Wget 1.10.2" differ very substantially from ours. It would be
more useful to see how a wget-1.10.2 that was built straight from our
source packages would compare.

Is /etc/ca-bundle.crt exactly the same on both systems?

And: are you sure the "working" version doesn't have "check_certificate
= off" in either the /etc/wgetrc or ~/.wgetrc?

...If you can verify all of that, then I'm not sure how to go forward:
we'd need access to try that server, I guess (which doesn't resolve, for
me).

Also: Hrvoje and I are both subscribed to this list; no need to Cc us.

It's recommended, when posting to lists, that you choose a subject
header that describes the problem you're having, rather than just
describing that you have a problem. "Valid SSL certificate not accepted"
would have been a reasonable subject.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer,
and GNU Wget Project Maintainer.
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH9RET7M8hyUobTrERArgYAJ9MwEOct/mhd0ic9/HzMTvuLXV74ACeJpvK
Ub3NwgJB5fLe4Z11agbCOWk=
=J4Pg
-END PGP SIGNATURE-

Problem with wget 1.11. Please help

[Amit Patel]

Hi  Hrvoje Niksic / Micah Cowan

I am having strange problem with wget 1.11 version. While accessing a 
webserver which is running on Verisign issued Trial certificates, wget 
1.11 gives following error even though that certificate is valid.


[EMAIL PROTECTED]:/root $ wget --ca-certificate=/etc/ca-bundle.crt 
https://rw01246.einfochips.com*


--2008-04-04 21:22:57--  https://rw01246.einfochips.com/
Resolving rw01246.einfochips.com... 10.101.1.246
Connecting to rw01246.einfochips.com|10.101.1.246|:443... connected.
ERROR: cannot verify rw01246.einfochips.com's certificate, issued by 
`/C=US/O=VeriSign, Inc./OU=For Test Purposes Only.  No 
assurances./OU=Terms of use at https://www.verisign.com/cps/testca 
(c)05/CN=VeriSign Trial Secure Server Test CA':

/*  certificate signature failure*/
To connect to rw01246.einfochips.com insecurely, use 
`--no-check-certificate'.

Unable to establish SSL connection.


However , if i try same with wget version 1.10.2  which is on another 
redhat machine, it is working perfect. I am not able to understand where 
the problem is. If you can spare some time and  help  help me to solve 
the issue , that would be great.


Following is the output of "ldd" command on both the machines. This may 
help you in debugging the issue.


*For wget version 1.11 (This version is having problem.)*

   libdl.so.2 => /lib/libdl.so.2 (0x40019000) [actual file - 
libdl-2.2.4.so ]
   librt.so.1 => /lib/librt.so.1 (0x4001d000) [actual file - 
librt-2.2.4.so ]
   libssl.so.0 => /usr/lib/libssl.so.0 (0x4003) [actual file - 
libssl.so.0.9.5a ]
   libcrypto.so.0 => /usr/lib/libcrypto.so.0 (0x4005e000) [actual 
file - libcrypto.so.0.9.5a ]
   libc.so.6 => /lib/libc.so.6 (0x4011a000) [actual file - 
libc-2.2.4.so ]
   libpthread.so.0 => /lib/libpthread.so.0 (0x40233000) [actual 
file -  libpthread-0.9.so ]
   /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x4000) [actual 
file - ld-2.2.4.so ]


*For wget version 1.10.2 (This works fine.)*  


   linux-gate.so.1 =>  (0x0064c000)
   libssl.so.6 => /lib/libssl.so.6 (0x0045b000) [actual file - 
libssl.so.0.9.8b]
   libcrypto.so.6 => /lib/libcrypto.so.6 (0x001dd000) [actual file 
- libcrypto.so.0.9.8b]
   libdl.so.2 => /lib/libdl.so.2 (0x00c4) [actual file - 
libdl-2.5.so]
   libz.so.1 => /usr/lib/libz.so.1 (0x00c88000) [actual file - 
libz.so.1.2.3]
   librt.so.1 => /lib/i686/nosegneg/librt.so.1 (0x060f9000) [actual 
file - librt-2.5.so]
   libc.so.6 => /lib/i686/nosegneg/libc.so.6 (0x00afd000) [actual 
file - libc-2.5.so]
   libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00373000) 
[actual file - libgssapi_krb5.so.2.2]
   libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x003a) [actual file 
- libkrb5.so.3.2]
   libcom_err.so.2 => /lib/libcom_err.so.2 (0x0033b000) [actual 
file - libcom_err.so.2.1]
   libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00433000) 
[actual file - libk5crypto.so.3.0]
   libresolv.so.2 => /lib/libresolv.so.2 (0x00326000) [actual file 
- libresolv-2.5.so]

   /lib/ld-linux.so.2 (0x0012e000)
   libpthread.so.0 => /lib/i686/nosegneg/libpthread.so.0 
(0x00c6f000) [actual file - libpthread-2.5.so]
   libkrb5support.so.0 => /usr/lib/libkrb5support.so.0 (0x00429000) 
[actual file - libkrb5support.so.0.1]



Waiting for positive response.

Thanks in advance,
Amit Patel
--
_
Disclaimer: This e-mail message and all attachments transmitted with it
are intended solely for the use of the addressee and may contain legally
privileged and confidential information. If the reader of this message
is not the intended recipient, or an employee or agent responsible for
delivering this message to the intended recipient, you are hereby
notified that any dissemination, distribution, copying, or other use of
this message or its attachments is strictly prohibited. If you have
received this message in error, please notify the sender immediately by
replying to this message and please delete it from your computer. Any
views expressed in this message are those of the individual sender
unless otherwise stated.Company has taken enough precautions to prevent
the spread of viruses. However the company accepts no liability for any
damage caused by any virus transmitted by this email.
__

Re: Problem with wget (v1.11) and FTP URL including ..

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Micah Cowan wrote:

> it looks to me like your URL
> (ftp://ftp.xxx.pwp.blueyonder.co.uk/../logs/access.20080309) is
> being transmuted by some URL "cleanup" code well before it ever gets to
> the FTP handle. My guess is that this change was intentionally done, as
> such a URL for HTTP would probably be wrong (I'm not sure that it's
> actually ill-formed; but for safety reasons it was probably a good idea
> to remove ".." from the beginning of paths in HTTP). However, it's
> perfectly fine for FTP URLs, and removing it for them is misbehavior.

This change was introduced in 2006 Feb, at the suggestion of Frank
McCown: http://article.gmane.org/gmane.comp.web.wget.general/5290/

The change can be seen here:
http://hg.addictivecode.org/wget/mainline/rev/798506c1ce67/

The change was introduced to comply with the recommended algorithm for
normalizing paths in RFC 3986 (section 5.2.4). So the change was
deliberate, and done to comply with recommended practice.

The thing is, of course, is that it sort of breaks the FTP scheme
defined in RFC 1738. Are there any updates to the FTP URL definition? I
don't see any (it seems to me that the RFC index would show an
"Updated-By" for 1738 to FTP URL updates, as it does for gopher and mailto).

I'm going to punt this so it doesn't hold up 1.11.1 (speaking of which,
has anyone been banging on the prerelease? :\ ). I vote that we ignore
what 3986 says with respect to ftp paths, perhaps passing a flag to the
path_simplify function so it knows whether to do that or not?

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH4aoa7M8hyUobTrERAnh2AJ9TBQwyknhPl9pUFzVuTnk9LtQ0rQCcCTO0
VNz+Bv6tVv6KvSl19yIVeBs=
=nRHg
-END PGP SIGNATURE-

Re: Problem with wget (v1.11) and FTP URL including ..

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Richard wrote:
> Hi Micah,
> 
> I've tried to post this reply to the mailing list but it's getting
> blocked by SpamAssassin so having to reply directly - hope that's OK.

Sure; it's probably due to the "xxx" strings within URLs. :)

I was thinking perhaps the "working" case with Wget 1.11 would be using
a so-called "FTP proxy"; an FTP server that proxies to other servers. Of
course, the FTP-handling logic for FTP-over-HTTP proxies is entirely
handled by the HTTP proxy server, so it makes sense that it would work
there.

The logs look like enough to fish out the problem, so I'll do some
snooping around to see what can be done about this.

Judging from this bit from the failing Wget 1.11 logs:

- --2008-03-10 18:59:10--
ftp://ftp.xxx.pwp.blueyonder.co.uk/logs/access.20080309.gz
Host `ftp.xxx.pwp.blueyonder.co.uk' has not issued a general basic
challenge.
Resolving webcache.virginmedia.com... 195.188.152.6

it looks to me like your URL
(ftp://ftp.xxx.pwp.blueyonder.co.uk/../logs/access.20080309) is
being transmuted by some URL "cleanup" code well before it ever gets to
the FTP handle. My guess is that this change was intentionally done, as
such a URL for HTTP would probably be wrong (I'm not sure that it's
actually ill-formed; but for safety reasons it was probably a good idea
to remove ".." from the beginning of paths in HTTP). However, it's
perfectly fine for FTP URLs, and removing it for them is misbehavior.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH1Y8t7M8hyUobTrERAietAJ4pzcZ9vVOTk4Bsy9wn89J5oHCyTQCdEgbR
VtaAtKiIhh1lXmTi9CAB9do=
=vfM1
-END PGP SIGNATURE-

Re: Problem with wget (v1.11) and FTP URL including ..

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Richard wrote:
> Hi,
> 
> Since upgrading wget from 1.10.2 to 1.11 (on a Sun Solaris 9 server) I
> am no longer able to retrieve files with a command similar to:
> 
> wget --user=xxx --password=xxx --output-document=logfile.txt
> ftp://ftp.username.myby.co.uk/../logs/logfile.txt
> 
> It reports the following:
> 
> ==> SYST ... done.==> PWD ... done.
> ==> TYPE I ... done.  ==> CWD /htdocs/logs ...
> No such directory `logs'.
> 
> The directory htdocs is the 'home' directory, and so it looks as if wget
> has not gone up one directory (ie. done the */../* part of the URL),
> before going into logs.
> 
> This worked fine in 1.10.2, and I have also just discovered that it DOES
> work in 1.11 if I go via a proxy server!
> 
> *Please CC me in on any replies as I have not subscribed to this list.*

I'd need to see the full logs (with --debug set) for both Wget 1.10.2
and the working (via proxy) and not-working Wget 1.11 cases. Or else, an
example URL that we can test directly that gives this behavior.

I'll try to set up a similar test of my own when I have a chance, but
the fact that it works when using a proxy makes me think that it's
server-dependent behavior, so I'll probably still end up needing logs.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.7 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD4DBQFH1XvD7M8hyUobTrERAuc7AKCHsMrHIkazzhNkAJtrG0epumGewwCXbp5G
c+DE7xGxTu1I8kdWJk5XzA==
=a1Gm
-END PGP SIGNATURE-

Problem with wget (v1.11) and FTP URL including ..

[Richard]

Hi,

Since upgrading wget from 1.10.2 to 1.11 (on a Sun Solaris 9 server) I am no
longer able to retrieve files with a command similar to:

wget --user=xxx --password=xxx --output-document=logfile.txt
ftp://ftp.username.myby.co.uk/../logs/logfile.txt

It reports the following:

==> SYST ... done.==> PWD ... done.
==> TYPE I ... done.  ==> CWD /htdocs/logs ...
No such directory `logs'.

The directory htdocs is the 'home' directory, and so it looks as if wget has
not gone up one directory (ie. done the */../* part of the URL), before
going into logs.

This worked fine in 1.10.2, and I have also just discovered that it DOES
work in 1.11 if I go via a proxy server!

*Please CC me in on any replies as I have not subscribed to this list.*

Thanks,

Richard van der Leeden

Problem with Wget on windows

[Hunny Garg]

Hi
I am trying to download from a ftp server using wget on windows. The
problem I am facing is that if there a directory having space in its
name then wget replaces that space character with @20 after downloading
on my machine. The client and server are both on windows machine
Basically I  want to maintain exactly the same directory structure as it
is on server machineis there anyway to achieve that.

Thanx in advance

Re: Problem with password.

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Rene Maldonado wrote:
> /usr/bin/wget -nc -x -nH
> --directory-prefix=/var/local/Process/Data/CSFN/tmp/ -a /tmp/wgetFTP.log
> ftp://myname:[EMAIL PROTECTED]@10.50.2.35/IN/

> How can I use the @ symbol on the password?

Percent-encode it: p%40ssword.

You can also use --password or --ftp-password.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHuz3K7M8hyUobTrERAjMxAJwIP5z1bEz1+zimjO/yjopuXV7nNQCggPSW
4TH4mdNCSysjSBwMkfEledw=
=kDGs
-END PGP SIGNATURE-

Problem with password.

[Rene Maldonado]

 

Hello, I'm trying to use Wget, in this form:

 

/usr/bin/wget -nc -x -nH
--directory-prefix=/var/local/Process/Data/CSFN/tmp/ -a /tmp/wgetFTP.log
ftp://myname:[EMAIL PROTECTED]@10.50.2.35/IN/

 

But, my password contains an arroba (@) and wget try to connect to
[EMAIL PROTECTED] 

 

How can I use the @ symbol on the password?

 

Thanks

  Rene

 

Installation problem with man file when using object dir during build

[Heinz-Ado Arnolds]

Dear Mr. Cowan,

when using a seperate object directory during build the installation procedure
tries to install the man page from this object dir instead of the source dir.

A simple patch is attached.

Kind regards,

Ado

diff -cr wget-1.11.orig/doc/Makefile.in wget-1.11/doc/Makefile.in
*** wget-1.11.orig/doc/Makefile.in  Sat Jan 26 10:26:56 2008
--- wget-1.11/doc/Makefile.in   Wed Feb 13 09:08:24 2008
***
*** 124,130 
  # install man page, creating install directory if necessary
  install.man: $(MAN)
$(top_srcdir)/mkinstalldirs $(DESTDIR)$(mandir)/man$(manext)
!   $(INSTALL_DATA) $(MAN) $(DESTDIR)$(mandir)/man$(manext)/$(MAN)
  
  # install sample.wgetrc
  install.wgetrc: $(srcdir)/sample.wgetrc
--- 124,130 
  # install man page, creating install directory if necessary
  install.man: $(MAN)
$(top_srcdir)/mkinstalldirs $(DESTDIR)$(mandir)/man$(manext)
!   $(INSTALL_DATA) $(srcdir)/$(MAN) $(DESTDIR)$(mandir)/man$(manext)/$(MAN)
  
  # install sample.wgetrc
  install.wgetrc: $(srcdir)/sample.wgetrc

Re: Minor fix to solve problem in wget-1.10.2 incorrectly parsing the date when doing FTP gets

[Philip Gladstone]

Micah Cowan wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Philip Gladstone wrote:
  

Brief outline:

When you do a wget from an ftp: url, it tries to copy over the last
modified date of the file (this is good). However, due to timezone
issues, the date that it gets can sometimes be in the future, and then
it guesses the year incorrectly. This patch fixes that problem.



(See original message, with patch, at
http://article.gmane.org/gmane.comp.web.wget.patches/2224)

Good catch.

However, on closer inspection, this patch doesn't quite fix the problem,
properly. It checks whether the file's time is in the first day of the
next month, but not whether the _current_ time is the last day of the
current month. We need to do a stronger check for whether it's truly
within 24 hours of the current time.


  
If you go that far, then you need to allow for the fact that local times 
can be (AFAIK) 25 hours apart in the worst case. If you want to allow 
for clock skew as well, then probably more than 25 hours would be a 
sensible amount. Maybe 96 hours.


philip

Re: Minor fix to solve problem in wget-1.10.2 incorrectly parsing the date when doing FTP gets

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Philip Gladstone wrote:
> Micah Cowan wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Philip Gladstone wrote:
>>  
>>> Brief outline:
>>>
>>> When you do a wget from an ftp: url, it tries to copy over the last
>>> modified date of the file (this is good). However, due to timezone
>>> issues, the date that it gets can sometimes be in the future, and then
>>> it guesses the year incorrectly. This patch fixes that problem.
>>> 
>>
>> (See original message, with patch, at
>> http://article.gmane.org/gmane.comp.web.wget.patches/2224)
>>
>> Good catch.
>>
>> However, on closer inspection, this patch doesn't quite fix the problem,
>> properly. It checks whether the file's time is in the first day of the
>> next month, but not whether the _current_ time is the last day of the
>> current month. We need to do a stronger check for whether it's truly
>> within 24 hours of the current time.
>
> If you go that far, then you need to allow for the fact that local times
> can be (AFAIK) 25 hours apart in the worst case. If you want to allow
> for clock skew as well, then probably more than 25 hours would be a
> sensible amount. Maybe 96 hours.

I think 24 hours is reasonable; it's a guess in any case. But I do think
that, if today is December 3, and the file is January 1, it should be
considered to use the current year (while the patch, as it stands, would
automatically consider it to be Jan 1 of next year).

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHWzre7M8hyUobTrERAuTdAKCGwei4rx9zZUmGDxgtNjYQm4aHkQCbBJMb
1YI/1C/J2kXoI13jTppnXKs=
=T02u
-END PGP SIGNATURE-

Re: Minor fix to solve problem in wget-1.10.2 incorrectly parsing the date when doing FTP gets

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Philip Gladstone wrote:
> Brief outline:
> 
> When you do a wget from an ftp: url, it tries to copy over the last
> modified date of the file (this is good). However, due to timezone
> issues, the date that it gets can sometimes be in the future, and then
> it guesses the year incorrectly. This patch fixes that problem.

(See original message, with patch, at
http://article.gmane.org/gmane.comp.web.wget.patches/2224)

Good catch.

However, on closer inspection, this patch doesn't quite fix the problem,
properly. It checks whether the file's time is in the first day of the
next month, but not whether the _current_ time is the last day of the
current month. We need to do a stronger check for whether it's truly
within 24 hours of the current time.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHWxZe7M8hyUobTrERAskXAJ4z6Ir7qIYMCXLHBjJVLlqmERMXOgCgkcDR
ihFdxJAfFU8l3qMo+AiLns4=
=wSIh
-END PGP SIGNATURE-

Re: Problem with WGET

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Sorokin Nikita wrote:
> Hello!
> I want to download all files with .RAR extension in specified directory
> (http://test.com/test/ and all RAR files...), but I don't know how to
> dothat.
> Please, help me...

Generally, real, live URLs are preferred to fake ones, to help us help you.

Have you tried --accept=.RAR,.rar ?

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFHQeJu7M8hyUobTrERCGobAJ4jHWJK0k0NICH/46qCCD57yzU9PACfTvbj
FoqVxkHvFFCUcf/CevZZ4ZI=
=rAss
-END PGP SIGNATURE-

Problem with WGET

[Sorokin Nikita]

Hello!
I want to download all files with .RAR extension in specified directory
(http://test.com/test/ and all RAR files...), but I don't know how to dothat.
Please, help me...

Thanks,
N.S.

Re: wget -c problem with current svn version

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Jochen Roderburg wrote:
> I see also a conflict between older changes by Mauro
> and the latest changes by Micah in this area.

Actually, I never made any changes to this area that I recall; just
merged in changes others made. :)

I'm not really sure of how all that works, either. The code was already
complicated, and the code from the b20323 branch hasn't helped much in
that regard. got_name, AFAICT, is a misnomer anyway, because it tracks
more than whether we've simply gotten a name.

I'd care a little more about that if I wasn't already planning to
rewrite http_loop in the near future. At any rate, though, it looks like
the new changes merit a closer look.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG7EMD7M8hyUobTrERCJT/AJ9gmWyHUjclbQNotDmW41kbgebENwCcCJWW
Vz50KZDbMDgLDmdkASPFThg=
=vJBt
-END PGP SIGNATURE-

Re: wget -c problem with current svn version

[Jochen Roderburg]

Zitat von Jochen Roderburg <[EMAIL PROTECTED]>:

>
> Continued download (wget -c) is not done in the current svn version with
> default
> options (where no HEAD is used). The download starts instead at byte 0 again.
> When other options require a HEAD, it works ok again.

Another astonishing test result:

With "wget -c -O file URL" continuation works fine on the -O file (!!!), it even
makes a timestamp on that file. I think this raises several questions   ;-)

First, I think, -c should also be in the family of options which are not
compatible with -O (where we already have -r, -p, -N).

Second, it could give hints where the problem with -c lies.

Actually I can understand in the code what happens, but i do not understand the
intended logic and cannot correct it, that is now really for Mauro and Micah.
There is a variable "got_name" in http.c which seems to be used for different
purposes. One usage is as indicator that -O is used and the other has something
to do with the -c logic. I see also a conflict between older changes by Mauro
and the latest changes by Micah in this area.

Interesting code snippets:

http.c, line 2143 ff.

  /* Decide whether or not to restart.  */
  if (opt.always_rest
  && got_name
  && stat (hstat.local_file, &st) == 0
  && S_ISREG (st.st_mode))
/* When -c is used, continue from on-disk size.  (Can't use
   hstat.len even if count>1 because we don't want a failed
   first attempt to clobber existing data.)  */
hstat.restval = st.st_size;

http.c, line 2634 ff.

  if (send_head_first)
{
  got_name = true;
  restart_loop = true;
}

in an older version this was

  if (opt.always_rest)
{
  got_name = true;
  restart_loop = true;
}


Regards, J.Roderburg

wget -c problem with current svn version

[Jochen Roderburg]

Continued download (wget -c) is not done in the current svn version with default
options (where no HEAD is used). The download starts instead at byte 0 again.
When other options require a HEAD, it works ok again. Perhaps the correction is
as easy as adding the '-c' case to those options that need a HEAD request.  ;-)

Regards, J.Roderburg

Log outputs for various versions:

Version 1.10.2 does no HEAD, but immediately a GET with Range

wget.1102 --debug -c http://ftp.uni-koeln.de/files.lst.gz

Setting --continue (continue) to 1
DEBUG output created by Wget 1.10.2 on linux-gnu.

--22:48:58--  http://ftp.uni-koeln.de/files.lst.gz
   => iles.lst.gz'
Resolving ftp.uni-koeln.de... 134.95.19.35
Caching ftp.uni-koeln.de => 134.95.19.35
Connecting to ftp.uni-koeln.de|134.95.19.35|:80... connected.
Created socket 3.
Releasing 0x08084a00 (new refcount 1).

---request begin---
GET /files.lst.gz HTTP/1.0
Range: bytes=6033568-
User-Agent: Wget/1.10.2
Accept: */*
Host: ftp.uni-koeln.de
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 206 Partial Content
Date: Thu, 13 Sep 2007 20:48:59 GMT
Server: Apache/2.0.46 (Red Hat)
Last-Modified: Wed, 12 Sep 2007 04:08:33 GMT
ETag: "1b7500ba-1524e9d-60055240"
Accept-Ranges: bytes
Content-Length: 16137725
Content-Range: bytes 6033568-22171292/22171293
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/x-gzip
Content-Encoding: x-gzip

---response end---
206 Partial Content
Registered socket 3 for persistent reuse.
Length: 22,171,293 (21M), 16,137,725 (15M) remaining [application/x-gzip]

59% [+++=>] 13,095,904
1.61M/sETA 00:05


SVN version from a month ago does a HEAD and a GET with Range

wget.111-svn-0708 --debug -c http://ftp.uni-koeln.de/files.lst.gz

Setting --continue (continue) to 1
DEBUG output created by Wget 1.10+devel on linux-gnu.

--22:52:40--  http://ftp.uni-koeln.de/files.lst.gz
Resolving ftp.uni-koeln.de... 134.95.19.35
Caching ftp.uni-koeln.de => 134.95.19.35
Connecting to ftp.uni-koeln.de|134.95.19.35|:80... connected.
Created socket 3.
Releasing 0x080884c8 (new refcount 1).

---request begin---
HEAD /files.lst.gz HTTP/1.0
User-Agent: Wget/1.10+devel
Accept: */*
Host: ftp.uni-koeln.de
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Thu, 13 Sep 2007 20:52:40 GMT
Server: Apache/2.0.46 (Red Hat)
Last-Modified: Wed, 12 Sep 2007 04:08:33 GMT
ETag: "1b7500ba-1524e9d-60055240"
Accept-Ranges: bytes
Content-Length: 22171293
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/x-gzip
Content-Encoding: x-gzip

---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 22171293 (21M) [application/x-gzip]
--22:52:40--  http://ftp.uni-koeln.de/files.lst.gz
Reusing existing connection to ftp.uni-koeln.de:80.
Reusing fd 3.

---request begin---
GET /files.lst.gz HTTP/1.0
Range: bytes=6033568-
User-Agent: Wget/1.10+devel
Accept: */*
Host: ftp.uni-koeln.de
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 206 Partial Content
Date: Thu, 13 Sep 2007 20:52:40 GMT
Server: Apache/2.0.46 (Red Hat)
Last-Modified: Wed, 12 Sep 2007 04:08:33 GMT
ETag: "1b7500ba-1524e9d-60055240"
Accept-Ranges: bytes
Content-Length: 16137725
Content-Range: bytes 6033568-22171292/22171293
Keep-Alive: timeout=15, max=99
Connection: Keep-Alive
Content-Type: application/x-gzip
Content-Encoding: x-gzip

---response end---
206 Partial Content
Length: 22171293 (21M), 16137725 (15M) remaining [application/x-gzip]
Saving to: iles.lst.gz'
58% [==> ] 13,030,816 
1.50M/s  eta 6s


Current SVN version does no HEAD and a GET for the whole file again

wget.111-svn-0709 --debug -c http://ftp.uni-koeln.de/files.lst.gz
Setting --continue (continue) to 1
DEBUG output created by Wget 1.10+devel on linux-gnu.

--22:56:39--  http://ftp.uni-koeln.de/files.lst.gz
Resolving ftp.uni-koeln.de... 134.95.19.35
Caching ftp.uni-koeln.de => 134.95.19.35
Connecting to ftp.uni-koeln.de|134.95.19.35|:80... connected.
Created socket 3.
Releasing 0x080884c8 (new refcount 1).

---request begin---
GET /files.lst.gz HTTP/1.0
User-Agent: Wget/1.10+devel
Accept: */*
Host: ftp.uni-koeln.de
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Thu, 13 Sep 2007 20:56:39 GMT
Server: Apache/2.0.46 (Red Hat)
Last-Modified: Wed, 12 Sep 2007 04:08:33 GMT
ETag: "1b7500ba-1524e9d-60055240"
Accept-Ranges: bytes
Content-Length: 22171293
Keep-Alive: timeout=15, max=100
Connection: Keep-Alive
Content-Type: application/x-gzip
Content-Encoding: x-gzip

---response end---
200 OK
Registered socket 3 for persistent reuse.
Length: 22171293 (21M) [application/x-gzip]
Saving to: iles.lst.gz'

26% [

Re: wget syntax problem ?

[Josh Williams]

On 9/6/07, Micah Cowan <[EMAIL PROTECTED]> wrote:
> Not really; we've been Cc'ing you. I don't think we knew whether you
> were subscribed or not, and so Cc'd you in case you weren't. Also, many
> of us just habitually hit Reply All to hit the message, so we don't
> accidentally send it to the message's author only. :)

aye. Gmail doesn't have that problem, though. If it finds a duplicate
message from a mailing list, it only shows me the one from the list.
Kind of nice.

Re: wget syntax problem ?

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Alan Thomas wrote:
> command.com
> 
> By the way, Josh and your messages are being put out to the list in
> dupicates (at least, that`s what I`m seeing on my end).

Not really; we've been Cc'ing you. I don't think we knew whether you
were subscribed or not, and so Cc'd you in case you weren't. Also, many
of us just habitually hit Reply All to hit the message, so we don't
accidentally send it to the message's author only. :)

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4Kys7M8hyUobTrERCCK4AJ9rOGMPa1Xcl/evqENs6pmN7AAncACfeWhd
nyC+OzJ3ME7vMqRsEoVNP68=
=n6JC
-END PGP SIGNATURE-

Re: wget syntax problem ?

[Alan Thomas]

command.com

By the way, Josh and your messages are being put out to the list in
dupicates (at least, that`s what I`m seeing on my end).

- Original Message - 
From: "Micah Cowan" <[EMAIL PROTECTED]>
To: "Alan Thomas" <[EMAIL PROTECTED]>
Cc: 
Sent: Thursday, September 06, 2007 9:34 PM
Subject: Re: wget syntax problem ?


> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> Alan Thomas wrote:
> > Please ignore.  It was needing the "\\", like Josh said.
>
> Out of curiosity, what command interpreter were you using? Was this
> command.com, or something else like rxvt/Cygwin?
>
> - --
> Micah J. Cowan
> Programmer, musician, typesetting enthusiast, gamer...
> http://micah.cowan.name/
>
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.6 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iD4DBQFG4Kqe7M8hyUobTrERCI3HAJjw+g0GsGE1b+6vhr+pu/QJAQIuAJ4o2UbP
> e3qqbx+ywsdRpTuIbx6VPQ==
> =792z
> -END PGP SIGNATURE-

Re: wget syntax problem ?

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Alan Thomas wrote:
> Please ignore.  It was needing the "\\", like Josh said.

Out of curiosity, what command interpreter were you using? Was this
command.com, or something else like rxvt/Cygwin?

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD4DBQFG4Kqe7M8hyUobTrERCI3HAJjw+g0GsGE1b+6vhr+pu/QJAQIuAJ4o2UbP
e3qqbx+ywsdRpTuIbx6VPQ==
=792z
-END PGP SIGNATURE-

Re: wget syntax problem ?

[Alan Thomas]

Please ignore.  It was needing the "\\", like Josh said.

- Original Message - 
From: "Alan Thomas" <[EMAIL PROTECTED]>
To: "Josh Williams" <[EMAIL PROTECTED]>; 
Sent: Thursday, September 06, 2007 9:25 PM
Subject: Re: wget syntax problem ?


> Wget does not like my use of the "--directory-prefix=" option.  Anyone
know
> why?
>
> - Original Message - 
> From: "Josh Williams" <[EMAIL PROTECTED]>
> To: "Alan Thomas" <[EMAIL PROTECTED]>
> Cc: 
> Sent: Thursday, September 06, 2007 8:53 PM
> Subject: Re: wget syntax problem ?
>
>
> > On 9/6/07, Alan Thomas <[EMAIL PROTECTED]> wrote:
> > >
> > >
> > >I know this is probably something simple I screwed up, but the
> following
> > > commands in a Windows batch file return the error "Bad command or file
> name"
> > > for the wget command
> > >
> > > cd ..
> > > wget --convert-links
> > > --directory-prefix="C:\WINDOWS\Profiles\Alan000\Desktop\wget\CNN\"
> > > --no-clobber "http://www.cnn.com";
> >
> > Don't use backslashes in filenames. If you do, use `\\` instead.
>

Re: wget syntax problem ?

[Micah Cowan]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Alan Thomas wrote:
>I know this is probably something simple I screwed up, but the
> following commands in a Windows batch file return the error "Bad command
> or file name" for the wget command

It sounds to me like you don't have wget in your PATH. Make sure that
wget is located somewhere where command.com (or whatever) can find it.

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer...
http://micah.cowan.name/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFG4Kki7M8hyUobTrERCCG9AJ90dQ95sGaqEwVyH7KOZQxwlL7xCQCfWeJz
v9aCRAPhJp3kqZtd6zS0KNs=
=IAsR
-END PGP SIGNATURE-

Re: wget syntax problem ?

[Alan Thomas]

Wget does not like my use of the "--directory-prefix=" option.  Anyone know
why?

- Original Message - 
From: "Josh Williams" <[EMAIL PROTECTED]>
To: "Alan Thomas" <[EMAIL PROTECTED]>
Cc: 
Sent: Thursday, September 06, 2007 8:53 PM
Subject: Re: wget syntax problem ?


> On 9/6/07, Alan Thomas <[EMAIL PROTECTED]> wrote:
> >
> >
> >I know this is probably something simple I screwed up, but the
following
> > commands in a Windows batch file return the error "Bad command or file
name"
> > for the wget command
> >
> > cd ..
> > wget --convert-links
> > --directory-prefix="C:\WINDOWS\Profiles\Alan000\Desktop\wget\CNN\"
> > --no-clobber "http://www.cnn.com";
>
> Don't use backslashes in filenames. If you do, use `\\` instead.

Re: wget syntax problem ?

[Josh Williams]

On 9/6/07, Alan Thomas <[EMAIL PROTECTED]> wrote:
>
>
>I know this is probably something simple I screwed up, but the following
> commands in a Windows batch file return the error "Bad command or file name"
> for the wget command
>
> cd ..
> wget --convert-links
> --directory-prefix="C:\WINDOWS\Profiles\Alan000\Desktop\wget\CNN\"
> --no-clobber "http://www.cnn.com";

Don't use backslashes in filenames. If you do, use `\\` instead.

mirroring problem: redundant lines in .listing files

[Seth Purcell]

Hi,
 
1. I'm using wget on a Windows Server x64 system, which I doubt is
common and so may have problems all its own, but I set up a mirror and
went away for the weekend, and when I logged in this morning, all the
.listing files in the mirror directories had grown to contain hundreds
of duplicate lines for each file, so the program was trying to download
the same files over and over again (and my download log file was 1.5GB).
Has anyone seen this before, or does anyone have a theory as to how this
occurred? It seems as though the program appends to the listing files
each time it runs rather than replaces them. Here's the command I'm
running:
 
wget --mirror -c --progress=dot:mega --ftp-user=
--ftp-pass= -oF:\Data\download_log.txt
ftp://datasiteftp.dataprovider.com
 
2. Also, I don't think any files have decreased in size, but perhaps
because of the above problem, I'm getting strange lines like the
following in my log file:
 
Length: 7,465,084 (7.1M), -1,052,611 (-1052611) remaining
 
[ skipping 6144K ]
6144K     ,  114% 0.00 B/s
 
Which seems worrisomely nonsensical, and causes me to wonder if the
files are being downloaded correctly.
 
I'm not subscribed to the list, so please CC me on replies.
 
Thanks for any help,
 
Seth Purcell
Quantitative Analyst
Cambridge Place Investment Management Inc.
100 Main Street - Suite 240
Concord, MA 01742 USA
 
Tel: +1 978 369 3134
Fax: +1 978 369 1483
Mob: +1 978 760 0032
[EMAIL PROTECTED]
www.cpim.co.uk <http://www.cpim.co.uk/> 
 

*
The contents of this message and any attachments may be privileged, 
confidential, proprietary or otherwise protected from
disclosure. If you are not the intended recipient of this e-mail and have 
received it in error, please forward it to
[EMAIL PROTECTED] and then delete it from your mailbox without making copies.


Internet communications are not secure, and Cambridge Place does not accept 
legal responsibility for the contents of this
message. Replies to this email may be monitored by Cambridge Place for 
operational or business reasons.





This message has been scanned for viruses by MailController - 
www.MailController.altohiway.com

RE: Problem with combinations of the -O , -p, and -k parameters in wget

[Tony Lewis]

Michiel de Boer wrote:

> Is there another way though to achieve the same thing?

You can always run wget and then rename the file afterward. If this happens
often, you might want write a shell script to handle it. Of course, If you
want all the references to the file to be converted, the script will be a
little more complicated. :-)

Tony

Re: Problem with combinations of the -O , -p, and -k parameters in wget

[Michiel de Boer]

Steven M. Schweda wrote:

From: Michiel de Boer


[...] Therefore I use -O to write to a more sensible name.  [...]


   Unfortunately, "-O" does not do name conversion, it simply directs
all the program output to a specified file, and this causes bad behavior
when "-O" is combined with many other options.  Use the "Search" feature
at http://www.mail-archive.com/wget@sunsite.dk/ (for "-O") to find many
similar complaints involving "-O".



   Steven M. Schweda   [EMAIL PROTECTED]
   382 South Warwick Street(+1) 651-699-9818
   Saint Paul  MN  55105-2547



Thanks for the reply :)
I had a conversation on freenode's #wget with micahcowan too, who told me
about this. Is there another way though to achieve the same thing? Namely the
problem that lies underneath is the fact that link conversion is based on the
filename the html is saved under. So for either to change the other must too,
and it would be a Good Thing if wget facilitated that somehow. At the moment,
I must parse the html with Perl to do the link conversion myself.

Regards,

Michiel de Boer

Re: Problem with combinations of the -O , -p, and -k parameters in wget

[Steven M. Schweda]

From: Michiel de Boer

> [...] Therefore I use -O to write to a more sensible name.  [...]

   Unfortunately, "-O" does not do name conversion, it simply directs
all the program output to a specified file, and this causes bad behavior
when "-O" is combined with many other options.  Use the "Search" feature
at http://www.mail-archive.com/wget@sunsite.dk/ (for "-O") to find many
similar complaints involving "-O".



   Steven M. Schweda   [EMAIL PROTECTED]
   382 South Warwick Street(+1) 651-699-9818
   Saint Paul  MN  55105-2547

Problem with combinations of the -O , -p, and -k parameters in wget

[Michiel de Boer]

Hi,

Since i'm not subscribed, please CC me the reply/replies.

Currently I have the following problem: I want to store a html page and its
prerequisites locally. The default name under which the html page is stored is
for example "index.php?query+string=foo.html", which IMHO is an ugly filename.

Therefore I use -O to write to a more sensible name. This also facilitates -k,
which converts links such as:
"http://site.org/index.php?query+string=foo#section";

to the local target: "document.html#section"

I also use -nd, so no directory structure is created, and -nc to avoid 
clobbering.

However, when I use -p to get the prerequisites in combination with -O, several
prerequisites are saved under the same name specified in -O, overwriting
eachother. When I don't use -O, the html file downloaded can't be renamed to a
sensible name because internally the converted links point to the old, ugly 
name: "index.php?query+string=foo.html#section"


So to summarize: the problem is I can't combine -O, -p and -k in a meaningful
way to do all this at once:

* save a document to a sensible name, which wget consecutively automatically
  uses for link conversion
* save the prerequisites (unless they've been stored in the same directory
  before)
* convert both links to prerequisites and links to id sections to local.

I could use sed -i or perl -pi -e to patch the html to do the conversion myself,
but IMO this is a suboptimal solution, because wget will be better at this and
therefore less prone to errors.

Any ideas?

Regards,
Michiel de Boer

Re: problem with HTTP mirroring

[Alexander Simon]

Damn.

Did not see this at all. in.tum.de is the short DNS entry of the department, 
sorry for not looking closer first.
It works fine with -H now, of course.
Thank you for your quick answer!

Alex

Am Montag, 11. Juni 2007 20:11:36 schrieben Sie:
>
>As I read the HTML, "i1.pdf" appears to be on a different server:
>
>   http://www2.in.tum.de/~seidl/Courses/SS2007/i1.pdf";>PDF
>
> Perhaps this option would help:
>
>   -H,  --span-hostsgo to foreign hosts when recursive.
>
>
>"wget -h" shows some other potentially useful options under
> "Recursive accept/reject":
>
>   -D,  --domains=LIST  comma-separated list of accepted
> domains. --exclude-domains=LIST  comma-separated list of rejected
> domains. [...]
>
> 
>
>Steven M. Schweda   [EMAIL PROTECTED]
>382 South Warwick Street(+1) 651-699-9818
>Saint Paul  MN  55105-2547

Re: problem with HTTP mirroring

[Steven M. Schweda]

From: Alexander Simon


> When calling
> "wget -A.pdf,.PDF,.doc,.DOC,.java,.class,.JAVA,.CLASS,.zip,.ZIP -m -nH
> -nd -l1 --header="Accept-language: de, en;q=0.8" 
> http://wwwseidl.informatik.tu-muenchen.de/lehre/vorlesungen/SS07/info2/index.php";;
> , wget should load some PDF files (i1.pdf, i2.pdf, i3.pdf, ...) that are 
> linked on this site.

   As I read the HTML, "i1.pdf" appears to be on a different server:

  http://www2.in.tum.de/~seidl/Courses/SS2007/i1.pdf";>PDF

Perhaps this option would help:

  -H,  --span-hostsgo to foreign hosts when recursive.


   "wget -h" shows some other potentially useful options under
"Recursive accept/reject":

  -D,  --domains=LIST  comma-separated list of accepted domains.
   --exclude-domains=LIST  comma-separated list of rejected domains.
[...]



   Steven M. Schweda   [EMAIL PROTECTED]
   382 South Warwick Street(+1) 651-699-9818
   Saint Paul  MN  55105-2547

problem with HTTP mirroring

[Alexander Simon]

Hi.

I wrote a little script to use wget to download my university scripts that are 
published after every lecture.
When calling
"wget -A.pdf,.PDF,.doc,.DOC,.java,.class,.JAVA,.CLASS,.zip,.ZIP -m -nH -nd -l1 
--header="Accept-language: 
de, en;q=0.8" 
http://wwwseidl.informatik.tu-muenchen.de/lehre/vorlesungen/SS07/info2/index.php";
, wget should load some PDF files (i1.pdf, i2.pdf, i3.pdf, ...) that are 
linked on this site. It works this way for other pages. Even a 
plain "wget -m" fails here. It does not even try to download these files.

Did I forgot an option or have I maybe found a bug?
Any ideas?

Thanks, Alex

Re: Problem with --reject option

[Steven M. Schweda]

From: Glenn Nieuwenhuyse

> wget -T 1 -t 1 -r --reject="robots.*" [...]
> 
> I would expect this not to download the robots.txt file, but still it
> does.

   Perhaps because "robots.txt" is a special case, and is not selected
by following links, and so is unaffected by the --reject option.

   A search for "robot" in the manual should reveal this:

  http://www.gnu.org/software/wget/manual/wget.html

robots = on/off
 Specify whether the norobots convention is respected by Wget,
 "on" by default. This switch controls both the /robots.txt and
 the nofollow aspect of the spec. See Robot Exclusion, for more
 details about this. Be sure you know what you are doing before
 turning this off.

So, adding "-e robots=off" to your command might help.



   Steven M. Schweda   [EMAIL PROTECTED]
   382 South Warwick Street(+1) 651-699-9818
   Saint Paul  MN  55105-2547

Problem with --reject option

[Glenn Nieuwenhuyse]

Hi all,

I'm using wget version 1.10.2 under windows and I want wget to avoid
downloading/saving all files called "robots.txt". I'm using the following
command line:

wget -T 1 -t 1 -r --reject="robots.*" http://150.158.230.231:1500

I would expect this not to download the robots.txt file, but still it does.
When I look at the directory where the files are stored once the command has
executed, the robots.txt file is still present.
Can anybody help me out with this one, because I'm probably missing
something trivial here.

Kind regards,

Glenn.

Wget image problem

[KJ]

Hello,
 This is the first time i'm mailing to this list so please forgive  
me if this is the wrong one. I'm having issues with a wget command in  
that it'll download all the site files except for the images. I've  
tried lots of variations but still no result.


The command i am using is:

wget -m -r -l 0 -p --page-requisites --convert-links -k -E –P -- 
directory-prefix=/home/xxx/mirror http://www.thewebsiteaddress.com


I'm hoping someone can tell me what i'm doing wrong.

Thank you for your time.

KJ

Re: wget problem with IBM Http Server2 = apache 2

[Steven M. Schweda]

   In your problem report, I see version numbers for everything but
wget.

   Does adding "-d" to the wget command tell you anything?

   Anything in the Web server logs?



   Steven M. Schweda   [EMAIL PROTECTED]
   382 South Warwick Street(+1) 651-699-9818
   Saint Paul  MN  55105-2547

wget problem with IBM Http Server2 = apache 2

[Stephane Brogi]

Hello,

We are using wget to check if URL applications are up on AIX 5.1 
production systems. 

Since i have upgraded IBMHTTP server from version 1.3.28 having no 
problems on this version with wget (running with websphere 5.1 plugin for 
http Server 1.3.x ) to IBM HTTP Server 2.0.47 (running now with websphere 
5.1 plugin for http Server 2.0.x ) i have an error message when checking 
the URLs.

Problem description:

./wget --server-response --timeout=5 --no-cache  -O - 
"http://qualineostaging.danweb.danet/QualiNeo/Monitor";

--15:23:23--  http://qualineostaging.danweb.danet/QualiNeo/Monitor
   => `-'
Resolving qualineostaging.danweb.danet... 10.254.18.6
Connecting to qualineostaging.danweb.danet|10.254.18.6|:80... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 
  Date: Fri, 02 Feb 2007 14:23:23 GMT
  Server: IBM_HTTP_Server/2.0.47.1 Apache/2.0.47 (Unix)
  Connection: close
  Content-Type: text/html; charset=ISO-8859-1
  Content-Language: en-US
15:23:23 ERROR -1: Malformed status line.

Note: I have No problem using ie6 or GET command on AIX local server.

I will check the websphere plugin trace to see if i can find some errors 
from it. 

Cordialement.

Stéphane Brogi
IBM Global Services, eBusiness Hosting Services
Tel: 04-92-11-55-14 / 365514

Re: problem with downloading when HREF has "../"

[Vladimir Volovich]

On Mon, 03 Apr 2006 17:15:52 +0200
 Mauro Tortonesi <[EMAIL PROTECTED]> wrote:

> > The fix will appear in the next release, 1.11.  Mauro's paragraph you
> > quoted (beginning with "i am going to test and apply your patch later
> > this week") referred to applying the patch to the version control
> > repository, not to the timeframe of releasing 1.11.
> > 
> > It is my understanding that 1.11 will be released within the next
> > couple of months; Mauro might give a more precise date.
> 
> wget 1.11 will definitely be released in the next couple of months, but i
> can't be more precise in this moment. at the beginning, i was thinking
> about adding support for regex, gnunet and fix gnutls support in that
> release. now i am reconsidering whether to delay these new features for
> 1.12 and focus on fixing the incredible number of recently reported bugs
> instead.

it's already 10 months since your promise to release wget 1.11,
and almost a year since i've reported this problem,
but wget 1.11 will hasn't been released. what are your plans for the new
release?

Best,
v.

Re: problem with no-parent option

[Mauro Tortonesi]

Piotr Stankiewicz wrote:

Hello!

I'm using wget for windows version 1.10.2.

I'm trying to download the contents of my photography site. For doing that I
created the following command:

wget --wait
2 --random-wait -r -l7 -H -p --convert-links --html-extension -Dpbase.com --
exclude-domains forum.pbase.com,search.pbase.com --no-parent -e robots=off
http://www.pbase.com/piotrstankiewicz

(I had to use -H option as the photos are placed at other servers that
www.pbase.com)

Unfortunately wget seems to ignore --no-parent option as it starts to
download also www.pbase.com/index.html
www.pbase.com/help.hmtl
documents and others placed in the main directory. I have impression it's
some kind of bug, although I'm not definitely wget expert. Could you try to
verify it please?


hi piotr,

both the url you specified:

http://www.pbase.com/piotrstankiewicz

and the urls you don't want to retrieve:

http://www.pbase.com/help.html
http://www.pbase.com/index.html

reside in the same directory, so the --no-parent option can't help you.

you should probably try to append '/' to the first url:

wget --wait 2 --random-wait -r -l7 -H -p --convert-links 
--html-extension -Dpbase.com --exclude-domains 
forum.pbase.com,search.pbase.com --no-parent -e robots=off

http://www.pbase.com/piotrstankiewicz/

this command should work.


Additionnaly I tried to use the option -R to exclude those files. In such a
case wget downloads those files and deletes it after but it follows the
links from those files (which is unwated by me). I found the information
that it's by design. 


correct. in recursive mode wget retrieves undesired html files to parse 
them for other urls to download, and deletes them after parsing.


But what about introducing any other option precising if the links from 
the unwated documents (specified with -R) should be followed or no (in 
some cases it's not welcome).


i agree. users should be able to tell wget not to retrieve undesired 
html files at all.


--
Aequam memento rebus in arduis servare mentem...

Mauro Tortonesi  http://www.tortonesi.com

University of Ferrara - Dept. of Eng.http://www.ing.unife.it
GNU Wget - HTTP/FTP file retrieval tool  http://www.gnu.org/software/wget
Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net
Ferrara Linux User Group http://www.ferrara.linux.it

RE: wget problem

[Cheng.chungting]

Dear Mauro,

  I have compiled the source and the following error is prompted during running 
"make install". But seems there is no error during running configure.

# make install
cd src && make CC='gcc' CPPFLAGS='' DEFS='-DHAVE_CONFIG_H 
-DSYSTEM_WGETRC=\"/usr/local/etc/wgetrc\" 
-DLOCALEDIR=\"/usr/local/share/locale\"'  CFLAGS='-O2 -Wall -Wno-implicit' 
LDFLAGS='' LIBS='-lssl -lcrypto -ldl -ldl '  prefix='/usr/local' 
exec_prefix='/usr/local' bindir='/usr/local/bin'  infodir='/usr/local/info' 
mandir='/usr/local/man' manext='1' install.bin
gcc -I. -I.-DHAVE_CONFIG_H 
-DSYSTEM_WGETRC=\"/usr/local/etc/wgetrc\" 
-DLOCALEDIR=\"/usr/local/share/locale\" -O2 -Wall -Wno-implicit -c connect.c
In file included from connect.c:41:
/usr/include/sys/socket.h:535: error: static declaration of 'sendfile' follows 
non-static declaration
/usr/include/sys/socket.h:506: error: previous declaration of 'sendfile' was 
here
/usr/include/sys/socket.h:536: error: static declaration of 'sendpath' follows 
non-static declaration
/usr/include/sys/socket.h:508: error: previous declaration of 'sendpath' was 
here
connect.c: In function 'bind_local':
connect.c:457: warning: passing argument 3 of 'getsockname' from incompatible 
pointer type
connect.c: In function 'accept_connection':
connect.c:507: warning: passing argument 3 of 'accept' from incompatible 
pointer type
connect.c: In function 'socket_ip_address':
connect.c:528: warning: passing argument 3 of 'getsockname' from incompatible 
pointer type
connect.c:530: warning: passing argument 3 of 'getpeername' from incompatible 
pointer type
*** Error exit code 1

Stop.
*** Error exit code 1

Stop. 


Regards,
Ting

-Original Message-
From: Mauro Tortonesi [mailto:[EMAIL PROTECTED] 
Sent: 04 January 2007 00:40
To: Chung Ting Cheng (HIT, ODT)
Cc: [EMAIL PROTECTED]
Subject: Re: wget problem

[EMAIL PROTECTED] wrote:
> Dear Mauro,
>  
>   Yes we have installed those prerequsite package but still failed. 
> We have tried the PA-RISC depot and it works although we are using 
> Itanium platform. We have tried another development machine and the 
> result is the same. So I suspect the depot information should be 
> incorrect.

hi cheng,

do you have a compiler on your machine? maybe you should just try to install 
wget from sources.

--
Aequam memento rebus in arduis servare mentem...

Mauro Tortonesi  http://www.tortonesi.com

University of Ferrara - Dept. of Eng.http://www.ing.unife.it
GNU Wget - HTTP/FTP file retrieval tool  http://www.gnu.org/software/wget
Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net
Ferrara Linux User Group http://www.ferrara.linux.it



The message represents the personal views and opinion of the individual sender 
and under no circumstances represents those of Hutchison Port Holdings Limited 
("HPH") or its Group Companies. The shareholders, directors and management of 
HPH and any of its Group Companies accept no responsibility and accordingly 
shall have no liability to any party whatsoever with respect to the contents of 
this message.

This message (including any attachments) is intended only for the use of the 
addressee(s) named above. It may contain information that is PRIVILEGED and 
CONFIDENTIAL and should not be read, copied or otherwise used by any other 
person.

If you are not the intended recipient, you are hereby notified that any use, 
retention, disclosure, copying, printing, forwarding or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please erase all copies of the message and its attachments and notify 
us immediately.
configure: configuring for GNU Wget 1.10
checking build system type... ia64-hp-hpux11.23
checking host system type... ia64-hp-hpux11.23
checking whether make sets $(MAKE)... yes
checking for a BSD-compatible install... ./install-sh -c
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ANSI C... none needed
checking how to run the C preprocessor... gcc -E
checking for egrep... grep -E
checking for AIX... no
checking for gcc option to accept ANSI C...
checking for a sed that does not truncate output... /usr/bin/sed
checking for ld used by gcc... /usr/ccs/bin/ld
checking if the linker (/usr/ccs/bin/ld) is GNU ld... no
checking for /usr/ccs/b

Re: wget problem

[Mauro Tortonesi]

[EMAIL PROTECTED] wrote:
> Dear Mauro,
>  
>   Yes we have installed those prerequsite package but still failed. 
> We have tried the PA-RISC depot and it works although we are using 
> Itanium platform. We have tried another development machine and
> the result is the same. So I suspect the depot information should
> be incorrect.

hi cheng,

do you have a compiler on your machine? maybe you should just try to
install wget from sources.

-- 
Aequam memento rebus in arduis servare mentem...

Mauro Tortonesi  http://www.tortonesi.com

University of Ferrara - Dept. of Eng.http://www.ing.unife.it
GNU Wget - HTTP/FTP file retrieval tool  http://www.gnu.org/software/wget
Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net
Ferrara Linux User Group http://www.ferrara.linux.it

RE: wget problem

[Cheng.chungting]

Dear Mauro,
 
  Yes we have installed those prerequsite package but still failed. We have 
tried the PA-RISC depot and it works although we are using Itanium platform. We 
have tried another development machine and the result is the same. So I suspect 
the depot information should be incorrect.
 
Regards,
Ting



From: Mauro Tortonesi [mailto:[EMAIL PROTECTED]
Sent: 1/3/2007 [Wed] 17:58
To: Chung Ting Cheng (HIT, ODT)
Cc: [EMAIL PROTECTED]
Subject: Re: wget problem



[EMAIL PROTECTED] wrote:
> Dear Sir,
> 
>   I have installed wget 1.10.2 into HP-UX 11.23 from 
> http://hpux.cs.utah.edu/hppd/hpux/Gnu/wget-1.10.2/. Also I have installed the 
> runtime dependency packages like libgcc, gettext, libiconv and openssl. 
> However when I run this and get some testing web content. The following 
> errors is prompted.
> 
> # wget http://10.1.1.15
> --12:46:00--  http://10.1.1.15/
>=> `index.html'
> Connecting to 10.1.1.15:80... connected.
> HTTP request sent, awaiting response... 200 OK
> /usr/lib/hpux32/dld.so: Unsatisfied code symbol '__umodsi3' in load module 
> '/usr/local/bin/wget'.
> Killed
>
>   Could you please help to tell me what's wrong on the issue? Thanks.

hi cheng,

i am not an expert of HP UX, but it seems you have a broken
installation. are you sure you correctly installed all the required
dependencies:

libgcc gettext libiconv openssl

(in particular libgcc)?

--
Aequam memento rebus in arduis servare mentem...

Mauro Tortonesi  http://www.tortonesi.com

University of Ferrara - Dept. of Eng.http://www.ing.unife.it
GNU Wget - HTTP/FTP file retrieval tool  http://www.gnu.org/software/wget
Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net
Ferrara Linux User Group http://www.ferrara.linux.it





The message represents the personal views and opinion of the individual sender 
and under no circumstances represents those of Hutchison Port Holdings Limited 
("HPH") or its Group Companies. The shareholders, directors and management of 
HPH and any of its Group Companies accept no responsibility and accordingly 
shall have no liability to any party whatsoever with respect to the contents of 
this message.

This message (including any attachments) is intended only for the use of the 
addressee(s) named above. It may contain information that is PRIVILEGED and 
CONFIDENTIAL and should not be read, copied or otherwise used by any other 
person.

If you are not the intended recipient, you are hereby notified that any use, 
retention, disclosure, copying, printing, forwarding or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please erase all copies of the message and its attachments and notify 
us immediately.

Re: wget problem

[Mauro Tortonesi]

[EMAIL PROTECTED] wrote:

Dear Sir,
 
  I have installed wget 1.10.2 into HP-UX 11.23 from http://hpux.cs.utah.edu/hppd/hpux/Gnu/wget-1.10.2/. Also I have installed the runtime dependency packages like libgcc, gettext, libiconv and openssl. However when I run this and get some testing web content. The following errors is prompted.
 
# wget http://10.1.1.15

--12:46:00--  http://10.1.1.15/
   => `index.html'
Connecting to 10.1.1.15:80... connected.
HTTP request sent, awaiting response... 200 OK
/usr/lib/hpux32/dld.so: Unsatisfied code symbol '__umodsi3' in load module 
'/usr/local/bin/wget'.
Killed

  Could you please help to tell me what's wrong on the issue? Thanks.


hi cheng,

i am not an expert of HP UX, but it seems you have a broken 
installation. are you sure you correctly installed all the required 
dependencies:


libgcc gettext libiconv openssl

(in particular libgcc)?

--
Aequam memento rebus in arduis servare mentem...

Mauro Tortonesi  http://www.tortonesi.com

University of Ferrara - Dept. of Eng.http://www.ing.unife.it
GNU Wget - HTTP/FTP file retrieval tool  http://www.gnu.org/software/wget
Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net
Ferrara Linux User Group http://www.ferrara.linux.it

wget problem

[Cheng.chungting]

Dear Sir,
 
  I have installed wget 1.10.2 into HP-UX 11.23 from 
http://hpux.cs.utah.edu/hppd/hpux/Gnu/wget-1.10.2/. Also I have installed the 
runtime dependency packages like libgcc, gettext, libiconv and openssl. However 
when I run this and get some testing web content. The following errors is 
prompted.
 
# wget http://10.1.1.15
--12:46:00--  http://10.1.1.15/
   => `index.html'
Connecting to 10.1.1.15:80... connected.
HTTP request sent, awaiting response... 200 OK
/usr/lib/hpux32/dld.so: Unsatisfied code symbol '__umodsi3' in load module 
'/usr/local/bin/wget'.
Killed

  Could you please help to tell me what's wrong on the issue? Thanks.
 

Regards,

Cheng Chung Ting

Systems Engineer

Operations Development & Technology

Hongkong International Terminals

Tel: (852) 2619-7739

Fax: (852) 2419-0165

Email: [EMAIL PROTECTED]

www.hit.com.hk  

 


The message represents the personal views and opinion of the individual sender 
and under no circumstances represents those of Hutchison Port Holdings Limited 
("HPH") or its Group Companies. The shareholders, directors and management of 
HPH and any of its Group Companies accept no responsibility and accordingly 
shall have no liability to any party whatsoever with respect to the contents of 
this message.

This message (including any attachments) is intended only for the use of the 
addressee(s) named above. It may contain information that is PRIVILEGED and 
CONFIDENTIAL and should not be read, copied or otherwise used by any other 
person.

If you are not the intended recipient, you are hereby notified that any use, 
retention, disclosure, copying, printing, forwarding or dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please erase all copies of the message and its attachments and notify 
us immediately.

problem with no-parent option

[Piotr Stankiewicz]

Hello!

I'm using wget for windows version 1.10.2.

I'm trying to download the contents of my photography site. For doing that I
created the following command:

wget --wait
2 --random-wait -r -l7 -H -p --convert-links --html-extension -Dpbase.com --
exclude-domains forum.pbase.com,search.pbase.com --no-parent -e robots=off
http://www.pbase.com/piotrstankiewicz

(I had to use -H option as the photos are placed at other servers that
www.pbase.com)

Unfortunately wget seems to ignore --no-parent option as it starts to
download also www.pbase.com/index.html
www.pbase.com/help.hmtl
documents and others placed in the main directory. I have impression it's
some kind of bug, although I'm not definitely wget expert. Could you try to
verify it please?

Additionnaly I tried to use the option -R to exclude those files. In such a
case wget downloads those files and deletes it after but it follows the
links from those files (which is unwated by me). I found the information
that it's by design. But what about introducing any other option precising
if the links from the unwated documents (specified with -R) should be
followed or no (in some cases it's not welcome).

With best regards

Piotr Stankiewicz


---

http://www.pbase.com/piotrstankiewicz

Re: problem at 4 gigabyte mark downloading wikipedia database file.

[Steven M. Schweda]

From: Jonathan Bazemore:

> [...] I am using wget 1.9 [...] up to about the 4 gig mark [...]

   Try the current version of wget, 1.10.2, which offers large-file
support on many systems, possibly including your unspecified one.

  http://www.gnu.org/software/wget/wget.html



   Steven M. Schweda   [EMAIL PROTECTED]
   382 South Warwick Street(+1) 651-699-9818
   Saint Paul  MN  55105-2547

re: problem at 4 gigabyte mark downloading wikipedia database file.

[Jonathan Bazemore]

Hello,

I am a former computer tech, and I've followed all
instructions closely regarding wget.  I am using wget
1.9 in conjunction with the wgetgui program.  

I have confirmed resumability with smaller binary
files, up to 2.3 gigabytes in size.  

What happens is, that when downloading the wikipedia
database, which is about 8 gigabytes, using wget, the
download proceeds and is resumable up to about the 4
gig mark, then, when I attempt resumption, the
internet connection appears to be working, but the
file just sits there, and doesn't increase in size.

I theorize that the datastream is being corrupted, and
my next step will be to "shave" pieces of the file off
the end, in several megabyte increments, until I reach
the uncorrupted part.  

Please let me know what's going on and why this is
happening at this email address, as I am not a
developer and not currently subscribed to the mailing
list, but I do need to have wget working properly to
get the database.  

Thanks,

Jonathan.

__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: wget POST 'multipart/form-data' problem (win xp sp2)

[Denis Golovan]

"Gerhard Blum" <[EMAIL PROTECTED]> news:[EMAIL PROTECTED]
> hi,
>
> i'm wondering if my mail 2 weeks ago was received...
> or am i too silly to ask correct, or is this problem off topic?
> could someone please be so kind and send a short reply - thanks a lot

  Hi, Gerhard Blum. I am also a new reader of this conference. It seems to 
me everyone is dead here.
Sorry, I can't help you with your problem. 

wget POST 'multipart/form-data' problem (win xp sp2)

[Gerhard Blum]

hi,

i'm wondering if my mail 2 weeks ago was received...
or am i too silly to ask correct, or is this problem off topic?
could someone please be so kind and send a short reply - thanks a lot

the summary of the problem is:
i need wget to send this header
CONTENT_TYPE: multipart/form-data; boundary=-...-202481336020143

here's my complete last mail:
=

i want wget to post a form which contains 3 fields and a file to upload
(enctype="multipart/form-data") at
http://www.pchelpware.com/creator/index.html

i use
wget --post-file C:\form.bin http://sc.uvnc.com/cgi-bin/upload2.pl

and found out that "C:\form.bin" has to look something like:

-202481336020143
Content-Disposition: form-data; name="upload_file"; filename="test.zip"
Content-Type: application/zip

[here comes the binary of the zip-file]
-202481336020143
Content-Disposition: form-data; name="userid"

foo
-202481336020143
Content-Disposition: form-data; name="password"

foobar
-202481336020143
Content-Disposition: form-data; name=".submit"

Upload
-202481336020143--

so far, so good.

but it doesn't work because wget is sending the header
CONTENT_TYPE: application/x-www-form-urlencoded

while the server of course expects
CONTENT_TYPE: multipart/form-data; boundary=-...-202481336020143

but if i use the --header option, this header becomes
HTTP_CONTENT_TYPE: multipart/form-data; boundary=-...-202481336020143
and wget is still sending
CONTENT_TYPE: application/x-www-form-urlencoded

how can i force wget to send the needed CONTENT_TYPE?
or what else am i doing wrong?

regards,
--
Gerhard Blum
mailto:[EMAIL PROTECTED]

RE: ERROR 500 problem

[Sandhu, Ranjit]

Maybe the server has some sort of limitations to hits from the same IP
address over a time period.  1400 pages is a lot, maybe they got mad at
you and send you all 500's from then on :)

Ranjit Sandhu
703.803.1755
SRA

-Original Message-
From: Yoav Atzmony [mailto:[EMAIL PROTECTED] 
Sent: Monday, December 11, 2006 12:24 PM
To: [EMAIL PROTECTED]
Subject: ERROR 500 problem

Hi, I hope someone can shed light on this problem.

I am trying to crawl a particular site, and am getting strange results.
I had crawled it successfully in the past but lately I only am able to
crawl about 1400 of the 8000 pages.  I am constantly getting (as
reported in a verbose log file):
HTTP request sent, awaiting response... 500 Internal Server Error
11:04:18 ERROR 500: Internal Server Error.

This error happens intermitently on some pages during the first 1400
pages, i.e.
http://www.ryland.com/find-your-new-home/29-northern-kentucky/1115-claib
orne/11777-shenandoah.html

But then this is what happens in the log file, and subsequently ALL
files receive the error 500 (as shown in a non-verbose log file):
WARNING: Certificate verification error for www.ryland.com: unable to
get local issuer certificate
16:58:03
URL:https://www.ryland.com/home/contact-us/29-1361-community-and-floor-p
lan-information.html
[257903/257903] ->
"files/www.ryland.com/home/contact-us/29-1361-community-and-floor-plan-i
nformation.html"
[1]
16:58:19 URL:http://www.ryland.com/home/29-1034-contact-us.html
[115640/115640] -> "files/www.ryland.com/home/29-1034-contact-us.html"
[1]
http://www.ryland.com/find-your-new-home/29-northern-kentucky/1034-frenc
[EMAIL PROTECTED]/driving-directions.html:
16:58:23 ERROR 500: Internal Server Error.
http://www.ryland.com/find-your-new-home/29-northern-kentucky/1034-frenc
h-quarter-orleans/11256-summit.html:
16:58:27 ERROR 500: Internal Server Error.

Now if I was to call wget only on the page that failed with ERROR 500,
it would crawl just fine.  Here are settings I am using:
wget  www.ryland.com -o LogRyland3.txt -t 5 --random-wait -v

I have run wget numerous times on this site, and I receive ERROR 500 on
different pages before it reaches the point where all pages fail from
then on, as shown above.

And I have appended the INI file which has more settings (below).
Again, crawling this site used to work fine.  And crawling failed pages
works when I crawl them individually. Any help would be GREATLY
appreciated!

INI FILE STARTS HERE---
# Rewrote the wgetrc / wget.ini file from scratch, based on the manual
for version 1.9

logfile = log.txt
tries = 1
timeout = 30
wait = 1
randomwait = on
quota = 5000m
restrict_file_names = windows
add_hostdir = on
span_hosts = off
dir_prefix = files
cache = off
recursive = on
use_proxy = off
robots = off
verbose = off
keep-session-cookies = on
save-cookies = sw_cookies.txt
check_certificate = off
#reclevel = 7

reject =
GIF,jpg,JPG,jpeg,JPEG,bmp,BMP,pdf,PDF,css,CSS,js,JS,mpeg,MPEG,mov,MOV,av
i,AVI,wmv,WMV,doc,DOC,ppt,PPT,csv,CSV,xls,XLS,txt,TXT,png,PNG,ra,RA,ram,
RAM,tif,TIF,zip,ZIP,rar,RAR,class,CLASS,swf,SWF,pl,xml,XML,mp3,MP3,sid,S
ID,ivr,IVR,psd,PSD,rft,RTF,dwf,DWF,abk,acl,acm,acp,act,acv,ad,adb,add,ad
m,adp,adr,af2,af3,afm,ai,aif,alb,all,ams,anc,ani,ans,api,apr,aps,arc,arj
,art,asa,asc,asd,asf,asm,ast,asx,att,avi,awd,b4,bak,bas,bat,bfc,bg,bi,bi
f,bin,bk,bks,bm1,bmk,bmp,brx,bs1,bsp,btm,cab,cal,cas,cat,cb,ccb,ccf,cch,
ccm,cda,cdf,cdi,cdr,cdt,cdx,cel,cfb,cfg,cgm,ch,chk,chp,cil,cim,cin,ck1,c
k2,ck3,ck4,ck5,ck6,cla,clp,cls,cmd,cmf,cmp,cmv,cnf,cnm,cnq,cnt,cob,cod,c
om,cpd,cpe,cpi,cpl,cpp,cpr,cpt,cpx,crd,crp,crt,csc,csp,css,csv,ct,ctl,cu
e,cur,cut,cv,cwk,cws,cxx,dat,dbf,dbx,dcr,dcs,dcx,ddf,def,der,dib,dic,dif
,dir,diz,dlg,dll,dmf,dmg,doc,dot,dpr,drv,drw,dsg,dsm,dsp,dsq,dsw,dwg,dxf
,emf,enc,eps,er1,erx,evy,ewl,exe,f77,f90,far,fav,fax,fh3,fif,fit,flc,fli
,flt,fmb,fmt,fmx,fog,fon,for,fot,fp,fp1,fp3,fpx,frm,frx,gal,gcp,ged,gem,
gen,gfc,gfi,gfx,gid,gif,gim,gix,gna,gnx,gra,grd,grp,gt2,gtk,gwx,gwz,gz,h
ed,hel,hex,hgl,hlp,hog,hpj,hpp,hqx,hst,ht,htx,ica,icb,icm,ico,idd,idq,if
f,igf,iif,ima,img,inc,inf,ini,inp,ins,iso,isp,isu,it,iw,jar,jav,jbf,jff,
jif,jmp,jn1,jpe,jpg,js,jtf,kdc,kfx,kye,lbm,ldb,leg,lha,lib,lis,log,lpd,l
rc,lst,lwo,lwp,lzh,lzs,m3d,mad,maf,mak,mam,map,maq,mar,mas,mat,max,maz,m
b1,mcc,mcs,mcw,mda,mdb,mde,mdl,mdn,mdw,mdz,med,mer,met,mi,mic,mid,mmf,mm
m,mod,mov,mp3,mpe,mpg,mpp,msg,msi,msn,msp,mtm,mus,mvb,mwp,nap,ncb,nsf,ns
t,ntf,obd,obj,obz,ocx,ofn,oft,okt,olb,ole,opt,or2,or3,org,p10,p65,pab,pa
k,pal,pat,pbk,pbm,pcd,pcl,pcs,pct,pcx,pdf,pdq,pfa,pfb,pfc,pfm,pgl,pgm,pi
c,pif,pig,pin,pix,pj,pkg,pl,plt,pm5,pm6,png,pnt,pot,pp4,ppa,ppm,pps,ppt,
pre,prf,prn,prs,prz,ps,psd,pst,ptm,pub,pwd,pwz,pxl,qad,qbw,qdt,qlb,qry,q
t,qtm,qxd,ra,ram,ras,raw,rc,rec,reg,res,rft,rle,rm,rmi,rov,rpt,rtf,rtm,s
3m,sam,sav,sc2,scc,scd,sch,scn,scp,scr,sct,sdl,sdr,sdt,sea,sep,shb,shg,s
hs,shw,sit,slk,snd,sqc,sqr,sty,svx,sys,t2t,tar,taz,tex,tga,tgz,the,thn,t
if,tig,tlb,tmp,top,

ERROR 500 problem

[Yoav Atzmony]

Hi, I hope someone can shed light on this problem.

I am trying to crawl a particular site, and am getting strange
results.  I had crawled it successfully in the past but lately I only
am able to crawl about 1400 of the 8000 pages.  I am constantly
getting (as reported in a verbose log file):
HTTP request sent, awaiting response... 500 Internal Server Error
11:04:18 ERROR 500: Internal Server Error.

This error happens intermitently on some pages during the first 1400 pages, i.e.
http://www.ryland.com/find-your-new-home/29-northern-kentucky/1115-claiborne/11777-shenandoah.html

But then this is what happens in the log file, and subsequently ALL
files receive the error 500 (as shown in a non-verbose log file):
WARNING: Certificate verification error for www.ryland.com: unable to
get local issuer certificate
16:58:03 
URL:https://www.ryland.com/home/contact-us/29-1361-community-and-floor-plan-information.html
[257903/257903] ->
"files/www.ryland.com/home/contact-us/29-1361-community-and-floor-plan-information.html"
[1]
16:58:19 URL:http://www.ryland.com/home/29-1034-contact-us.html
[115640/115640] -> "files/www.ryland.com/home/29-1034-contact-us.html"
[1]
http://www.ryland.com/find-your-new-home/29-northern-kentucky/[EMAIL 
PROTECTED]/driving-directions.html:
16:58:23 ERROR 500: Internal Server Error.
http://www.ryland.com/find-your-new-home/29-northern-kentucky/1034-french-quarter-orleans/11256-summit.html:
16:58:27 ERROR 500: Internal Server Error.

Now if I was to call wget only on the page that failed with ERROR 500,
it would crawl just fine.  Here are settings I am using:
wget  www.ryland.com -o LogRyland3.txt -t 5 --random-wait -v

I have run wget numerous times on this site, and I receive ERROR 500
on different pages before it reaches the point where all pages fail
from then on, as shown above.

And I have appended the INI file which has more settings (below).
Again, crawling this site used to work fine.  And crawling failed
pages works when I crawl them individually. Any help would be GREATLY
appreciated!

INI FILE STARTS HERE---
# Rewrote the wgetrc / wget.ini file from scratch, based on the manual
for version 1.9

logfile = log.txt
tries = 1
timeout = 30
wait = 1
randomwait = on
quota = 5000m
restrict_file_names = windows
add_hostdir = on
span_hosts = off
dir_prefix = files
cache = off
recursive = on
use_proxy = off
robots = off
verbose = off
keep-session-cookies = on
save-cookies = sw_cookies.txt
check_certificate = off
#reclevel = 7

reject = 
GIF,jpg,JPG,jpeg,JPEG,bmp,BMP,pdf,PDF,css,CSS,js,JS,mpeg,MPEG,mov,MOV,avi,AVI,wmv,WMV,doc,DOC,ppt,PPT,csv,CSV,xls,XLS,txt,TXT,png,PNG,ra,RA,ram,RAM,tif,TIF,zip,ZIP,rar,RAR,class,CLASS,swf,SWF,pl,xml,XML,mp3,MP3,sid,SID,ivr,IVR,psd,PSD,rft,RTF,dwf,DWF,abk,acl,acm,acp,act,acv,ad,adb,add,adm,adp,adr,af2,af3,afm,ai,aif,alb,all,ams,anc,ani,ans,api,apr,aps,arc,arj,art,asa,asc,asd,asf,asm,ast,asx,att,avi,awd,b4,bak,bas,bat,bfc,bg,bi,bif,bin,bk,bks,bm1,bmk,bmp,brx,bs1,bsp,btm,cab,cal,cas,cat,cb,ccb,ccf,cch,ccm,cda,cdf,cdi,cdr,cdt,cdx,cel,cfb,cfg,cgm,ch,chk,chp,cil,cim,cin,ck1,ck2,ck3,ck4,ck5,ck6,cla,clp,cls,cmd,cmf,cmp,cmv,cnf,cnm,cnq,cnt,cob,cod,com,cpd,cpe,cpi,cpl,cpp,cpr,cpt,cpx,crd,crp,crt,csc,csp,css,csv,ct,ctl,cue,cur,cut,cv,cwk,cws,cxx,dat,dbf,dbx,dcr,dcs,dcx,ddf,def,der,dib,dic,dif,dir,diz,dlg,dll,dmf,dmg,doc,dot,dpr,drv,drw,dsg,dsm,dsp,dsq,dsw,dwg,dxf,emf,enc,eps,er1,erx,evy,ewl,exe,f77,f90,far,fav,fax,fh3,fif,fit,flc,fli,flt,fmb,fmt,fmx,fog,fon,for,fot,fp,fp1,fp3,fpx,frm,frx,gal,gcp,ged,gem,gen,gfc,gfi,gfx,gid,gif,gim,gix,gna,gnx,gra,grd,grp,gt2,gtk,gwx,gwz,gz,hed,hel,hex,hgl,hlp,hog,hpj,hpp,hqx,hst,ht,htx,ica,icb,icm,ico,idd,idq,iff,igf,iif,ima,img,inc,inf,ini,inp,ins,iso,isp,isu,it,iw,jar,jav,jbf,jff,jif,jmp,jn1,jpe,jpg,js,jtf,kdc,kfx,kye,lbm,ldb,leg,lha,lib,lis,log,lpd,lrc,lst,lwo,lwp,lzh,lzs,m3d,mad,maf,mak,mam,map,maq,mar,mas,mat,max,maz,mb1,mcc,mcs,mcw,mda,mdb,mde,mdl,mdn,mdw,mdz,med,mer,met,mi,mic,mid,mmf,mmm,mod,mov,mp3,mpe,mpg,mpp,msg,msi,msn,msp,mtm,mus,mvb,mwp,nap,ncb,nsf,nst,ntf,obd,obj,obz,ocx,ofn,oft,okt,olb,ole,opt,or2,or3,org,p10,p65,pab,pak,pal,pat,pbk,pbm,pcd,pcl,pcs,pct,pcx,pdf,pdq,pfa,pfb,pfc,pfm,pgl,pgm,pic,pif,pig,pin,pix,pj,pkg,pl,plt,pm5,pm6,png,pnt,pot,pp4,ppa,ppm,pps,ppt,pre,prf,prn,prs,prz,ps,psd,pst,ptm,pub,pwd,pwz,pxl,qad,qbw,qdt,qlb,qry,qt,qtm,qxd,ra,ram,ras,raw,rc,rec,reg,res,rft,rle,rm,rmi,rov,rpt,rtf,rtm,s3m,sam,sav,sc2,scc,scd,sch,scn,scp,scr,sct,sdl,sdr,sdt,sea,sep,shb,shg,shs,shw,sit,slk,snd,sqc,sqr,sty,svx,sys,t2t,tar,taz,tex,tga,tgz,the,thn,tif,tig,tlb,tmp,top,trm,trn,ttf,twf,tww,tx8,txt,udf,ult,url,use,vbp,vbw,vbx,vcf,vda,vi,vlb,voc,vp,vsd,vst,vsw,vxd,wad,wav,wb1,wb2,wbk,wbl,wcm,wdb,web,wgp,wid,wil,wiz,wk1,wk3,wk4,wks,wlf,wll,wmf,wow,wp,wp4,wp5,wp6,wpd,wpg,wps,wpt,wpw,wq1,wq2,wri,wrl,ws1,ws2,ws3,ws4,ws5,ws6,ws7,wsd,wvl,xar,xla,xlb,xlc,xld,xlk,xlm,xls,xlt,xlv,xlw,xm,xr1,xy3,xy4,xyp,xyw,yal,ybk,zip,zoo,
ABK,ACL,ACM,ACP,ACT,ACV,AD,ADB,ADD,ADM,ADP,ADR,AF2,AF3,AFM,AI,AIF,ALB,ALL,A

POST multipart/form-data Problem

[Gerhard Blum]

hi,

i want wget to post a form which contains 3 fields and a file to upload 
(enctype="multipart/form-data")

http://www.pchelpware.com/creator/index.html

i found out, using the option
--post-file "C:\form.bin"

that C:\form.bin has to look something like:

-202481336020143
Content-Disposition: form-data; name="upload_file"; filename="test.zip"
Content-Type: application/zip

[here comes the binary of the zip-file]
-202481336020143
Content-Disposition: form-data; name="userid"

foo
-202481336020143
Content-Disposition: form-data; name="password"

foobar
-202481336020143
Content-Disposition: form-data; name=".submit"

Upload
-202481336020143--

so far, so good.

but it only works half because wget is sending this with the header:
CONTENT_TYPE: application/x-www-form-urlencoded

while the server of course only accepts:
CONTENT_TYPE: multipart/form-data; boundary=-...-202481336020143

if i use the --header option, this header becomes:
HTTP_CONTENT_TYPE: multipart/form-data; boundary=...
and wget still uses
CONTENT_TYPE: application/x-www-form-urlencoded

how can i force wget to send the correct CONTENT_TYPE?
or what else am i doing wrong?

regards, gerhard

wget redirect handling problem

[Eyal Udassin]

Hello,
 
I'm trying to download the site   www.shamir7.com
with wget with the following parameters:
-v -r -l5
 
Unfortunately, wget stops at the first request. I can see that it gets this
response from the server:
--
HTTP/1.1 302
Cache-Control: private
Content-Length: 9
Location: /default.aspx
Server: Microsoft-IIS/6.0
MicrosoftSharePointTeamServices: 6.0.2.6568
IWS_IPOTRACK: PB107
X-Powered-By: ASP.NET
Public-Extension:  
http://schemas.microsoft.com/repl-2
Date: Mon, 30 Oct 2006 22:47:24 GMT
Connection: close
 
302 FOUND
 
--
For some reason wget is not able to follow the link to default.aspx and
stops.
Am I doing something wrong or is this a bug?
 
Thanks in advance,
Eyal.

Re: wget 1.11 beta1 another time-stamping problem

[Jochen Roderburg]

Zitat von Jochen Roderburg <[EMAIL PROTECTED]>:

> In the time-stamping mode wget always issued first a HEAD request when there
> was
> a local file, and later a GET request when after inspecting the HEAD outpout
> it
> found out that it should do so.
>
> The wget 1.11 now *always* does the HEAD request, so this problem may be a
> little related to the other just-repaired problem.

Now I even stumbled over a case where this behaviour leads to an error, namely
when the server doesn't like the HEAD request and responds with an error.
Therefore my additional question: Is this HEAD request intended or is it an
error? Has it perhaps to do with the new "Content-Disposition" stuff?

I encountered the new problem when downloading a new Eudora Beta. This is
delivered via a cgi which makes a redirection to the real file link.
A HEAD request for the original link is answered with "500 Server Error".


wget.111b1 -d http://www.eudora.com/cgi-bin/export.cgi?productid=EUDORA_win_7106

DEBUG output created by Wget 1.11-beta-1 on linux-gnu.

--10:53:19--  http://www.eudora.com/cgi-bin/export.cgi?productid=EUDORA_win_7106
Resolving www.eudora.com... 199.106.114.30
Caching www.eudora.com => 199.106.114.30
Connecting to www.eudora.com|199.106.114.30|:80... connected.
Created socket 3.
Releasing 0x08086920 (new refcount 1).

---request begin---
HEAD /cgi-bin/export.cgi?productid=EUDORA_win_7106 HTTP/1.0
User-Agent: Wget/1.11-beta-1
Accept: */*
Host: www.eudora.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 500 Server Error
Server: Netscape-Enterprise/6.0
Date: Wed, 30 Aug 2006 08:53:20 GMT
Content-length: 305
Content-type: text/html
Connection: keep-alive

---response end---
500 Server Error
Registered socket 3 for persistent reuse.
10:53:21 ERROR 500: Server Error.



wget.1102 -d http://www.eudora.com/cgi-bin/export.cgi?productid=EUDORA_win_7106

DEBUG output created by Wget 1.10.2 on linux-gnu.

--10:51:22--  http://www.eudora.com/cgi-bin/export.cgi?productid=EUDORA_win_7106
   => `export.cgi?productid=EUDORA_win_7106'
Resolving www.eudora.com... 199.106.114.30
Caching www.eudora.com => 199.106.114.30
Connecting to www.eudora.com|199.106.114.30|:80... connected.
Created socket 3.
Releasing 0x08084e60 (new refcount 1).

---request begin---
GET /cgi-bin/export.cgi?productid=EUDORA_win_7106 HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: www.eudora.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 302 Moved Temporarily
Server: Netscape-Enterprise/6.0
Date: Wed, 30 Aug 2006 08:51:21 GMT
Location:
http://www.eudora.com/download/eudora/windows/7.1/beta/Eudora_7.1.0.6_beta.exe
Content-length: 0
Connection: keep-alive

---response end---
302 Moved Temporarily
Registered socket 3 for persistent reuse.
Location:
http://www.eudora.com/download/eudora/windows/7.1/beta/Eudora_7.1.0.6_beta.exe
[
following]
Skipping 0 bytes of body: [] done.
--10:51:22-- 
http://www.eudora.com/download/eudora/windows/7.1/beta/Eudora_7.1.0.6_beta.e
xe
   => `Eudora_7.1.0.6_beta.exe'
Reusing existing connection to www.eudora.com:80.
Reusing fd 3.

---request begin---
GET /download/eudora/windows/7.1/beta/Eudora_7.1.0.6_beta.exe HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: www.eudora.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Server: Netscape-Enterprise/6.0
Date: Wed, 30 Aug 2006 08:51:21 GMT
Content-type: application/octet-stream
Last-modified: Mon, 28 Aug 2006 21:29:37 GMT
Content-length: 17403352
Accept-ranges: bytes
Connection: keep-alive

---response end---
200 OK
Length: 17,403,352 (17M) [application/octet-stream]

100%[==>] 17,403,352   322.84K/s   
ETA 00:00

10:52:29 (256.51 KB/s) - `Eudora_7.1.0.6_beta.exe' saved [17403352/17403352]


Regards, J.Roderburg

wget 1.11 beta1 another time-stamping problem

[Jochen Roderburg]

Unfortunately the time-stamping saga continues  ;-)

In the time-stamping mode wget always issued first a HEAD request when there was
a local file, and later a GET request when after inspecting the HEAD outpout it
found out that it should do so.

The wget 1.11 now *always* does the HEAD request, so this problem may be a
little related to the other just-repaired problem.
The error, however, is that it now uses the time-data from the HEAD output as
timestamp for the local file and not the time-data from the GET request.

This could theoretically even be a problem with a direct site-transfer, when the
remote file changes between the HEAD and the GET, but the pratical case where it
occured is in connection with a proxy-cache. When the proxy-cache has a cached
file-copy which is older than the file on the orginal site, the HEAD delivers
the data from the cached file, but when upon the GET the proxy itself decides
to retrieve the newer version (or is forced to do that with the wget --no-cache
option) we get the discrepancy: we get the *newer* file downloaded but with the
*older* time-stamp.

And a real-life example to illustrate the issue:


HEAD -p http://wwwcache.uni-koeln.de:8080
http://www.extractnow.com/extractnow.exe

200 OK
Date: Wed, 23 Aug 2006 12:15:42 GMT
Accept-Ranges: bytes
Age: 165431
ETag: "98caa15d43c4c61:4da"
Server: Microsoft-IIS/6.0
Content-Length: 981504
Content-Type: application/octet-stream
Last-Modified: Sun, 20 Aug 2006 10:28:23 GMT
Client-Date: Sun, 27 Aug 2006 10:03:17 GMT
Client-Response-Num: 1
Proxy-Connection: close
X-Cache: HIT from wwwcache.uni-koeln.de
X-Powered-By: ASP.NET

HEAD http://www.extractnow.com/extractnow.exe

200 OK
Date: Sun, 27 Aug 2006 10:05:10 GMT
Accept-Ranges: bytes
ETag: "4e9432fc57c9c61:4da"
Server: Microsoft-IIS/6.0
Content-Length: 983005
Content-Type: application/octet-stream
Last-Modified: Sat, 26 Aug 2006 21:38:35 GMT
Client-Date: Sun, 27 Aug 2006 10:05:09 GMT
Client-Response-Num: 1
X-Powered-By: ASP.NET


The two HEAD (HEAD utility from the lwp-package) requests show that the
cache has a file version from 20 Aug 2006 and the site has a file version from
26 Aug 2006


wget.111b1 -d http://www.extractnow.com/extractnow.exe

DEBUG output created by Wget 1.11-beta-1 on linux-gnu.

--12:06:18--  http://www.extractnow.com/extractnow.exe
Resolving wwwcache.uni-koeln.de... 134.95.19.61
Caching wwwcache.uni-koeln.de => 134.95.19.61
Connecting to wwwcache.uni-koeln.de|134.95.19.61|:8080... connected.
Created socket 3.
Releasing 0x08086950 (new refcount 1).

---request begin---
HEAD http://www.extractnow.com/extractnow.exe HTTP/1.0
User-Agent: Wget/1.11-beta-1
Accept: */*
Host: www.extractnow.com

---request end---
Proxy request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Content-Length: 981504
Content-Type: application/octet-stream
Last-Modified: Sun, 20 Aug 2006 10:28:23 GMT
Accept-Ranges: bytes
ETag: "98caa15d43c4c61:4da"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Wed, 23 Aug 2006 12:15:42 GMT
Age: 165612
X-Cache: HIT from wwwcache.uni-koeln.de
Proxy-Connection: close

---response end---
200 OK
Length: 981504 (958K) [application/octet-stream]
Closed fd 3
--12:06:18--  http://www.extractnow.com/extractnow.exe
Found wwwcache.uni-koeln.de in host_name_addresses_map (0x8086950)
Connecting to wwwcache.uni-koeln.de|134.95.19.61|:8080... connected.
Created socket 3.
Releasing 0x08086950 (new refcount 1).

---request begin---
GET http://www.extractnow.com/extractnow.exe HTTP/1.0
User-Agent: Wget/1.11-beta-1
Accept: */*
Host: www.extractnow.com

---request end---
Proxy request sent, awaiting response...
---response begin---
HTTP/1.0 200 OK
Content-Length: 983005
Content-Type: application/octet-stream
Last-Modified: Sat, 26 Aug 2006 21:38:35 GMT
Accept-Ranges: bytes
ETag: "4e9432fc57c9c61:4da"
Server: Microsoft-IIS/6.0
X-Powered-By: ASP.NET
Date: Sun, 27 Aug 2006 10:06:20 GMT
X-Cache: MISS from wwwcache.uni-koeln.de
Proxy-Connection: close

---response end---
200 OK
Length: 983005 (960K) [application/octet-stream]
Saving to: `extractnow.exe'

100%[>] 983,005
265K/s   in 3.6
s

Closed fd 3
12:06:22 (265 KB/s) - `extractnow.exe' saved [983005/983005]


And the result on the local disk:

...983005 20.08.2006 12:28 extractnow.exe

The filesizes show that the newer version was downloaded but it got the
time-stamp of the older one.


Btw, a quick work-around is to download it a second time, the cache has now the
newer file with newer file data, wget requests it new because it now sees the
local file as older, the file is retrieved directly from the cache and gets the
correct time-stamp now  ;-)


Best regards,
Jochen Roderburg

wget 1.11 beta1 SSL configuration problem

[Jochen Roderburg]

There seems to a configure problem with the options to specify the directories
where the SSL installation resides.

I have the SSL that I want in /usr/local and in wget 1.10.2 the configure option
--with-libssl-prefix=/usr/local worked.

Part of configure output:

checking for libssl... yes
checking how to link with libssl... /usr/local/lib/libssl.so
/usr/local/lib/libcrypto.so -Wl,-rpath
-Wl,/usr/local/lib
configure: compiling in support for SSL

and in the Makefiles I have:

LIBS = -lintl -ldl -lrt  /usr/local/lib/libssl.so /usr/local/lib/libcrypto.so
-Wl,-rpath -Wl,/usr/local/lib

With wget-1.11-beta-1 however I get the configure output:

checking how to link with libssl... -lssl -lcrypto
configure: compiling in support for SSL via OpenSSL

and

LIBS = -lintl -ldl -lrt  -lssl -lcrypto

Somehow the specified directory /usr/local seems to be ignored  ;-)
I *have* a SSL under /usr from the base system installation, but the current and
correctly configured version that I actually use is in /usr/local.

J.Roderburg

URLs such as "site.com/folder/bob.php?a=1" and robots.txt problem

[henka]

Hello,

Given the following robots.txt file:

User-agent: *
Disallow: /folder/bob.php?
...

One would expect that if wget tries to download a link to
/folder/bob.php?a=1 that it would exclude it because of the robots rule
line - but it doesn't (my reading of the RFC indicates that it should
exclude it).

If you remove the "?" from the robots rule, then it works as expected.
However, this is not what you would expect - let's say you *want* the
/folder/bob.php file to be downloaded, but not all the other dynamic
content (eg, /folder/bob.php?a=1 and /folder/bob.php?a=100, etc).

I've been searching around but cannot find an obvious answer:  is this
behaviour expected, or is this a bug?

Thanks
Henka

RE: wget problem

[Paula]

Thanks a lot Pat.
I will try it out. 
Paula

-Original Message-
From: Willener, Pat [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, June 27, 2006 8:40 PM
To: Paula; [EMAIL PROTECTED]
Subject: [SPAM] RE: [SPAM] RE: wget problem

Yes, wget is not a DOS command, but you can download the Windows version
from
http://xoomer.alice.it/hherold/

Regards,
Pat 

-Original Message-
From: Paula [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, June 28, 2006 12:37 PM
To: Willener, Pat; [EMAIL PROTECTED]
Subject: RE: [SPAM] RE: wget problem

Thanks Pat, yes I understand, but I thought that wget was a built-in DOS
command, but it is not. 
So I am trying to find where I can download it from.
I am looking at your GNU.org website.
Any suggestion or direction you can give me about the download?
Thanks..
Paula 

-Original Message-
From: Willener, Pat [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, June 27, 2006 8:26 PM
To: [EMAIL PROTECTED]
Cc: Paula
Subject: [SPAM] RE: wget problem

.OR... specify the full path name - C:\> "C:\Program Files\wget\wget"
parameters

.OR... add the path to the %PATH% - C:\> PATH %PATH%;C:\Program Files\wget

.OR... drag and drop a shortcut to the C: root

-Original Message-
From: Sandhu, Ranjit [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, June 28, 2006 2:00 AM
To: Paula; [EMAIL PROTECTED]
Subject: RE: wget problem

Paula,

Go to the directory where you have WGET installed, ie. The directory in
which wget.exe is located.  The error is saying windows cannot find the
"wget" program.

Ranjit Sandhu
SRA

-Original Message-
From: Paula [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, June 27, 2006 12:31 PM
To: [EMAIL PROTECTED]
Subject: wget problem

Hello GNU, when I type in C:\>wget http://.com I get the
message:
Wget is not recognized as an internal or external command, operable
program or batch file.
What am I doing wrong?
I start out from the C prompt.
Thank you.
Paula Van Berkom

RE: [SPAM] RE: wget problem

[Willener, Pat]

Yes, wget is not a DOS command, but you can download the Windows version from
http://xoomer.alice.it/hherold/

Regards,
Pat 

-Original Message-
From: Paula [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, June 28, 2006 12:37 PM
To: Willener, Pat; [EMAIL PROTECTED]
Subject: RE: [SPAM] RE: wget problem

Thanks Pat, yes I understand, but I thought that wget was a built-in DOS
command, but it is not. 
So I am trying to find where I can download it from.
I am looking at your GNU.org website.
Any suggestion or direction you can give me about the download?
Thanks..
Paula 

-Original Message-
From: Willener, Pat [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, June 27, 2006 8:26 PM
To: [EMAIL PROTECTED]
Cc: Paula
Subject: [SPAM] RE: wget problem

..OR... specify the full path name - C:\> "C:\Program Files\wget\wget"
parameters

..OR... add the path to the %PATH% - C:\> PATH %PATH%;C:\Program Files\wget

..OR... drag and drop a shortcut to the C: root

-Original Message-
From: Sandhu, Ranjit [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, June 28, 2006 2:00 AM
To: Paula; [EMAIL PROTECTED]
Subject: RE: wget problem

Paula,

Go to the directory where you have WGET installed, ie. The directory in
which wget.exe is located.  The error is saying windows cannot find the
"wget" program.

Ranjit Sandhu
SRA

-Original Message-
From: Paula [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, June 27, 2006 12:31 PM
To: [EMAIL PROTECTED]
Subject: wget problem

Hello GNU, when I type in C:\>wget http://.com I get the
message:
Wget is not recognized as an internal or external command, operable
program or batch file.
What am I doing wrong?
I start out from the C prompt.
Thank you.
Paula Van Berkom

RE: [SPAM] RE: wget problem

[Paula]

Thanks Pat, yes I understand, but I thought that wget was a built-in DOS
command, but it is not. 
So I am trying to find where I can download it from.
I am looking at your GNU.org website.
Any suggestion or direction you can give me about the download?
Thanks..
Paula 

-Original Message-
From: Willener, Pat [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, June 27, 2006 8:26 PM
To: [EMAIL PROTECTED]
Cc: Paula
Subject: [SPAM] RE: wget problem

..OR... specify the full path name - C:\> "C:\Program Files\wget\wget"
parameters

..OR... add the path to the %PATH% - C:\> PATH %PATH%;C:\Program Files\wget

..OR... drag and drop a shortcut to the C: root

-Original Message-
From: Sandhu, Ranjit [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, June 28, 2006 2:00 AM
To: Paula; [EMAIL PROTECTED]
Subject: RE: wget problem

Paula,

Go to the directory where you have WGET installed, ie. The directory in
which wget.exe is located.  The error is saying windows cannot find the
"wget" program.

Ranjit Sandhu
SRA

-Original Message-
From: Paula [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, June 27, 2006 12:31 PM
To: [EMAIL PROTECTED]
Subject: wget problem

Hello GNU, when I type in C:\>wget http://.com I get the
message:
Wget is not recognized as an internal or external command, operable
program or batch file.
What am I doing wrong?
I start out from the C prompt.
Thank you.
Paula Van Berkom

RE: wget problem

[Willener, Pat]

...OR... specify the full path name - C:\> "C:\Program Files\wget\wget" 
parameters

...OR... add the path to the %PATH% - C:\> PATH %PATH%;C:\Program Files\wget

...OR... drag and drop a shortcut to the C: root

-Original Message-
From: Sandhu, Ranjit [mailto:[EMAIL PROTECTED] 
Sent: Wednesday, June 28, 2006 2:00 AM
To: Paula; [EMAIL PROTECTED]
Subject: RE: wget problem

Paula,

Go to the directory where you have WGET installed, ie. The directory in
which wget.exe is located.  The error is saying windows cannot find the
"wget" program.

Ranjit Sandhu
SRA

-Original Message-
From: Paula [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, June 27, 2006 12:31 PM
To: [EMAIL PROTECTED]
Subject: wget problem

Hello GNU, when I type in C:\>wget http://.com I get the
message:
Wget is not recognized as an internal or external command, operable
program or batch file.
What am I doing wrong?
I start out from the C prompt.
Thank you.
Paula Van Berkom

RE: wget problem

[Sandhu, Ranjit]

Paula,

Go to the directory where you have WGET installed, ie. The directory in
which wget.exe is located.  The error is saying windows cannot find the
"wget" program.

Ranjit Sandhu
SRA

-Original Message-
From: Paula [mailto:[EMAIL PROTECTED] 
Sent: Tuesday, June 27, 2006 12:31 PM
To: [EMAIL PROTECTED]
Subject: wget problem

Hello GNU, when I type in C:\>wget http://.com I get the
message:
Wget is not recognized as an internal or external command, operable
program or batch file.
What am I doing wrong?
I start out from the C prompt.
Thank you.
Paula Van Berkom

wget problem

[Paula]

Hello GNU, when I type in C:\>wget http://.com I get the message:
Wget is not recognized as an internal or external command, operable program
or batch file.
What am I doing wrong?
I start out from the C prompt.
Thank you.
Paula Van Berkom

Problem when timeout

[Oliver Schulze L.]

Hi,
I'm having a problem while downloading from a Microsoft FTP server.

The problem is that the connection is timeout/close while downloading,
then wget retry to download the file, but it receives a "file not found" 
error.


Is this problem with the MS server or wget?

Here is tog of the error, Thanks
Oliver
-
--03:11:51--  
ftp://user1:[EMAIL PROTECTED]//long/path/to/file/File01%20de%205%20horas%20%2056%20-%2083.ppt
  => `ftp.example.com/long/path/to/file/File01 de 5 horas  56 - 
83.ppt'

==> CWD /long/path/to/file ... done.
==> PASV ... done.==> REST 2679100 ... done.
==> RETR File01 de 5 horas  56 - 83.ppt ... done.
Length: 51,856,384 (49M), 49,177,284 (47M) remaining

  [ skipping 2600K ]
2600K ,, ,, .. .. ..  5%   
16.95 KB/s
2650K .. .. .. .. ..  5%   
54.81 KB/s
2700K .. .. .. .. ..  5%   
64.26 KB/s
2750K .. .. .. .. ..  5%   
57.79 KB/s
2800K .. .. .. .. ..  5%   
65.73 KB/s

2850K .. .
...
11950K .. .. .. .. .. 23%   
48.25 KB/s
12000K .. .. .. .. .. 23%   
23.22 KB/s
12050K .. ..  23%
2.16 MB/s


03:16:03 (42.86 KB/s) - Data connection: Connection timed out; Control 
connection closed.

Retrying.

--03:16:34--  
ftp://user1:[EMAIL PROTECTED]//long/path/to/file/File01%20de%205%20horas%20%2056%20-%2083.ppt
 (try: 2) => `ftp.example.com/long/path/to/file/File01 de 5 horas  56 - 
83.ppt'

Connecting to ftp.example.com|123.123.123.123|:21... connected.
Logging in as user1 ... Logged in!
==> SYST ... done.==> PWD ... done.
==> TYPE I ... done.  ==> CWD not required.
==> PASV ... done.==> REST 12360360 ... done.
==> RETR File01 de 5 horas  56 - 83.ppt ...
No such file `File01 de 5 horas  56 - 83.ppt'.

The sizes do not match (local 3309820) -- retrieving.

--03:16:41--  
ftp://user1:[EMAIL PROTECTED]//long/path/to/file/File01%20de%205%20horas%20-%2084%20to%20104.ppt
  => `ftp.example.com/long/path/to/file/File01 de 5 horas - 84 
to 104.ppt'

==> CWD /long/path/to/file ... done.
==> PASV ... done.==> REST 3309820 ... done.
==> RETR File01 de 5 horas - 84 to 104.ppt ... done.
Length: 30,419,968 (29M), 27,110,148 (26M) remaining

  [ skipping 3200K ]
3200K ,, ,, ,, ,, .. 10%   
10.75 KB/s
3250K .. .. .. .. .. 11%   
39.65 KB/s

3300K .. .. ..

--
Oliver Schulze L.
<[EMAIL PROTECTED]>

Re: Problem with recursion and standard output

[Steven M. Schweda]

   As always, it might help to see which version of Wget, which Wget
command was used, what the actual output was, and which operating system
was used.  However, ...

   You're right.  And the most likely fix will be to add an error
message telling you that "-O" and "-r" (and several other options) are
incompatible.  You might review some of the other recent "-O" complaints
at "http://www.mail-archive.com/wget@sunsite.dk/";, and/or consider that,
because Wget does recursion by looking for links in the files it
downloads, you'd be asking the program to be reading and writing to the
same file at the same time, which, while not necessarily impossible,
would require a significantly different method of operation.

   What would be the value of the mess which would result from such a
Wget command if it _did_ work?



   Steven M. Schweda   [EMAIL PROTECTED]
   382 South Warwick Street(+1) 651-699-9818
   Saint Paul  MN  55105-2547

Problem with recursion and standard output

[Matthew Seth Flaschen]

Wget's recursion doesn't seem to function properly when you output the 
downloaded file to standard output ( using "-O -").  It only downloads 
the first file.


Matthew Flaschen

URLs such as "site.com/folder/bob.php?a=1" and robots.txt problem

[henka]

Hello,

Given the following robots.txt file:

User-agent: *
Disallow: /folder/bob.php?
...

One would expect that if wget tries to download a link to
/folder/bob.php?a=1 that it would exclude it because of the robots rule
line - but it doesn't (my reading of the RFC indicates that it should
exclude it).

If you remove the "?" from the robots rule, then it works as expected. 
However, this is not what you would expect - let's say you *want* the
/folder/bob.php file to be downloaded, but not all the other dynamic
content (eg, /folder/bob.php?a=1 and /folder/bob.php?a=100, etc).

I've been searching around but cannot find an obvious answer:  is this
behaviour expected, or is this a bug?

Any comments would be appreciated.

RE: wget www.openbc.com post-data/cookie problem

[Tony Lewis]

Erich Steinboeck wrote:

> Is there a way to trace the browser traffic and compare
> that to the wget traffic, to see where they differ.

You can use a web proxy. I like Achilles:
http://www.mavensecurity.com/achilles 

Tony

Problem with read timeout and retry option

[Lilian JS]

Hi !     I need wget to quit trying to download a file if it encounters any problem more than 2 times. So I specified a connection timeout and a dns timeout and retry count as 2. This works fine. If connection cannot be established for two tries, wget exits. Now the problem is, if the network connectivity is lost in between. Here the read timeout comes into picture. If there is not network activity for "read timeout", then wget starts the file download all over again. Retry does not have any effect here. Each time read timeout occurs, wget tries for a new download, and again read timeout happens. When this happens, it seems as if wget is stuck, it never exits. I want this also to happen only 'n' times. How can I configure this?     Would be glad if anyone could help me out. Thanks in advance.        Cheers  Lilian.-
		Love cheap thrills? Enjoy PC-to-Phone  calls to 30+ countries for just 2¢/min with Yahoo! Messenger with Voice.

Re: wget www.openbc.com post-data/cookie problem

[Erich Steinboeck]

as you can see the problem is with the web server, that does not return 
a cookie (by means of the Set-Cookie header) to wget.


Yes, that's exactly the problem. Logging on with a standard browser 
*does* return a cookie and successfully logs you on.


Trying the same with wget, the server does *not* return a cookie, but 
instead returns message "Your browser cookies couldn't be recognized." 
in the downloaded HTML.


Two questions:

1. Is there a way to trace the browser traffic and compare that to the 
wget traffic, to see where they differ.


2. Do you have a working wget example for any publicly accessible 
server, which uses POST and successfully returns a cookie?


Thanks!

Re: wget www.openbc.com post-data/cookie problem

[Mauro Tortonesi]

Erich Steinboeck wrote:

Mauro Tortonesi wrote:

this might be a problem with your server. could you please provide us 
with the output of wget with the -S option turned on?


[...]

>

---response begin---
HTTP/1.1 200 OK
Date: Tue, 02 May 2006 15:01:45 GMT
Server: Apache
Expires: Now
Pragma: no-cache
Cache-control: private
Connection: close
Content-Type: text/html; charset=UTF-8


hi eric,

as you can see the problem is with the web server, that does not return 
a cookie (by means of the Set-Cookie header) to wget.


--
Aequam memento rebus in arduis servare mentem...

Mauro Tortonesi  http://www.tortonesi.com

University of Ferrara - Dept. of Eng.http://www.ing.unife.it
GNU Wget - HTTP/FTP file retrieval tool  http://www.gnu.org/software/wget
Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net
Ferrara Linux User Group http://www.ferrara.linux.it

Re: wget www.openbc.com post-data/cookie problem

[Erich Steinboeck]

Mauro Tortonesi wrote:
this might be a problem with your server. could you please provide us 
with the output of wget with the -S option turned on?


Mauro, this is the wget -S output:

wget -S --no-check-certificate --keep-session-cookies --save-cookies 
cookies.txt --post-data 
"login_user_name=&login_password=" 
https://www.openbc.com

--17:00:19--  https://www.openbc.com/
   => `index.html'
Resolving www.openbc.com... 213.238.59.20
Connecting to www.openbc.com|213.238.59.20|:443... connected.
WARNING: Certificate verification error for www.openbc.com: unable to 
get local

issuer certificate
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Tue, 02 May 2006 15:00:20 GMT
  Server: Apache
  Expires: Now
  Pragma: no-cache
  Cache-control: private
  Connection: close
  Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]
--17:00:19--  https://www.openbc.com/
   => `index.html'
Resolving www.openbc.com... 213.238.59.20
Connecting to www.openbc.com|213.238.59.20|:443... connected.
WARNING: Certificate verification error for www.openbc.com: unable to 
get local

issuer certificate
HTTP request sent, awaiting response...
  HTTP/1.1 200 OK
  Date: Tue, 02 May 2006 15:00:20 GMT
  Server: Apache
  Expires: Now
  Pragma: no-cache
  Cache-control: private
  Connection: close
  Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]



And this is the wget -S -dv output:

wget -dv -S --no-check-certificate --keep-session-cookies --save-cookies 
cookies.txt --post-data 
"login_user_name=&login_password=" 
https://www.openbc.com


Setting --verbose (verbose) to 1
Setting --output-file (logfile) to wget.out
Setting --server-response (serverresponse) to 1
Setting --check-certificate (checkcertificate) to 0
Setting --keep-session-cookies (keepsessioncookies) to 1
Setting --save-cookies (savecookies) to cookies.txt
Setting --post-data (postdata) to 
login_user_name=&login_password=


DEBUG output created by Wget 1.10.2 on Windows.

--17:01:44--  https://www.openbc.com/
   => `index.html'
Resolving www.openbc.com... seconds 0.00, 213.238.59.20
Caching www.openbc.com => 213.238.59.20
Connecting to www.openbc.com|213.238.59.20|:443... seconds 0.00, connected.
Created socket 1912.
Releasing 0x00392648 (new refcount 1).
Initiating SSL handshake.
Handshake successful; connected socket 1912 to SSL handle 0x00931540
certificate:
  subject: 
/C=DE/O=www.openbc.com/OU=https://services.choicepoint.net/get.jsp?3470767950/OU=See 
www.geotrust.com/resources/cps (c)05/OU=Domain Control Validated - 
QuickSSL(R)/CN=www.openbc.com
  issuer:  /C=US/O=Equifax Secure Inc./CN=Equifax Secure Global 
eBusiness CA-1
WARNING: Certificate verification error for www.openbc.com: unable to 
get local issuer certificate


---request begin---
POST / HTTP/1.0
User-Agent: Wget/1.10.2
Accept: */*
Host: www.openbc.com
Connection: Keep-Alive
Content-Type: application/x-www-form-urlencoded
Content-Length: 54

---request end---
[POST data: 
login_user_name=&login_password=]

HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Date: Tue, 02 May 2006 15:01:45 GMT
Server: Apache
Expires: Now
Pragma: no-cache
Cache-control: private
Connection: close
Content-Type: text/html; charset=UTF-8

---response end---

  HTTP/1.1 200 OK
  Date: Tue, 02 May 2006 15:01:45 GMT
  Server: Apache
  Expires: Now
  Pragma: no-cache
  Cache-control: private
  Connection: close
  Content-Type: text/html; charset=UTF-8
Length: unspecified [text/html]

0K   196.25 KB/s

Closed 1912/SSL 0x931540
17:01:45 (196.25 KB/s) - `index.html' saved [8233]

Saving cookies to cookies.txt.
Done saving cookies.

Re: wget www.openbc.com post-data/cookie problem

[Mauro Tortonesi]

Erich Steinboeck wrote:
Being new to wget (I'm using GNU Wget 1.10.2 for Windows) I'm trying to 
log into www.openbc.com.  It works perfectly with a browser, but I can't 
get it to work with wget.


...


Can anyone help?  What am I doing wrong here?  Thanks!!


this might be a problem with your server. could you please provide us 
with the output of wget with the -S option turned on?


--
Aequam memento rebus in arduis servare mentem...

Mauro Tortonesi  http://www.tortonesi.com

University of Ferrara - Dept. of Eng.http://www.ing.unife.it
GNU Wget - HTTP/FTP file retrieval tool  http://www.gnu.org/software/wget
Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net
Ferrara Linux User Group http://www.ferrara.linux.it

wget www.openbc.com post-data/cookie problem

[Erich Steinboeck]

Being new to wget (I'm using GNU Wget 1.10.2 for Windows) I'm trying to 
log into www.openbc.com.  It works perfectly with a browser, but I can't 
get it to work with wget.


As www.openbc.com page source contains
  
  

I'm trying
wget --no-check-certificate --keep-session-cookies --save-cookies 
cookies.txt --post-data 
"login_user_name=&login_password=" 
https://www.openbc.com


and I'm receiving

--22:06:33--  https://www.openbc.com/
   => `index.html'
Resolving www.openbc.com... 213.238.59.20
Connecting to www.openbc.com|213.238.59.20|:443... connected.
WARNING: Certificate verification error for www.openbc.com: unable to 
get local issuer certificate

HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
[ <=> ] 8.233 --.--K/s
22:06:33 (193.24 KB/s) - `index.html' saved [8233]

But the cookie file is empty:
# HTTP cookie file.
# Generated by Wget on 2006-04-24 22:06:33.
# Edit at your own risk.

Browsing index.html, I'm finding two strange messages:

"Your browser cookies couldn't be recognized." and
"You need to have cookies activated to log in."

I tried adding wget options
--cookies (just in case --no-cookies was has somehow been set) and
-U "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8) Gecko/20051107 
Firefox/1.5" (just in case if openbc.com checks the user-agent)


But no change.

Can anyone help?  What am I doing wrong here?  Thanks!!

wget www.openbc.com post-data/cookie problem

[Erich Steinboeck]

Being new to wget (I'm using GNU Wget 1.10.2 for Windows) I'm trying to 
log into www.openbc.com.  It works perfectly with a browser, but I can't 
get it to work with wget.


As www.openbc.com page source contains
  
  

I'm trying
wget --no-check-certificate --keep-session-cookies --save-cookies 
cookies.txt --post-data 
"login_user_name=&login_password=" 
https://www.openbc.com


and I'm receiving

--22:06:33--  https://www.openbc.com/
   => `index.html'
Resolving www.openbc.com... 213.238.59.20
Connecting to www.openbc.com|213.238.59.20|:443... connected.
WARNING: Certificate verification error for www.openbc.com: unable to 
get local issuer certificate

HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
[ <=> ] 8.233 --.--K/s
22:06:33 (193.24 KB/s) - `index.html' saved [8233]

But the cookie file is empty:
# HTTP cookie file.
# Generated by Wget on 2006-04-24 22:06:33.
# Edit at your own risk.

Browsing index.html, I'm finding two strange messages:

"Your browser cookies couldn't be recognized." and
"You need to have cookies activated to log in."

I tried adding wget options
--cookies (just in case --no-cookies was has somehow been set) and
-U "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8) Gecko/20051107 
Firefox/1.5" (just in case if openbc.com checks the user-agent)


But no change.

Can anyone help?  What am I doing wrong here?  Thanks!!

wget www.openbc.com post-data/cookie problem

[Erich Steinboeck]

Being new to wget (I'm using GNU Wget 1.10.2 for Windows) I'm trying to 
log into www.openbc.com.  It works perfectly with a browser, but I can't 
get it to work with wget.


As www.openbc.com page source contains
  
  

I'm trying
wget --no-check-certificate --keep-session-cookies --save-cookies 
cookies.txt --post-data 
"login_user_name=&login_password=" 
https://www.openbc.com


and I'm receiving

--22:06:33--  https://www.openbc.com/
   => `index.html'
Resolving www.openbc.com... 213.238.59.20
Connecting to www.openbc.com|213.238.59.20|:443... connected.
WARNING: Certificate verification error for www.openbc.com: unable to 
get local issuer certificate

HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
[ <=> ] 8.233 --.--K/s
22:06:33 (193.24 KB/s) - `index.html' saved [8233]

But the cookie file is empty:
# HTTP cookie file.
# Generated by Wget on 2006-04-24 22:06:33.
# Edit at your own risk.

Browsing index.html, I'm finding two strange messages:

"Your browser cookies couldn't be recognized." and
"You need to have cookies activated to log in."

I tried adding wget options
--cookies (just in case --no-cookies was has somehow been set) and
-U "Mozilla/5.0 (Windows; U; Windows NT 5.1; nl; rv:1.8) Gecko/20051107 
Firefox/1.5" (just in case if openbc.com checks the user-agent)


But no change.

Can anyone help?  What am I doing wrong here?  Thanks!!

RE: Problem with double slashes in URI

[Zembower, Kevin]

That worked perfectly. Thank you for your help. I had tried replacing
one of the slashes with %2F, but didn't think of using two, in
combination with the existing slash after the 'ccp3'.

Thanks, again.

-Kevin

-Original Message-
From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] 
Sent: Friday, March 31, 2006 5:17 PM
To: Zembower, Kevin
Cc: wget@sunsite.dk
Subject: Re: Problem with double slashes in URI

"Zembower, Kevin" <[EMAIL PROTECTED]> writes:

> [EMAIL PROTECTED]:/tmp$ wget --timestamping --no-host-directories --glob=on
> --recursive --cut-dirs=4
> 'ftp://xxx:[EMAIL PROTECTED]/%2Fccp1/data/shared/news/motd/qotd.txt'

If you need double slash, you must spell it explicitly:

wget [...]
ftp://xxx:[EMAIL PROTECTED]/%2F%2Fccp1/data/shared/news/motd/qotd.txt

Substituting ccp3 for something that I can connect to:

$ wget -S
ftp://gnjilux.srk.fer.hr/%2F%2Fccp1/data/shared/news/motd/qotd.txt
...
--> CWD //ccp1/data/shared/news/motd

That's with Wget 1.10.2  It might not work on versions before 1.10.