Re: with recursive wget status code does not reflect success/failure of operation
Alle 09:06, sabato 17 settembre 2005, Steven M. Schweda ha scritto: I suppose that it's a waste of time and space to point this out here, but native VMS status codes include a severity field (the low three bits), with popular values being (from STSDEF.H): #define STS$K_WARNING 0 /* WARNING */ #define STS$K_SUCCESS 1 /* SUCCESSFUL COMPLETION */ #define STS$K_ERROR 2 /* ERROR */ #define STS$K_INFO 3/* INFORMATION */ #define STS$K_SEVERE 4 /* SEVERE ERROR */ Note that success (including informational) is odd, while anything worse is even. While an OS like UNIX is handicapped by not having such a useful convention, it would make sense for Wget to have status values which distinguish among degrees of success and failure, like the ones suggested by Hrvoje Niksic. yes, i think it's a good idea. Ideally, the values used could be defined in some central location, allowing convenient replacement with suitable VMS-specific values when the time comes. (Naturally, _all_ exit() calls and/or return statements should use one of the pre-defined values.) mmh, i don't understand why we should use VMS-specific values in wget. And, as long as I'm wasting time and space, I'll note that I'd still like to see my VMS-related (and other) changes integrated into the main Wget code stream. the wget code is going through a major refactoring effort. later on, just before releasing wget 2.0, i promise i will re-evaluate your patches and merge them if they're not too intrusive. -- Aequam memento rebus in arduis servare mentem... Mauro Tortonesi http://www.tortonesi.com University of Ferrara - Dept. of Eng.http://www.ing.unife.it GNU Wget - HTTP/FTP file retrieval tool http://www.gnu.org/software/wget Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net Ferrara Linux User Group http://www.ferrara.linux.it
Re: with recursive wget status code does not reflect success/failure of operation
From: Mauro Tortonesi [EMAIL PROTECTED] Ideally, the values used could be defined in some central location, allowing convenient replacement with suitable VMS-specific values when the time comes. (Naturally, _all_ exit() calls and/or return statements should use one of the pre-defined values.) mmh, i don't understand why we should use VMS-specific values in wget. On VMS (not elsewhere), Wget should use VMS-specific values. The VMS C RTL is willing to convert 0 into a generic success code, but 1 (EPERM, Not owner) and 2 (ENOENT, No such file or directory) would tend to confuse the users (and the rest of the OS). Having the exit codes defined in a central location would make it easy to adapt them as needed. Having to search the code for every instance of return 1 or exit(2) would make it too complicated. Steven M. Schweda (+1) 651-699-9818 382 South Warwick Street[EMAIL PROTECTED] Saint Paul MN 55105-2547
Re: with recursive wget status code does not reflect success/failure of operation
Alle 18:06, lunedì 19 settembre 2005, Hrvoje Niksic ha scritto: Mauro Tortonesi [EMAIL PROTECTED] writes: mmh, i don't understand why we should use VMS-specific values in wget. The closest Unix has to offer are these BSD-specific values which few programs use: /* * SYSEXITS.H -- Exit status codes for system programs. * *This include file attempts to categorize possible error *exit statuses for system programs, notably delivermail *and the Berkeley network. * *Error numbers begin at EX__BASE to reduce the possibility of *clashing with other exit statuses that random programs may *already return. The meaning of the codes is approximately *as follows: * *EX_USAGE -- The command was used incorrectly, e.g., with *the wrong number of arguments, a bad flag, a bad *syntax in a parameter, or whatever. *EX_DATAERR -- The input data was incorrect in some way. *This should only be used for user's data not *system files. *EX_NOINPUT -- An input file (not a system file) did not *exist or was not readable. This could also include *errors like No message to a mailer (if it cared *to catch it). *EX_NOUSER -- The user specified did not exist. This might *be used for mail addresses or remote logins. *EX_NOHOST -- The host specified did not exist. This is used *in mail addresses or network requests. *EX_UNAVAILABLE -- A service is unavailable. This can occur *if a support program or file does not exist. This *can also be used as a catchall message when something *you wanted to do doesn't work, but you don't know *why. *EX_SOFTWARE -- An internal software error has been detected. *This should be limited to non-operating system related *errors as possible. *EX_OSERR -- An operating system error has been detected. *This is intended to be used for such things as cannot *fork, cannot create pipe, or the like. It includes *things like getuid returning a user that does not *exist in the passwd file. *EX_OSFILE -- Some system file (e.g., /etc/passwd, /etc/utmp, *etc.) does not exist, cannot be opened, or has some *sort of error (e.g., syntax error). *EX_CANTCREAT -- A (user specified) output file cannot be *created. *EX_IOERR -- An error occurred while doing I/O on some file. *EX_TEMPFAIL -- temporary failure, indicating something that *is not really an error. In sendmail, this means *that a mailer (e.g.) could not create a connection, *and the request should be reattempted later. *EX_PROTOCOL -- the remote system returned something that *was not possible during a protocol exchange. *EX_NOPERM -- You did not have sufficient permission to *perform the operation. This is not intended for *file system problems, which should use NOINPUT or *CANTCREAT, but rather for higher level permissions. */ yes, but i was thinking to define wget specific error codes. are there any major objections to this policy? -- Aequam memento rebus in arduis servare mentem... Mauro Tortonesi http://www.tortonesi.com University of Ferrara - Dept. of Eng.http://www.ing.unife.it GNU Wget - HTTP/FTP file retrieval tool http://www.gnu.org/software/wget Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net Ferrara Linux User Group http://www.ferrara.linux.it
Re: with recursive wget status code does not reflect success/failure of operation
Mauro Tortonesi [EMAIL PROTECTED] writes: yes, but i was thinking to define wget specific error codes. I wouldn't object to those. The scripting people might find them useful.
with recursive wget status code does not reflect success/failure of operation
I'm not sure if this is a bug or a feature, but with recursive operation, if a get fails and retrieve_tree bails out then no sensible error codes are returned to main.c (errors are only passed up if the user's quota was full, the URL was invalid or there was a write error) so retrieve_tree always returns RETROK causing the main function to return with an exit status of 0 even if there was an error. With single wgets you get a non-zero error code if the operation failed. This is kind of annoying if you are trying to determine of a recursive operation completed successfully in a shell script. Is there a good reason why retrieve tree doesn't just return the status of the last failed operation on failure? Even if this weren't the default behaviour it would be useful as at the moment there is no way to find out if a recursive get failed or succeeded. owen cliffe ---
Re: with recursive wget status code does not reflect success/failure of operation
Owen Cliffe [EMAIL PROTECTED] writes: Is there a good reason why retrieve tree doesn't just return the status of the last failed operation on failure? The original reason (which I don't claim to be good) is because Wget doesn't stop upon on error, it continues. Because of this returning a non-zero error code felt wrong, because the download has after all finished succesffully. The quota exceeded case is an exception consistent with this logic because, when quota is exceeded, Wget really terminates the entire download. But you present a convincing argument otherwise. Maybe Wget should use different error codes for different cases, like: 0 -- all files downloaded successfully (not counting errors in robots.txt and such) 1 -- some errors encountered 2 -- fatal errors encountered (such as the quota exceeded case), download aborted What do others think about this?
Re: with recursive wget status code does not reflect success/failure of operation
Alle 18:58, mercoledì 14 settembre 2005, Hrvoje Niksic ha scritto: Owen Cliffe [EMAIL PROTECTED] writes: Is there a good reason why retrieve tree doesn't just return the status of the last failed operation on failure? The original reason (which I don't claim to be good) is because Wget doesn't stop upon on error, it continues. Because of this returning a non-zero error code felt wrong, because the download has after all finished succesffully. The quota exceeded case is an exception consistent with this logic because, when quota is exceeded, Wget really terminates the entire download. But you present a convincing argument otherwise. Maybe Wget should use different error codes for different cases, like: 0 -- all files downloaded successfully (not counting errors in robots.txt and such) 1 -- some errors encountered 2 -- fatal errors encountered (such as the quota exceeded case), download aborted What do others think about this? i think that's a good point. -- Aequam memento rebus in arduis servare mentem... Mauro Tortonesi http://www.tortonesi.com University of Ferrara - Dept. of Eng.http://www.ing.unife.it GNU Wget - HTTP/FTP file retrieval tool http://www.gnu.org/software/wget Deep Space 6 - IPv6 for Linuxhttp://www.deepspace6.net Ferrara Linux User Group http://www.ferrara.linux.it
Success!
I downloaded the official wget 1.9 sources and, following some suggestions that were given to me, temporarily renamed a bunch of SSL libraries that were already on my system that seemed to be confusing the linker... And it all works! Compiled wget 1.9 successfully under Mac OS X 10.2.8 on a dual G5 PowerMac. I needed to rename/move /usr/bin/openssl, /usr/include/openssl, and /usr/lib/libssl* and /usr/lib/libcrypto* because I'd compiled the latest OpenSSL libraries and installed them under /usr/local Best Regards, Rob Poole [EMAIL PROTECTED]
Re: Read error (Success) in headers using wget and ssl
Hi, That's odd, the server certificate is valid, but indeed I can't verify it too with openssl s_client. Then again, I also get errors when using openssl s_client checking against https://woz-online.amsterdam.nl while wget has absolutely no problem fetching that URL. (and with the correct client certificate it can retrieve the page too). Therefor I wouldn't expect a Read error (Success) in retreiving https://145.222.136.165 maybe a warning about verification, but it should continue, like it does for the woz-online.amsterdam.nl URL. BTW, the dns has been updated, so https://www.bouwfondsbbs.nl can now be used instead of the 145.222.135.165 Just for your info, I'm using wget to poll regularly check if the site's still up. Clues, anybody :) ? Dimitri On Fri, 19 Sep 2003, Doug Kaufman wrote: On Fri, 19 Sep 2003, Hrvoje Niksic wrote: wget https://145.222.135.165/index.htm --13:46:36-- https://145.222.135.165/index.htm = `index.htm' Connecting to 145.222.135.165:443... connected. HTTP request sent, awaiting response... Read error (Success) in headers. Retrying. --13:46:37-- https://145.222.135.165/index.htm (try: 2) = `index.htm' Connecting to 145.222.135.165:443... connected. HTTP request sent, awaiting response... Read error (Success) in headers. Retrying. --- Expected: Unable to establish SSL connection. because it's using client certificates, but when using the client certificate the same error occurs, so this doesn't seem a clientcertificate problem, thought it might be that wget is having trouble checking that it does need a client certificate ?! The problem seems to be a bad server certificate, or at least one not in the usual database of trusted certificates. When I connect with openssl s_client I get error:num=20 (unable to get local certificate) and also error:num=27 (certificate not trusted). The text from the server certificate follows: Certificate: Data: Version: 3 (0x2) Serial Number: 47:f7:ee:c0:35:19:65:6c:f2:16:ac:67:ae:e6:48:2e Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, O=RSA Data Security, Inc., OU=Secure Server Certification Authority Validity Not Before: Aug 26 00:00:00 2003 GMT Not After : Aug 25 23:59:59 2004 GMT Subject: C=NL, ST=Utrecht, L=Amersfoort, O=bouwfonds hypotheken, OU=Informatie Management, OU=Terms of use at pki.pinkroccade.com/rpa (c) 02, OU=Authenticated by PinkRoccade, OU=Member, VeriSign Trust Network, CN=www.bouwfondsbbs.nl Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (1024 bit) Modulus (1024 bit): 00:bd:53:94:ec:57:e7:68:05:cf:53:5e:88:24:63: ca:07:3c:0d:63:df:73:20:5c:20:37:76:e4:9c:89: eb:76:bb:55:de:41:3f:12:5f:cb:b8:fb:23:ac:7b: 48:00:50:55:51:18:cc:df:bd:62:67:85:9c:4f:99: b6:db:e0:56:e0:ab:38:33:ae:15:0d:b4:a5:c3:77: f1:1a:91:f1:15:55:14:e5:f3:7b:65:56:38:cf:ef: 4e:3a:3c:23:8a:ce:83:6b:e4:06:55:fe:ca:09:39: 25:a0:54:28:84:16:1f:12:14:ad:12:ee:05:23:e2: b7:bd:e5:73:2b:cd:85:22:11 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Key Encipherment X509v3 CRL Distribution Points: URI:http://crl.verisign.com/RSASecureServer.crl X509v3 Certificate Policies: Policy: 2.16.840.1.113733.1.7.1.1 CPS: https://www.verisign.com/CPS User Notice: Organization: VeriSign, Inc. Number: 1 Explicit Text: VeriSign's CPS incorp. by reference liab. ltd. (c)97 VeriSign X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication Authority Information Access: OCSP - URI:http://ocsp.verisign.com Signature Algorithm: sha1WithRSAEncryption 1d:46:35:f6:53:80:e8:39:1f:ff:ca:f5:7d:fd:64:06:7b:76: 78:44:1e:d3:0a:59:c5:af:2d:fe:41:19:c8:d2:db:a0:9a:8a: c6:65:87:49:ad:c0:cd:d1:b5:e6:66:c7:ac:f6:88:f5:dd:84: 58:fb:9c:d3:93:e5:81:74:99:29:90:a6:3d:40:23:7a:11:97: 60:2f:65:44:b8:33:9d:54:56:58:8f:2b:fb:c3:1c:28:7f:15: ef:aa:fa:33:ba:12:1f:d8:82:89:8d:f0:a0:f7:a5:e1:b7:05: 40:91:b3:71:a8:b1:cf:e3:2a:7b:05:89:f2:99:19:e7:cb Doug -- Doug Kaufman Internet: [EMAIL PROTECTED]
Read error (Success) in headers using wget and ssl
Hi, I'm having trouble connecting with wget to a site using SSL: wget https://145.222.135.165/index.htm --13:46:36-- https://145.222.135.165/index.htm = `index.htm' Connecting to 145.222.135.165:443... connected. HTTP request sent, awaiting response... Read error (Success) in headers. Retrying. --13:46:37-- https://145.222.135.165/index.htm (try: 2) = `index.htm' Connecting to 145.222.135.165:443... connected. HTTP request sent, awaiting response... Read error (Success) in headers. Retrying. --- Expected: Unable to establish SSL connection. because it's using client certificates, but when using the client certificate the same error occurs, so this doesn't seem a clientcertificate problem, thought it might be that wget is having trouble checking that it does need a client certificate ?! Ofcourse using IE as a browser (and the client certificate), no problem... Any idea how to fix this ? I used wget 1.8.2 and a nightly cvs of 20030909, same problem (Please reply directly too as I'm not on the list) Best regards, Dimitri
Re: Read error (Success) in headers using wget and ssl
Dimitri Ars [EMAIL PROTECTED] writes: I'm having trouble connecting with wget to a site using SSL: [...] I can repeat this, but currently I don't understand enough about SSL to fix it. Christian, could you please help? wget https://145.222.135.165/index.htm --13:46:36-- https://145.222.135.165/index.htm = `index.htm' Connecting to 145.222.135.165:443... connected. HTTP request sent, awaiting response... Read error (Success) in headers. Retrying. --13:46:37-- https://145.222.135.165/index.htm (try: 2) = `index.htm' Connecting to 145.222.135.165:443... connected. HTTP request sent, awaiting response... Read error (Success) in headers. Retrying. --- Expected: Unable to establish SSL connection. because it's using client certificates, but when using the client certificate the same error occurs, so this doesn't seem a clientcertificate problem, thought it might be that wget is having trouble checking that it does need a client certificate ?! Ofcourse using IE as a browser (and the client certificate), no problem... Any idea how to fix this ? I used wget 1.8.2 and a nightly cvs of 20030909, same problem (Please reply directly too as I'm not on the list) Best regards, Dimitri
Re: Read error (Success) in headers using wget and ssl
On Fri, 19 Sep 2003, Hrvoje Niksic wrote: wget https://145.222.135.165/index.htm --13:46:36-- https://145.222.135.165/index.htm = `index.htm' Connecting to 145.222.135.165:443... connected. HTTP request sent, awaiting response... Read error (Success) in headers. Retrying. --13:46:37-- https://145.222.135.165/index.htm (try: 2) = `index.htm' Connecting to 145.222.135.165:443... connected. HTTP request sent, awaiting response... Read error (Success) in headers. Retrying. --- Expected: Unable to establish SSL connection. because it's using client certificates, but when using the client certificate the same error occurs, so this doesn't seem a clientcertificate problem, thought it might be that wget is having trouble checking that it does need a client certificate ?! The problem seems to be a bad server certificate, or at least one not in the usual database of trusted certificates. When I connect with openssl s_client I get error:num=20 (unable to get local certificate) and also error:num=27 (certificate not trusted). The text from the server certificate follows: Certificate: Data: Version: 3 (0x2) Serial Number: 47:f7:ee:c0:35:19:65:6c:f2:16:ac:67:ae:e6:48:2e Signature Algorithm: sha1WithRSAEncryption Issuer: C=US, O=RSA Data Security, Inc., OU=Secure Server Certification Authority Validity Not Before: Aug 26 00:00:00 2003 GMT Not After : Aug 25 23:59:59 2004 GMT Subject: C=NL, ST=Utrecht, L=Amersfoort, O=bouwfonds hypotheken, OU=Informatie Management, OU=Terms of use at pki.pinkroccade.com/rpa (c) 02, OU=Authenticated by PinkRoccade, OU=Member, VeriSign Trust Network, CN=www.bouwfondsbbs.nl Subject Public Key Info: Public Key Algorithm: rsaEncryption RSA Public Key: (1024 bit) Modulus (1024 bit): 00:bd:53:94:ec:57:e7:68:05:cf:53:5e:88:24:63: ca:07:3c:0d:63:df:73:20:5c:20:37:76:e4:9c:89: eb:76:bb:55:de:41:3f:12:5f:cb:b8:fb:23:ac:7b: 48:00:50:55:51:18:cc:df:bd:62:67:85:9c:4f:99: b6:db:e0:56:e0:ab:38:33:ae:15:0d:b4:a5:c3:77: f1:1a:91:f1:15:55:14:e5:f3:7b:65:56:38:cf:ef: 4e:3a:3c:23:8a:ce:83:6b:e4:06:55:fe:ca:09:39: 25:a0:54:28:84:16:1f:12:14:ad:12:ee:05:23:e2: b7:bd:e5:73:2b:cd:85:22:11 Exponent: 65537 (0x10001) X509v3 extensions: X509v3 Basic Constraints: CA:FALSE X509v3 Key Usage: Digital Signature, Key Encipherment X509v3 CRL Distribution Points: URI:http://crl.verisign.com/RSASecureServer.crl X509v3 Certificate Policies: Policy: 2.16.840.1.113733.1.7.1.1 CPS: https://www.verisign.com/CPS User Notice: Organization: VeriSign, Inc. Number: 1 Explicit Text: VeriSign's CPS incorp. by reference liab. ltd. (c)97 VeriSign X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication Authority Information Access: OCSP - URI:http://ocsp.verisign.com Signature Algorithm: sha1WithRSAEncryption 1d:46:35:f6:53:80:e8:39:1f:ff:ca:f5:7d:fd:64:06:7b:76: 78:44:1e:d3:0a:59:c5:af:2d:fe:41:19:c8:d2:db:a0:9a:8a: c6:65:87:49:ad:c0:cd:d1:b5:e6:66:c7:ac:f6:88:f5:dd:84: 58:fb:9c:d3:93:e5:81:74:99:29:90:a6:3d:40:23:7a:11:97: 60:2f:65:44:b8:33:9d:54:56:58:8f:2b:fb:c3:1c:28:7f:15: ef:aa:fa:33:ba:12:1f:d8:82:89:8d:f0:a0:f7:a5:e1:b7:05: 40:91:b3:71:a8:b1:cf:e3:2a:7b:05:89:f2:99:19:e7:cb Doug -- Doug Kaufman Internet: [EMAIL PROTECTED]
Trying to Mirror a site without success....
Hello all, I am new to this as I have just downloaded wget and an aqua shell for OSX in order to try to download the posts that reside in my online community at www.communityzero.com/imperium. The problem I seem to be having is that any request to download recursively, or even just one level, runs into problems with the site's way of handling page requests via a registered account. When I access the pages through my browser (with cookies enabled) I access the pages fine, but as wget does not submit my cookie details I get bumped to the community main page or login page with either 'session expired' errors or requests to log-in register. I have tried to make the site 'public' (so that anyone can view it) but not only does CZ have community specific passwords, it also requires you to register as a CZ member first. Is there any way around this? I have hundreds of posts to copy... Many thanks for any help you can offer. Marco Berni ps. Please cc me as I am not registered to the list. in a world without walls and fences who needs Windows and Gates
1.7.1-pre1 on NCR MP-RAS: success
Executive summary: complete success. On NCR MP-RAS, Wget 1.7.1-pre1 configured and compiled fine, and passed a few simple tests. The -lnsl/-lsocket and MAP_FAILED problems seen with previous versions did not occur. No SSL library is installed on the system. ./configure --with-ssl detected that correctly. The resulting executable worked fine with HTTP. For https: URLs, it prints Unknown/unsupported protocol and exits. A binary made with plain ./configure without --with-ssl exhibits the same exact behaviour. Should you need the logs, they're at http://www.teaser.fr/~amajorel/mpras/jp/wget-1.7.1-pre1.config.log.gz http://www.teaser.fr/~amajorel/mpras/jp/wget-1.7.1-pre1.config.log.with-ssl.gz Thanks to everyone involved. -- André Majorel Work: [EMAIL PROTECTED] Home: [EMAIL PROTECTED] http://www.teaser.fr/~amajorel/
Re: 1.7.1-pre1 on NCR MP-RAS: success
Andre Majorel [EMAIL PROTECTED] writes: Executive summary: complete success. [...] Thanks a lot for testing.
