Re: [Bug-wget] wget fails to encode spaces in URLs
Hi Volker, I see it now, thanks. This small patch makes sure the url is parsed in any case. Cheers, Giuseppe === modified file 'src/retr.c' --- src/retr.c 2011-06-05 12:31:24 + +++ src/retr.c 2011-06-08 09:29:20 + @@ -1005,9 +1005,7 @@ break; } - /* Need to reparse the url, since it didn't have iri information. */ - if (opt.enable_iri) - parsed_url = url_parse (cur_url-url-url, NULL, tmpiri, true); + parsed_url = url_parse (cur_url-url-url, NULL, tmpiri, true); if ((opt.recursive || opt.page_requisites) (cur_url-url-scheme != SCHEME_FTP || getproxy (cur_url-url))) Volker Kuhlmann list0...@paradise.net.nz writes: Hi Giuseppe, Thanks! I compiled it with libproxy: same problem. I then compiled it with just ./configure --prefix=/tmp/.../ make ./src/wget -i- http://downloads.sourceforge.net/project/bandwidthd/bandwidthd/bandwidthd 2.0.1/bandwidthd-2.0.1.tgz?r=ts=1307308092use_mirror=transact ^D (note the space after bandwidthd) and wireshark gives me: GET /project/bandwidthd/bandwidthd/bandwidthd 2.0.1/bandwidthd-2.0.1.tgz?r=ts=1307308092use_mirror=transact HTTP/1.1 User-Agent: Wget/1.12-2504 (linux-gnu) Accept: */* Host: downloads.sourceforge.net Connection: Keep-Alive Sorry NOT FIXED. My system and user wgetrc contain prefer-family = none use_proxy = off dirstruct = on timestamping = on dot_bytes = 64k dot_spacing = 10 dots_in_line = 50 backup_converted = on Volker
Re: [Bug-wget] Issue with TOMCAT SSL server wget
please keep the mailing list CC'ed in your replies. It seems the server doesn't accept the client certificate. Are you sure the cert.pem certificate is included in keystore.jks? Giuseppe brad bruggemann bradley.bruggem...@gmail.com writes: Giuseppe, There's a correction to my original post. The output that I get when I run the original command (with secure-protocol) is: OpenSSL: error:14094412:SSL routines:SSL3_READ_BYTES:sslv3 alert bad certificate When I run it without secure-protocol i get: OpenSSL: error:140943F2:SSL routines:SSL3_READ_BYTES:sslv3 alert unexpected message On Wed, Jun 8, 2011 at 7:08 AM, Giuseppe Scrivano gscriv...@gnu.org wrote: brad bruggemann bradley.bruggem...@gmail.com writes: Use wget to grab file: wget --secure-protocol=TLSv1 --certificate-type=PEM --certificate=/ path.to/cert.pem --password= https://IP_ADDRESS:1234/file.txt -o /tmp/file.txt what does happen if you don't specify --secure-protocol? Cheers, Giuseppe
Re: [Bug-wget] CNET download links not working with WGET
No, it's not working. It downloads part of the URL and creates a file named 3001-8022_4-10804572.html@spi=077d9109e846975d0db9532bd610588f.1 which is 68 KB. I cannot wget to treat the string of characters as a whole URL. Please help, I really need to get this script working and the only place to download this file is from CNET. So... looks like it works, then. Your command shell isn't complaining about weird command names, wget is clearly requesting the full and correct URL, it follows redirections, and saves using the final redirection URL (the latest sources wouldn't follow that last step - it'd save using the request URI by default). If you dislike the filename, then provided you have a recent enough version of wget you can add the --content-disposition option if the server provides a rename header (Content-Disposition); or else use -E to have wget force the file name to end in .html -mjc (05/26/2011 12:19 PM), Jeff Givens wrote: Hi, I know this is an older topic but thanks for replying. I forgot to mention I had already what you listed below and this is the output I get: C:\DOWNLOADwget http://dw.com.com/redir?edId=3siteId=4oId=3000-8022_ 4-10804572ontId=8022_4spi=077d9109e846975d0db9532bd610588flop=linktag=tdw_dl textltype=dl_dlnowpid=11665648mfgId=6290020merId=6290020pguid=HFsQLwoOYJQAA BuImQcAAAGmdestUrl=http%3A%2F%2Fdownload.cnet.com%2F3001-8022_4-10804572.html%3 Fspi%3D077d9109e846975d0db9532bd610588f --2011-05-02 12:34:20-- http://dw.com.com/redir?edId=3siteId=4oId=3000-8022_4 -10804572ontId=8022_4spi=077d9109e846975d0db9532bd610588flop=linktag=tdw_dlt extltype=dl_dlnowpid=11665648mfgId=6290020merId=6290020pguid=HFsQLwoOYJQAAB uImQcAAAGmdestUrl=http%3A%2F%2Fdownload.cnet.com%2F3001-8022_4-10804572.html%3F spi%3D077d9109e846975d0db9532bd610588f Resolving dw.com.com... 216.239.113.95 Connecting to dw.com.com|216.239.113.95|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://download.cnet.com/3001-8022_4-10804572.html?spi=077d9109e846975 d0db9532bd610588f [following] --2011-05-02 12:34:21-- http://download.cnet.com/3001-8022_4-10804572.html?spi= 077d9109e846975d0db9532bd610588f Resolving download.cnet.com... 64.30.224.58 Connecting to download.cnet.com|64.30.224.58|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: `3001-8022_4-10804572.html@spi=077d9109e846975d0db9532bd610588f.1' [= ] 69,240 77.3K/s in 0.9s 2011-05-02 12:34:22 (77.3 KB/s) - `3001-8022_4-10804572.html@spi=077d9109e846975 d0db9532bd610588f.1' saved [69240] C:\DOWNLOAD Thanks for your help. -Jeff hello, the character in the url is interpreted by your shell. Try using something like: wget URL Cheers, Giuseppe Jeff Givensj...@sds.net writes: Hello, I am having an issue downloading files via download links from CNET. It appears to locate some of the URL but stops at the first siteId part. I have included the debug information as well. Thanks in advance for your help. C:\DOWNLOAD\wget http://dw.com.com/redir?edId=3siteId=4oId=300 0-8022_4-10804572ontId=8022_4spi=077d9109e846975d0db9532bd610588flop=linktag =tdw_dltextltype=dl_dlnowpid=11665648mfgId=6290020merId=6290020pguid=HFsQLw oOYJQAABuImQcAAAGmdestUrl=http%3A%2F%2Fdownload.cnet.com%2F3001-8022_4-10804572 .html%3Fspi%3D077d9109e846975d0db9532bd610588f --2011-04-19 11:30:35-- http://dw.com.com/redir?edId=3 Resolving dw.com.com... 216.239.113.95 Connecting to dw.com.com|216.239.113.95|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://dw.com.com/redir/redx/?edId=3 [following] --2011-04-19 11:30:36-- http://dw.com.com/redir/redx/?edId=3 Reusing existing connection to dw.com.com:80. HTTP request sent, awaiting response... 404 Not Found 2011-04-19 11:30:36 ERROR 404: Not Found. 'siteId' is not recognized as an internal or external command, operable program or batch file. 'oId' is not recognized as an internal or external command, operable program or batch file. 'ontId' is not recognized as an internal or external command, operable program or batch file. 'spi' is not recognized as an internal or external command, operable program or batch file. 'lop' is not recognized as an internal or external command, operable program or batch file. 'tag' is not recognized as an internal or external command, operable program or batch file. 'ltype' is not recognized as an internal or external command, operable program or batch file. 'pid' is not recognized as an internal or external command, operable program or batch file. 'mfgId' is not recognized as an internal or external command, operable program or batch file. 'merId' is not recognized as an internal or external command, operable program or batch file. 'pguid' is not recognized as an internal or external command, operable program or batch file. 'destUrl' is not recognized as an internal or external command, operable program or
Re: [Bug-wget] CNET download links not working with WGET
If you read the most recent output of wget that you gave (after quoting the URL), it _does_ treat the string of characters as a whole URL. The server redirects it to a shorter URL. If I enter that same URL into a browser, it does the same redirection there, and results in an HTML page, just like what wget gets. That page seems to have some JavaScript or something that initiates a separate download of something else; I suppose that something else is what you wanted. As you may know, wget doesn't execute JavaScript code from a webpage, so you'll need to find the real URL to the thing you wanted to download, and feed that to wget. -mjc On 06/08/2011 09:38 AM, Jeff Givens wrote No, it's not working. It downloads part of the URL and creates a file named 3001-8022_4-10804572.html@spi=077d9109e846975d0db9532bd610588f.1 which is 68 KB. I cannot wget to treat the string of characters as a whole URL. Please help, I really need to get this script working and the only place to download this file is from CNET. So... looks like it works, then. Your command shell isn't complaining about weird command names, wget is clearly requesting the full and correct URL, it follows redirections, and saves using the final redirection URL (the latest sources wouldn't follow that last step - it'd save using the request URI by default). If you dislike the filename, then provided you have a recent enough version of wget you can add the --content-disposition option if the server provides a rename header (Content-Disposition); or else use -E to have wget force the file name to end in .html -mjc (05/26/2011 12:19 PM), Jeff Givens wrote: Hi, I know this is an older topic but thanks for replying. I forgot to mention I had already what you listed below and this is the output I get: C:\DOWNLOADwget http://dw.com.com/redir?edId=3siteId=4oId=3000-8022_ 4-10804572ontId=8022_4spi=077d9109e846975d0db9532bd610588flop=linktag=tdw_dl textltype=dl_dlnowpid=11665648mfgId=6290020merId=6290020pguid=HFsQLwoOYJQAA BuImQcAAAGmdestUrl=http%3A%2F%2Fdownload.cnet.com%2F3001-8022_4-10804572.html%3 Fspi%3D077d9109e846975d0db9532bd610588f --2011-05-02 12:34:20-- http://dw.com.com/redir?edId=3siteId=4oId=3000-8022_4 -10804572ontId=8022_4spi=077d9109e846975d0db9532bd610588flop=linktag=tdw_dlt extltype=dl_dlnowpid=11665648mfgId=6290020merId=6290020pguid=HFsQLwoOYJQAAB uImQcAAAGmdestUrl=http%3A%2F%2Fdownload.cnet.com%2F3001-8022_4-10804572.html%3F spi%3D077d9109e846975d0db9532bd610588f Resolving dw.com.com... 216.239.113.95 Connecting to dw.com.com|216.239.113.95|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://download.cnet.com/3001-8022_4-10804572.html?spi=077d9109e846975 d0db9532bd610588f [following] --2011-05-02 12:34:21-- http://download.cnet.com/3001-8022_4-10804572.html?spi= 077d9109e846975d0db9532bd610588f Resolving download.cnet.com... 64.30.224.58 Connecting to download.cnet.com|64.30.224.58|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: `3001-8022_4-10804572.html@spi=077d9109e846975d0db9532bd610588f.1' [= ] 69,240 77.3K/s in 0.9s 2011-05-02 12:34:22 (77.3 KB/s) - `3001-8022_4-10804572.html@spi=077d9109e846975 d0db9532bd610588f.1' saved [69240] C:\DOWNLOAD Thanks for your help. - Jeff
Re: [Bug-wget] CNET download links not working with WGET
Micah, thanks for your help. That was the piece that I was missing. I didn't realize it was re-directing to another site. I was able to find out the other site it was going to, download the executable and then I just put in a command to re-name the exe file since it was named the URL. Thanks again for your help. If you read the most recent output of wget that you gave (after quoting the URL), it _does_ treat the string of characters as a whole URL. The server redirects it to a shorter URL. If I enter that same URL into a browser, it does the same redirection there, and results in an HTML page, just like what wget gets. That page seems to have some JavaScript or something that initiates a separate download of something else; I suppose that something else is what you wanted. As you may know, wget doesn't execute JavaScript code from a webpage, so you'll need to find the real URL to the thing you wanted to download, and feed that to wget. -mjc On 06/08/2011 09:38 AM, Jeff Givens wrote No, it's not working. It downloads part of the URL and creates a file named 3001-8022_4-10804572.html@spi=077d9109e846975d0db9532bd610588f.1 which is 68 KB. I cannot wget to treat the string of characters as a whole URL. Please help, I really need to get this script working and the only place to download this file is from CNET. So... looks like it works, then. Your command shell isn't complaining about weird command names, wget is clearly requesting the full and correct URL, it follows redirections, and saves using the final redirection URL (the latest sources wouldn't follow that last step - it'd save using the request URI by default). If you dislike the filename, then provided you have a recent enough version of wget you can add the --content-disposition option if the server provides a rename header (Content-Disposition); or else use -E to have wget force the file name to end in .html -mjc (05/26/2011 12:19 PM), Jeff Givens wrote: Hi, I know this is an older topic but thanks for replying. I forgot to mention I had already what you listed below and this is the output I get: C:\DOWNLOADwget http://dw.com.com/redir?edId=3siteId=4oId=3000-8022_ 4-10804572ontId=8022_4spi=077d9109e846975d0db9532bd610588flop=linktag=tdw_dl textltype=dl_dlnowpid=11665648mfgId=6290020merId=6290020pguid=HFsQLwoOYJQAA BuImQcAAAGmdestUrl=http%3A%2F%2Fdownload.cnet.com%2F3001-8022_4-10804572.html%3 Fspi%3D077d9109e846975d0db9532bd610588f --2011-05-02 12:34:20-- http://dw.com.com/redir?edId=3siteId=4oId=3000-8022_4 -10804572ontId=8022_4spi=077d9109e846975d0db9532bd610588flop=linktag=tdw_dlt extltype=dl_dlnowpid=11665648mfgId=6290020merId=6290020pguid=HFsQLwoOYJQAAB uImQcAAAGmdestUrl=http%3A%2F%2Fdownload.cnet.com%2F3001-8022_4-10804572.html%3F spi%3D077d9109e846975d0db9532bd610588f Resolving dw.com.com... 216.239.113.95 Connecting to dw.com.com|216.239.113.95|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://download.cnet.com/3001-8022_4-10804572.html?spi=077d9109e846975 d0db9532bd610588f [following] --2011-05-02 12:34:21-- http://download.cnet.com/3001-8022_4-10804572.html?spi= 077d9109e846975d0db9532bd610588f Resolving download.cnet.com... 64.30.224.58 Connecting to download.cnet.com|64.30.224.58|:80... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/html] Saving to: `3001-8022_4-10804572.html@spi=077d9109e846975d0db9532bd610588f.1' [= ] 69,240 77.3K/s in 0.9s 2011-05-02 12:34:22 (77.3 KB/s) - `3001-8022_4-10804572.html@spi=077d9109e846975 d0db9532bd610588f.1' saved [69240] C:\DOWNLOAD Thanks for your help. - Jeff
