Re: Syntax question ...
Daniel Stenberg <[EMAIL PROTECTED]> writes: > On Thu, 22 Jan 2004, Simons, Rick wrote: > >> curl https://server/file -uuser:pass >> Virtual user user logged in. [...] > In my eyes, this looks like the correct output from curl. Wasn't it? I think that Rick expects to see a complete HTML page rather than the "Virtual user X logged in" message. Both Wget and curl exhibit the incorrect output. Rick, if you are inclined to fiddle some more with this, another possibility would be to trace the communication between a browser where the page works and the web server. Since trace and tcpdump don't play well with SSL, the easiest way (I'm aware of) to accomplish this is using something like the "liveheaders" Mozilla extension; get it from http://livehttpheaders.mozdev.org/. Specifically, use livehttpheaders to see what headers Mozilla was sending, and use Wget's `--header' option to send out the same request. See if things work then. If not, it might be useful to compare the response in both cases. If things fail with Wget again, then the next thing to blame would be OpenSSL... it might be useful to see if *any* OpenSSL-using browser (e.g. `links' or `lynx') can display the page correctly.
RE: Syntax question ...
On Thu, 22 Jan 2004, Simons, Rick wrote: > curl https://server/file -uuser:pass > Virtual user user logged in. > > No file created locally. Chalk it up as a http server flaw? Uh, curl doesn't create any file when used like that. It outputs the downloaded data to stdout unless you use an option to direct it elsewhere. You can use "curl -v -i [URL]" to get to see the full request curl sends and all the headers it receives back. Then those could be compared to what wget sends/gets. In my eyes, this looks like the correct output from curl. Wasn't it? -- -=- Daniel Stenberg -=- http://daniel.haxx.se -=- ech`echo xiun|tr nu oc|sed 'sx\([sx]\)\([xoi]\)xo un\2\1 is xg'`ol
RE: Syntax question ...
I tried all your suggestions except the cvs one, and the results were the same. curl -V curl 7.9.8 (i386-redhat-linux-gnu) libcurl 7.9.8 (OpenSSL 0.9.7a) (ipv6 enabled) curl https://server/file -uuser:pass Virtual user user logged in. No file created locally. Chalk it up as a http server flaw? -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Thursday, January 22, 2004 1:10 PM To: Simons, Rick Cc: '[EMAIL PROTECTED]' Subject: Re: Syntax question ... Thanks for persisting with this. It doesn't look like a mishandled redirection -- the response headers exist and they don't request a redirection or any kind of refresh. access_log shows that 30 bytes have been transmitted. As it happens, the string "Virtual user ricks logged in.\n" is exactly thirty bytes long. If there is an error, it seems to be on the server's side. At least I can't see that Wget is doing anything obviously wrong in the request. To modify the request, you could try one or more of these: * Use `--user-agent' to pretend to be a web browser. The server software might be checking for browser versions. * Use `--no-http-keep-alive' to get rid of the `Connection' header. * Use `--header "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q= 0.8,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1"' to force the Accept header to the value that Mozilla uses and that apparently works on your site. For this to work, you need the CVS version of Wget. Even better would be if you could try the same site with another OpenSSL-using client, such as "curl". If it doesn't work, the problem might be related to the site. If it does work, we can take a look at what curl does differently.
Re: Syntax question ...
Thanks for persisting with this. It doesn't look like a mishandled redirection -- the response headers exist and they don't request a redirection or any kind of refresh. access_log shows that 30 bytes have been transmitted. As it happens, the string "Virtual user ricks logged in.\n" is exactly thirty bytes long. If there is an error, it seems to be on the server's side. At least I can't see that Wget is doing anything obviously wrong in the request. To modify the request, you could try one or more of these: * Use `--user-agent' to pretend to be a web browser. The server software might be checking for browser versions. * Use `--no-http-keep-alive' to get rid of the `Connection' header. * Use `--header "Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,image/jpeg,image/gif;q=0.2,*/*;q=0.1"' to force the Accept header to the value that Mozilla uses and that apparently works on your site. For this to work, you need the CVS version of Wget. Even better would be if you could try the same site with another OpenSSL-using client, such as "curl". If it doesn't work, the problem might be related to the site. If it does work, we can take a look at what curl does differently.
RE: Syntax question ...
Thanks for the continued assistance, i'd like to get this nailed down to a http server issue or a wget issue so I can complete or stop wget certification of our product. wget -o output -d -S https://server/file --http-user=user --http-passwd=pass cat file Virtual user ricks logged in. cat output DEBUG output created by Wget 1.9.1 on linux-gnu. --09:13:48-- https://server/file => `file' Resolving server... ip Caching server => ip Connecting to server[ip]:443... connected. Created socket 4. Releasing 0x8122b98 (new refcount 1). ---request begin--- GET /file HTTP/1.0 User-Agent: Wget/1.9.1 Host: host Accept: */* Connection: Keep-Alive Authorization: Basic cmlja3M6cmlja3MyNjI2 ---request end--- HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Thu, 22 Jan 2004 14:22:45 GMT 2 Date: Thu, 22 Jan 2004 14:22:45 GMTServer: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/0.9.6g SecureTransport/4.1.2 3 Server: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/0.9.6g SecureTransport/4.1.2Set-Cookie: FDX=8gLvC68DA9Q7oK7WokR89w==; path=/ 4 Set-Cookie: FDX=8gLvC68DA9Q7oK7WokR89w==; path=/ Stored cookie server 443 / nonpermanent 0 FDX 8gLvC68DA9Q7oK7WokR89w== Accept-Ranges: bytes 5 Accept-Ranges: bytesExpires: Thu, 01 Jan 1970 00:00:00 GMT 6 Expires: Thu, 01 Jan 1970 00:00:00 GMTFeatures: CHPWD;RTCK;STCK;ASC 7 Features: CHPWD;RTCK;STCK;ASCConnection: close 8 Connection: closeContent-Type: text/plain; charset=UTF-8 9 Content-Type: text/plain; charset=UTF-8 0K 292.97 KB/s Closing fd 4 09:13:49 (292.97 KB/s) - `file' saved [30] cat ssl_log [Thu Jan 22 08:22:45 200 16984] [info] Connection to child 9 established (server server:443, client ip) [Thu Jan 22 08:22:45 200 16984] [info] Seeding PRNG with 1160 bytes of entropy [Thu Jan 22 08:22:45 200 16984] [info] Initial (No.1) HTTPS request received for child 9 (server server:443) [Thu Jan 22 08:22:46 200 16984] [info] Connection to child 9 closed with standard shutdown (server server:443, client ip) cat ssl_access_log 10.211.90.71 - user [22/Jan/2004:08:22:46 -0600] "GET /file HTTP/1.0" 200 30 cat ssl_error_log [Thu Jan 22 08:22:45 2004] [info] VIRTUAL HTTP LOGIN FROM ip [ip], user (class virt) cat error_log [Thu Jan 22 08:22:45 2004] [info] [16984] Setting tables to 0 [Thu Jan 22 08:22:45 2004] [info] [16984] fdx_request_handler called for URL /file [Thu Jan 22 08:22:45 2004] [info] [16984] Timeout set to 300 [Thu Jan 22 08:22:45 2004] [info] [16984] fdx_authenticate Starting new session [Thu Jan 22 08:22:45 2004] [info] [16984] Timeout set to 300 [Thu Jan 22 08:22:45 2004] [info] [16984] isWSProxy=0 timeout=1074781665 lgi_failure_threshold=3 [Thu Jan 22 08:22:45 2004] [info] [16984] Is Browser client 1 [Thu Jan 22 08:22:45 2004] [info] [16984] fdx_check_credentials: useldap 0 reqldap 0 user user [Thu Jan 22 08:22:45 2004] [info] [16984] Timeout set to 300 [Thu Jan 22 08:22:45 2004] [info] [16984] fdx_map_tilde: pw.dir "server/directory/" ui.type "1" r->uri "file" Is that last line the issue? -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Thursday, January 22, 2004 9:00 AM To: Simons, Rick Cc: '[EMAIL PROTECTED]' Subject: Re: Syntax question ... "Simons, Rick" <[EMAIL PROTECTED]> writes: > Another followup question(s), and thanks for the continued assistance ...: > > -S > --server-response > Print the headers sent by HTTP servers and responses sent by FTP servers. > > I misinterpreted this switch that the file would still be > downloaded, but the console would see the server messages not > that the server messages would be concatinated onto the file. Actually, your interpretation is correct. I mistook `-S' for `-s' which tells Wget to prepend the headers to the file. Sorry about the confusion. In other words, what you wrote should have worked. > Perhaps it can be reworded in future versions? Also, shouldn't the > contents of the file still appear in the file, along with the server > responses??? It should. > cat testfile > Virtual user username logged in. I have seen this bug reported before, always in conjunction with SSL, but I cannot repeat it. What is the server response? Is the server maybe sending a redirection that Wget does not recognize?
Re: Syntax question ...
"Simons, Rick" <[EMAIL PROTECTED]> writes: > Another followup question(s), and thanks for the continued assistance ...: > > -S > --server-response > Print the headers sent by HTTP servers and responses sent by FTP servers. > > I misinterpreted this switch that the file would still be > downloaded, but the console would see the server messages not > that the server messages would be concatinated onto the file. Actually, your interpretation is correct. I mistook `-S' for `-s' which tells Wget to prepend the headers to the file. Sorry about the confusion. In other words, what you wrote should have worked. > Perhaps it can be reworded in future versions? Also, shouldn't the > contents of the file still appear in the file, along with the server > responses??? It should. > cat testfile > Virtual user username logged in. I have seen this bug reported before, always in conjunction with SSL, but I cannot repeat it. What is the server response? Is the server maybe sending a redirection that Wget does not recognize?
Re: Syntax question ...
"Simons, Rick" <[EMAIL PROTECTED]> writes: > I got wget compiled with ssl support now, and have a followup question ... > I'm getting the local file created but populated with a server response, not > the actual contents of the remote file. See example: > > wget -d -S https://server/testfile --http-user=user --http-passwd=pass The `-S' option tells Wget to prepend the server response to the contents of the file. It's not a good idea to use that option unless you mean it.
RE: Syntax question ...
Another followup question(s), and thanks for the continued assistance ...: -S --server-response Print the headers sent by HTTP servers and responses sent by FTP servers. I misinterpreted this switch that the file would still be downloaded, but the console would see the server messages not that the server messages would be concatinated onto the file. Perhaps it can be reworded in future versions? Also, shouldn't the contents of the file still appear in the file, along with the server responses??? wget https://server/testfile --http-user=user --http-passwd=pass --11:24:28-- https://server/testfile => `testfile' Resolving server... ip Connecting to server[ip]:443... connected. HTTP request sent, awaiting response... 200 OK Length: unspecified [text/plain] [ <=> ] 30 --.--K/s 11:24:29 (292.97 KB/s) - `testfile' saved [30] cat testfile Virtual user username logged in. As you can see, the actual contents of the remote file are still not in the local copy ... and the -S switch was not used in this example. Thoughts? -Original Message- From: Post, Mark K [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 21, 2004 11:18 AM To: '[EMAIL PROTECTED]' Subject: RE: Syntax question ... Well, that's what you're telling it to do with the -S option, so why are you surprised? "man wget", then "/-S" Mark Post -Original Message- From: Simons, Rick [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 21, 2004 11:09 AM To: '[EMAIL PROTECTED]' Subject: RE: Syntax question ... I got wget compiled with ssl support now, and have a followup question ... I'm getting the local file created but populated with a server response, not the actual contents of the remote file. See example: wget -d -S https://server/testfile --http-user=user --http-passwd=pass DEBUG output created by Wget 1.9.1 on linux-gnu. --10:55:06-- https://server/testfile => `testfile' Resolving server... ip Caching server => ip Connecting to server[ip]:443... connected. Created socket 3. Releasing 0x81229f0 (new refcount 1). ---request begin--- GET /testfile HTTP/1.0 User-Agent: Wget/1.9.1 Host: server Accept: */* Connection: Keep-Alive Authorization: Basic cmlja3M6cmlja3MyNjI2 ---request end--- HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Wed, 21 Jan 2004 16:04:01 GMT 2 Date: Wed, 21 Jan 2004 16:04:01 GMTServer: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/ 0.9.6g SecureTransport/4.1.2 3 Server: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/0.9.6g SecureTransport/4.1.2Set-Coo kie: FDX=ocjoMt028Um+ri2vZQ0L6g==; path=/ 4 Set-Cookie: FDX=ocjoMt028Um+ri2vZQ0L6g==; path=/ Stored cookie filed2 443 / nonpermanent 0 FDX ocjoMt028Um+ri2vZQ0L6g== Accept-Ranges: bytes 5 Accept-Ranges: bytesExpires: Thu, 01 Jan 1970 00:00:00 GMT 6 Expires: Thu, 01 Jan 1970 00:00:00 GMTFeatures: CHPWD;RTCK;STCK;ASC 7 Features: CHPWD;RTCK;STCK;ASCConnection: close 8 Connection: closeContent-Type: text/plain; charset=UTF-8 9 Content-Type: text/plain; charset=UTF-8 [ <=> ] 30--.--K/s Closing fd 3 10:55:07 (292.97 KB/s) - `testfile' saved [30] cat testfile Virtual user username logged in. ssl access log: ip - user [21/Jan/2004:10:04:02 -0600] "GET /testfile HTTP/1.0" 200 30 ssl error log: [Wed Jan 21 10:04:01 2004] [info] VIRTUAL HTTP LOGIN FROM ip [ip], user (class virt) Further thoughts or suggestions? -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 21, 2004 9:41 AM To: Simons, Rick Cc: '[EMAIL PROTECTED]' Subject: Re: Syntax question ... "Simons, Rick" <[EMAIL PROTECTED]> writes: > Greetings all. > > I've posted in the past, but never really have gotten connectivity to a > https server I support using the wget application. I've looked in the > manual, on the website and searched the Internet but am not getting very > far. > > > wget -V > GNU Wget 1.9 > > wget -d -S https://server/file > https://server/file: Unsupported scheme. This error message indicates that your version of Wget is compiled without SSL support. > I then decided (based on previous instruction from this group) to recompile > wget with ssl. This is on a RH9 box, with openssl libs in > /usr/include/openssl > > ./configure --with-ssl=/usr/include/openssl/ > > Looking for SSL libraries in /usr/include/openssl/ > checking for includes... not found > ERROR: Failed to find OpenSSL libraries. Try just `./configure', it should find the SSL libraries in the default location. At least it does for me -- I use RH9.
RE: Syntax question ...
Well, that's what you're telling it to do with the -S option, so why are you surprised? "man wget", then "/-S" Mark Post -Original Message- From: Simons, Rick [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 21, 2004 11:09 AM To: '[EMAIL PROTECTED]' Subject: RE: Syntax question ... I got wget compiled with ssl support now, and have a followup question ... I'm getting the local file created but populated with a server response, not the actual contents of the remote file. See example: wget -d -S https://server/testfile --http-user=user --http-passwd=pass DEBUG output created by Wget 1.9.1 on linux-gnu. --10:55:06-- https://server/testfile => `testfile' Resolving server... ip Caching server => ip Connecting to server[ip]:443... connected. Created socket 3. Releasing 0x81229f0 (new refcount 1). ---request begin--- GET /testfile HTTP/1.0 User-Agent: Wget/1.9.1 Host: server Accept: */* Connection: Keep-Alive Authorization: Basic cmlja3M6cmlja3MyNjI2 ---request end--- HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Wed, 21 Jan 2004 16:04:01 GMT 2 Date: Wed, 21 Jan 2004 16:04:01 GMTServer: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/ 0.9.6g SecureTransport/4.1.2 3 Server: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/0.9.6g SecureTransport/4.1.2Set-Coo kie: FDX=ocjoMt028Um+ri2vZQ0L6g==; path=/ 4 Set-Cookie: FDX=ocjoMt028Um+ri2vZQ0L6g==; path=/ Stored cookie filed2 443 / nonpermanent 0 FDX ocjoMt028Um+ri2vZQ0L6g== Accept-Ranges: bytes 5 Accept-Ranges: bytesExpires: Thu, 01 Jan 1970 00:00:00 GMT 6 Expires: Thu, 01 Jan 1970 00:00:00 GMTFeatures: CHPWD;RTCK;STCK;ASC 7 Features: CHPWD;RTCK;STCK;ASCConnection: close 8 Connection: closeContent-Type: text/plain; charset=UTF-8 9 Content-Type: text/plain; charset=UTF-8 [ <=> ] 30--.--K/s Closing fd 3 10:55:07 (292.97 KB/s) - `testfile' saved [30] cat testfile Virtual user username logged in. ssl access log: ip - user [21/Jan/2004:10:04:02 -0600] "GET /testfile HTTP/1.0" 200 30 ssl error log: [Wed Jan 21 10:04:01 2004] [info] VIRTUAL HTTP LOGIN FROM ip [ip], user (class virt) Further thoughts or suggestions? -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 21, 2004 9:41 AM To: Simons, Rick Cc: '[EMAIL PROTECTED]' Subject: Re: Syntax question ... "Simons, Rick" <[EMAIL PROTECTED]> writes: > Greetings all. > > I've posted in the past, but never really have gotten connectivity to a > https server I support using the wget application. I've looked in the > manual, on the website and searched the Internet but am not getting very > far. > > > wget -V > GNU Wget 1.9 > > wget -d -S https://server/file > https://server/file: Unsupported scheme. This error message indicates that your version of Wget is compiled without SSL support. > I then decided (based on previous instruction from this group) to recompile > wget with ssl. This is on a RH9 box, with openssl libs in > /usr/include/openssl > > ./configure --with-ssl=/usr/include/openssl/ > > Looking for SSL libraries in /usr/include/openssl/ > checking for includes... not found > ERROR: Failed to find OpenSSL libraries. Try just `./configure', it should find the SSL libraries in the default location. At least it does for me -- I use RH9.
RE: Syntax question ...
I got wget compiled with ssl support now, and have a followup question ... I'm getting the local file created but populated with a server response, not the actual contents of the remote file. See example: wget -d -S https://server/testfile --http-user=user --http-passwd=pass DEBUG output created by Wget 1.9.1 on linux-gnu. --10:55:06-- https://server/testfile => `testfile' Resolving server... ip Caching server => ip Connecting to server[ip]:443... connected. Created socket 3. Releasing 0x81229f0 (new refcount 1). ---request begin--- GET /testfile HTTP/1.0 User-Agent: Wget/1.9.1 Host: server Accept: */* Connection: Keep-Alive Authorization: Basic cmlja3M6cmlja3MyNjI2 ---request end--- HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Wed, 21 Jan 2004 16:04:01 GMT 2 Date: Wed, 21 Jan 2004 16:04:01 GMTServer: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/ 0.9.6g SecureTransport/4.1.2 3 Server: Apache/1.3.26 (Unix) mod_ssl/2.8.10 OpenSSL/0.9.6g SecureTransport/4.1.2Set-Coo kie: FDX=ocjoMt028Um+ri2vZQ0L6g==; path=/ 4 Set-Cookie: FDX=ocjoMt028Um+ri2vZQ0L6g==; path=/ Stored cookie filed2 443 / nonpermanent 0 FDX ocjoMt028Um+ri2vZQ0L6g== Accept-Ranges: bytes 5 Accept-Ranges: bytesExpires: Thu, 01 Jan 1970 00:00:00 GMT 6 Expires: Thu, 01 Jan 1970 00:00:00 GMTFeatures: CHPWD;RTCK;STCK;ASC 7 Features: CHPWD;RTCK;STCK;ASCConnection: close 8 Connection: closeContent-Type: text/plain; charset=UTF-8 9 Content-Type: text/plain; charset=UTF-8 [ <=> ] 30--.--K/s Closing fd 3 10:55:07 (292.97 KB/s) - `testfile' saved [30] cat testfile Virtual user username logged in. ssl access log: ip - user [21/Jan/2004:10:04:02 -0600] "GET /testfile HTTP/1.0" 200 30 ssl error log: [Wed Jan 21 10:04:01 2004] [info] VIRTUAL HTTP LOGIN FROM ip [ip], user (class virt) Further thoughts or suggestions? -Original Message- From: Hrvoje Niksic [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 21, 2004 9:41 AM To: Simons, Rick Cc: '[EMAIL PROTECTED]' Subject: Re: Syntax question ... "Simons, Rick" <[EMAIL PROTECTED]> writes: > Greetings all. > > I've posted in the past, but never really have gotten connectivity to a > https server I support using the wget application. I've looked in the > manual, on the website and searched the Internet but am not getting very > far. > > > wget -V > GNU Wget 1.9 > > wget -d -S https://server/file > https://server/file: Unsupported scheme. This error message indicates that your version of Wget is compiled without SSL support. > I then decided (based on previous instruction from this group) to recompile > wget with ssl. This is on a RH9 box, with openssl libs in > /usr/include/openssl > > ./configure --with-ssl=/usr/include/openssl/ > > Looking for SSL libraries in /usr/include/openssl/ > checking for includes... not found > ERROR: Failed to find OpenSSL libraries. Try just `./configure', it should find the SSL libraries in the default location. At least it does for me -- I use RH9.
Re: Syntax question ...
"Simons, Rick" <[EMAIL PROTECTED]> writes: > Greetings all. > > I've posted in the past, but never really have gotten connectivity to a > https server I support using the wget application. I've looked in the > manual, on the website and searched the Internet but am not getting very > far. > > > wget -V > GNU Wget 1.9 > > wget -d -S https://server/file > https://server/file: Unsupported scheme. This error message indicates that your version of Wget is compiled without SSL support. > I then decided (based on previous instruction from this group) to recompile > wget with ssl. This is on a RH9 box, with openssl libs in > /usr/include/openssl > > ./configure --with-ssl=/usr/include/openssl/ > > Looking for SSL libraries in /usr/include/openssl/ > checking for includes... not found > ERROR: Failed to find OpenSSL libraries. Try just `./configure', it should find the SSL libraries in the default location. At least it does for me -- I use RH9.
