Hi, I have trouble getting wget to work when downloading a file using http.curl works fine for the same URL.
Both the commands are being run from the same server at the command line, OS : RH Linux mdc1pr012 2.6.18-238.9.1.el5 Following are the commands and the debug messages, any insights into what the problem could be are appreciated. [dsop@mdc1pr012]$ wget -d -S http://www.preprod.abc.com/tools/90067660.csv Setting --server-response (serverresponse) to 1 DEBUG output created by Wget 1.11.4 Red Hat modified on linux-gnu. --2011-10-14 18:00:58-- http://www.preprod.abc.com/tools/90067660.csv Resolving www.preprod.abc.com... 184.31.131.61 Caching www.preprod.abc.com => 184.31.131.61 Connecting to www.preprod.abc.com|184.31.131.61|:80... connected. Created socket 3. Releasing 0x0000000007683bc0 (new refcount 1). ---request begin--- GET /tools/90067660.csv HTTP/1.0 User-Agent: Wget/1.11.4 Red Hat modified Accept: */* Host: www.preprod.abc.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 302 Found Location: http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y= Cache-Control: no-cache Pragma: no-cache Content-Type: text/html; charset=utf-8 Connection: close Content-Length: 925 ---response end--- HTTP/1.1 302 Found Location: http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y= Cache-Control: no-cache Pragma: no-cache Content-Type: text/html; charset=utf-8 Connection: close Content-Length: 925 Location: http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y=[following] Closed fd 3 --2011-10-14 18:00:58-- http://fd000xnchegrn02/?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y= Resolving fd000xnchegrn02... 11.48.43.72 Caching fd000xnchegrn02 => 11.48.43.72 Connecting to fd000xnchegrn02|11.48.43.72|:80... connected. Created socket 3. Releasing 0x00000000076806e0 (new refcount 1). ---request begin--- GET /?cfru=aHR0cDovL3d3dy5wcmVwcm9kbWFjeXMuZmRzLmNvbS90b29scy85MDA2NzY2MC5jc3Y= HTTP/1.0 User-Agent: Wget/1.11.4 Red Hat modified Accept: */* Host: fd000xnchegrn02 Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 401 Unauthorized Cache-Control: no-cache Pragma: no-cache WWW-Authenticate: NTLM WWW-Authenticate: BASIC realm="Federated_Department_Stores" Content-Type: text/html; charset=utf-8 Proxy-Connection: close Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/ Connection: close Content-Length: 1114 ---response end--- HTTP/1.1 401 Unauthorized Cache-Control: no-cache Pragma: no-cache WWW-Authenticate: NTLM WWW-Authenticate: BASIC realm="Federated_Department_Stores" Content-Type: text/html; charset=utf-8 Proxy-Connection: close Set-Cookie: BCSI-CS-3d1fe99b15515258=2; Path=/ Connection: close Content-Length: 1114 Closed fd 3 Authorization failed. >>>>>>>>>>>>>>> Following is the curl trace info for the same URL [dsop@mdc1pr012 ~]$ curl --trace-ascii tr.out -o out.dat http://www.preprodmacys.fds.com/tools/90067660.csv % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 155k 100 155k 0 0 449k 0 --:--:-- --:--:-- --:--:-- 550k [dsop@mdc1pr012 ~]$ more tr.out == Info: About to connect() to www.preprod.abc.com port 80 == Info: Trying 184.31.131.61... == Info: connected == Info: Connected to www.preprod.abc.com (184.31.131.61) port 80 => Send header, 186 bytes (0xba) 0000: GET /tools/90067660.csv HTTP/1.1 0022: User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 0062: OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5 008b: Host: www.preprod.abc.com 00ab: Accept: */* 00b8: <= Recv header, 17 bytes (0x11) 0000: HTTP/1.1 200 OK <= Recv header, 25 bytes (0x19) 0000: Server: IBM_HTTP_Server <= Recv header, 46 bytes (0x2e) 0000: Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT <= Recv header, 30 bytes (0x1e) 0000: ETag: "b19b9-26f41-7f2b4580" <= Recv header, 22 bytes (0x16) 0000: Accept-Ranges: bytes <= Recv header, 26 bytes (0x1a) 0000: Content-Type: text/plain <= Recv header, 37 bytes (0x25) 0000: Date: Fri, 14 Oct 2011 22:06:15 GMT <= Recv header, 24 bytes (0x18) 0000: Content-Length: 159553 <= Recv header, 24 bytes (0x18) 0000: Connection: Keep-Alive <= Recv header, 127 bytes (0x7f) 0000: Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=4117DB; Domain=.prep 0040: rodmacys.fds.com; expires=Fri, 14-Oct-11 22:36:15 GMT; Path=/ <= Recv header, 70 bytes (0x46) 0000: Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc 0040: .com <= Recv header, 104 bytes (0x68) 0000: Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:36:15 GMT 0040: ; path=/; domain=.preprod.abc.com <= Recv data, 5078 bytes (0x13d6) 0000: 90067660,po-xx-xx-xx,popup, 001d: 90067660,ex-xx-xx-xx,external sites, 0043: 90067660,im-xx-xx-xx,imports, .........MORE LINES.............. <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Also when we run wget from another server against the same URL it works fine...the version on the other server is 1.10.4. So we got that version and installed it on the server mentioned in the above example and tried but we still got the same error message. Following is the debug log from the successful run on the another server : scripts@md000ysesx92 /home/scripts(0)$ wget -d -S http://www.preprod.abc.com/tools/90067660.csv Setting --server-response (serverresponse) to 1 DEBUG output created by Wget 1.10.2 (Red Hat modified) on linux-gnu. --15:03:42-- http://www.preprod.abc.com/tools/90067660.csv => `90067660.csv.3' Resolving www.preprod.abc.com... 184.31.131.61 Caching www.preprod.abc.com => 184.31.131.61 Connecting to www.preprod.abc.com|184.31.131.61|:80... connected. Created socket 3. Releasing 0x000000000063b390 (new refcount 1). ---request begin--- GET /tools/90067660.csv HTTP/1.0 User-Agent: Wget/1.10.2 (Red Hat modified) Accept: */* Host: www.preprod.abc.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.0 200 OK Server: IBM_HTTP_Server Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT ETag: "b19b9-26f41-7f2b4580" Accept-Ranges: bytes Content-Type: text/plain Content-Length: 159553 Date: Fri, 14 Oct 2011 22:03:42 GMT Connection: keep-alive Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=.preprod.abc.com; expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/ Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/; domain=.preprod.abc.com ---response end--- HTTP/1.0 200 OK Server: IBM_HTTP_Server Last-Modified: Fri, 18 Mar 2011 22:42:46 GMT ETag: "b19b9-26f41-7f2b4580" Accept-Ranges: bytes Content-Type: text/plain Content-Length: 159553 Date: Fri, 14 Oct 2011 22:03:42 GMT Connection: keep-alive Set-Cookie: BC_HA_7039BD33D34B38D1_70737B63=41174D; Domain=. preprod.abc.com; expires=Fri, 14-Oct-11 22:33:42 GMT; Path=/ Set-Cookie: shippingCountry=US; path=/; domain=.preprod.abc.com Set-Cookie: stop_mobi=yes; expires=Fri, 14-Oct-2011 22:33:42 GMT; path=/; domain=.preprod.abc.com Registered socket 3 for persistent reuse. cdm: 1 2 3 4 5 6 7 8 Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry 2011-10-14 15:33:42] BC_HA_7039BD33D34B38D1_70737B63 41174D cdm: 1 2 3 4 5 6 7 8 Stored cookie preprod.abc.com -1 (ANY) / <session> <insecure> [expiry none] shippingCountry US cdm: 1 2 3 4 5 6 7 8 Stored cookie preprod.abc.com -1 (ANY) / <permanent> <insecure> [expiry 2011-10-14 15:33:42] stop_mobi yes Length: 159,553 (156K) [text/plain] 100%[=================================================================================================================================================================================================>] 159,553 105.31K/s 15:03:44 (105.15 KB/s) - `90067660.csv.3' saved [159553/159553] PS : I changed the domain names mentioned in the logs above Thanks, Vishwanath
