I was hoping that someone on the list might be able to assist me. I've been successfully using wget 1.10.2 in an internal intranet environment to spider IIS 6.0 sites and Sharepoint sites. Here's an example of a verbose, debug enabled spider:
C:\wget>wget -v -d --http-user="DOMAIN\FIRST_LAST" --http-password="XXXXX" --no-proxy http://some.sharepointsite.com Setting --http-user (httpuser) to "DOMAIN\FIRST_LAST" Setting --http-password (httppassword) to XXXXX Setting --proxy (useproxy) to 0 DEBUG output created by Wget 1.10.2 on Windows. --01:30:28-- http://some.sharepointsite.com/ => `index.html' Resolving some.sharepointsite.com... seconds 0.00, XXX.XXX.XXX.XXX Caching some.sharepointsite.com => XXX.XXX.XXX.XXX Connecting to some.sharepointsite.com|XXX.XXX.XXX.XXX|:80... seconds 0.00, connected. Created socket 1892. Releasing 0x003d49a8 (new refcount 1). ---request begin--- GET / HTTP/1.0 User-Agent: Wget/1.10.2 Accept: */* Authorization: Basic YW1lcmljYXNcYnJ5YW5faG9mZnBhdWlyOlNBUzE2QDliOGo3Nmg= Host: some.sharepointsite.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 401 Access denied Connection: close Date: Sun, 02 Nov 2008 06:30:28 GMT Server: Microsoft-IIS/6.0 WWW-Authenticate: NTLM MicrosoftSharePointTeamServices: 12.0.0.6315 X-Powered-By: ASP.NET Content-type: text/html ---response end--- 401 Access denied Closed fd 1892 Empty NTLM message, starting transaction. Creating a type-1 NTLM message. Found some.sharepointsite.com in host_name_addresses_map (003D49A8) Connecting to some.sharepointsite.com|XXX.XXX.XXX.XXX|:80... seconds 0.00, connected. Created socket 1892. Releasing 0x003d49a8 (new refcount 1). ---request begin--- GET / HTTP/1.0 User-Agent: Wget/1.10.2 Accept: */* Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA= Host: some.sharepointsite.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 401 Unauthorized Content-Length: 1539 Content-Type: text/html Server: Microsoft-IIS/6.0 WWW-Authenticate: NTLM TlRMTVNTUAACAAAAAAAAADgAAAACAgACazTDgoeCGb4AAAAAAAAAAAAA AA4AAAABQLODgAAAA8= MicrosoftSharePointTeamServices: 12.0.0.6315 X-Powered-By: ASP.NET Date: Sun, 02 Nov 2008 06:30:28 GMT Connection: keep-alive ---response end--- 401 Unauthorized Registered socket 1892 for persistent reuse. Skipping 1539 bytes of body: [<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd";> <HTML><HEAD><TITLE>You are not authorized to view this page</TITLE> <META HTTP-EQUIV="Content-Type" Content="text/html; charset=Windows-1252"> <STYLE type="text/css"> BODY { font: 8pt/12pt verdana } H1 { font: 13pt/15pt verdana } H2 { font: 8pt/12pt verdana } A:link { color: red } A:visited { color: maroon } </STYLE> </HEAD><BODY><TABLE width=500 border=0 cellspacing=10><TR><TD> <h1>You are not authorized to view this page</h1> You do not have permission to view this directory or page using the credentials that you supplied. <hr> <p>Please try the following:</p> <ul> <li>Contact the Web site administrator if you believe you should be able to vie this directory or page.</li> <li>Click the <a href="javascript:location.reload()">Refresh</a> button to try gain with different credentials.</li> </ul> <h2>HTTP Error 401.1 - Unauthorized: Access is denied due to invalid credential .<br>Internet Information Services (IIS)</h2> <hr> <p>Technical Information (for support personnel)</p> <ul> <li>Go to <a href="http://go.microsoft.com/fwlink/?linkid=8180";>Microsoft Produ t Support Services</a> and perform a title search for the words <b>HTTP</b> and <b>401</b>.</li> <li>Open <b>IIS Help</b>, which is accessible in IIS Manager (inetmgr), and search for topics titled <b>Authentication</b>, <b>Access Control</b>, and <b>About Custom Error Messages</b>.</li> </ul> </TD></TR></TABLE></BODY></HTML> ] done. Received a type-2 NTLM message. Creating a type-3 NTLM message. Reusing existing connection to some.sharepointsite.com:80. Reusing fd 1892. ---request begin--- GET / HTTP/1.0 User-Agent: Wget/1.10.2 Accept: */* Authorization: NTLM TlRMTVNTUAADAAAAGAAYAFcAAAAYABgAbwAAAAgACABAAAAADwAPAEgAAAA AAAAVwAAAAAAAACHAAAAAYIAAGFtZXJpY2FzYnJ5YW5faG9mZnBhdWlyZYGWc7NKTESDADIU3oGrrRH hZk7fx6hFn4RRpBS29cxvACr4AXt2Tb1Mib9Ldcv Host: some.sharepointsite.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 302 Object Moved Content-Length: 162 Content-Type: text/html Location: http://some.sharepointsite.com/Pages/default.aspx Server: Microsoft-IIS/6.0 MicrosoftSharePointTeamServices: 12.0.0.6315 X-Powered-By: ASP.NET Date: Sun, 02 Nov 2008 06:30:28 GMT Connection: keep-alive ---response end--- 302 Object Moved Location: http://some.sharepointsite.com/Pages/default.aspx [following] Skipping 162 bytes of body: [<head><title>Document Moved</title></head> <body><h1>Object Moved</h1>This document may be found <a HREF="http://moss.dell com/Pages/default.aspx">here</a></body>] done. --01:30:28-- http://some.sharepointsite.com/Pages/default.aspx => `default.aspx' Reusing existing connection to some.sharepointsite.com:80. Reusing fd 1892. ---request begin--- GET /Pages/default.aspx HTTP/1.0 User-Agent: Wget/1.10.2 Accept: */* Host: some.sharepointsite.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 200 OK Connection: keep-alive Date: Sun, 02 Nov 2008 06:30:29 GMT Server: Microsoft-IIS/6.0 MicrosoftSharePointTeamServices: 12.0.0.6315 X-Powered-By: ASP.NET X-AspNet-Version: 2.0.50727 Set-Cookie: WSS_KeepSessionAuthenticated=80; path=/ Cache-Control: private, max-age=0 Expires: Sat, 18 Oct 2008 06:30:28 GMT Last-Modified: Sun, 02 Nov 2008 06:30:28 GMT Content-Type: text/html; charset=utf-8 Content-Length: 28419 ---response end--- 200 OK Stored cookie some.sharepointsite.com -1 (ANY) / <session> <insecure> [expiry none] WSS_K epSessionAuthenticated 80 Length: 28,419 (28K) [text/html] 100%[====================================>] 28,419 --.--K/s 01:30:29 (112.46 MB/s) - `default.aspx' saved [28419/28419] Recently, I downloaded the new 1.11.4 wget from the mirrors and wanted to use some of the new features (like --ignore-case). But now, when I attempt to run the exact same command, I get this: C:\wget1.11.4>wget -v -d --http-user="DOMAIN\First_Last" --http-password= "[EMAIL PROTECTED]" --no-proxy http://some.sharepointsite.com Setting --http-user (httpuser) to DOMAIN\First_Last Setting --http-password (httppassword) to [EMAIL PROTECTED] Setting --proxy (useproxy) to 0 DEBUG output created by Wget 1.11.4 on Windows-MSVC. --2008-11-02 01:38:24-- http://some.sharepointsite.com/ Host `some.sharepointsite.com' has not issued a general basic challenge. Resolving some.sharepointsite.com... seconds 0.00, XXX.XXX.XXX.XXX Caching some.sharepointsite.com => XXX.XXX.XXX.XXX Connecting to some.sharepointsite.com|XXX.XXX.XXX.XXX|:80... seconds 0.00, connected. Created socket 1892. Releasing 0x003e2b68 (new refcount 1). ---request begin--- GET / HTTP/1.0 User-Agent: Wget/1.11.4 Accept: */* Host: some.sharepointsite.com Connection: Keep-Alive ---request end--- HTTP request sent, awaiting response... ---response begin--- HTTP/1.1 401 Access denied Connection: close Date: Sun, 02 Nov 2008 06:38:24 GMT Server: Microsoft-IIS/6.0 WWW-Authenticate: NTLM MicrosoftSharePointTeamServices: 12.0.0.6315 X-Powered-By: ASP.NET Content-type: text/html ---response end--- 401 Access denied Closed fd 1892 Unknown authentication scheme. Authorization failed. It seems as though the new 1.11.4 build is not completing the entire multi-step NTLM handshake like the 1.10.2 version does. Has anyone else seen similar behavior or perhaps could suggest a few troubleshooting steps to guide me in the right direction? I did find a discussion here, but I was unable to validate - http://www.experts-exchange.com/Networking/Linux_Networking/Q_23371057.html?sfQueryTermInfo=1+ntlm+wget Also, as you can tell in the examples, I've modified the username, but it is Active Directory, so it does have the "\" between the DOMAIN and First_Last. In addition, the password I am using contains the special characters "$" and "@" They both work with the 1.10.2 build, but not the new 1.11.4. Thanks in Advance! BJ Hoffpauir
