I posted this authentication failure problem to the htdig mailing list and got no response. Hopefully developers can have some ideas/solutions on it. Thanks! Frank ---------- Forwarded message ---------- Date: Tue, 28 Sep 1999 10:00:17 -0500 (EST) From: Frank Guangxin Liu <[EMAIL PROTECTED]> To: [EMAIL PROTECTED] Subject: [htdig] htdig fail 2 It seems htdig handles -u option not correct. Today I found htdig failed to index a site which requires no authentication. To narrow the problem, I set start_url: http://bpnet.cummins.com/ limit_urls_to: ${start_url} and rundig -vvvv gives: 1:0:http://bpnet.cummins.com/ New server: bpnet.cummins.com, 80 Retrieval command for http://bpnet.cummins.com/robots.txt: GET /robots.txt HTTP/1.0 User-Agent: htdig/3.1.3 ([EMAIL PROTECTED]) Host: bpnet.cummins.com Header line: HTTP/1.1 404 Object Not Found Header line: Server: Microsoft-IIS/4.0 Header line: Date: Tue, 28 Sep 1999 14:44:47 GMT Header line: Content-Length: 461 Header line: Content-Type: text/html Header line: returnStatus = 1 pushed pick: bpnet.cummins.com, # servers = 1 0:0:0:http://bpnet.cummins.com/: Retrieval command for http://bpnet.cummins.com/: GET / HTTP/1.0 User-Agent: htdig/3.1.3 ([EMAIL PROTECTED]) Authorization: Basic aHRkaWc6aHRkaWcxMA== Host: bpnet.cummins.com Header line: HTTP/1.1 401 Access Denied Header line: WWW-Authenticate: NTLM Header line: WWW-Authenticate: Basic realm="bpnet.cummins.com" Header line: Content-Length: 537 Header line: Content-Type: text/html Header line: returnStatus = 5 not authorized pick: bpnet.cummins.com, # servers = 1 htdig: Run complete htdig: 1 server seen: htdig: bpnet.cummins.com:80 1 document Here is the source of index.htm <!-- body_default.htm --> <HTML> <HEAD> <TITLE>bpcity</TITLE> <META NAME="Generator" CONTENT="NetObjects Fusion 3.0 for Windows"> <FRAMESET BORDER=0 FRAMESPACING=0 FRAMEBORDER=0 ROWS="*,12"> <FRAME NAME="body" SRC="./default.htm" SCROLLING="auto" MARGINWIDTH="2" MARGINHEIGHT="1"> <FRAME NAME="footer" SRC="./footer_defaultmasterborder.htm" SCROLLING="auto" MARGINWIDTH="2" MARGINHEIGHT="1" FRAMEBORDER="no" BORDER="0" NORESIZE> </FRAMESET> </HEAD> </HTML> I can view this url without a problem from netscape browser. By looking at rundig output, it seems like it tries and fails at authentication. While using netscape browser, it doesn;'t prompt for authentication at all and works just fine. When I take a closer look at the rundig script, I found I have a -u xxxxx:xxxx option to htdig because this is my generic rundig script that index the whole intranet and some sites require this xxxxx:xxxx password... After I take out -u xxxxx:xxx option from rundig script, it can index this site without a problem. Now it seems to me htdig may need to do another try if password failed by the server. Frank ------------------------------------ To unsubscribe from the htdig3-dev mailing list, send a message to [EMAIL PROTECTED] containing the single word "unsubscribe" in the SUBJECT of the message.
