You can log the browser session and then compare the working browser session header log with the httpClient header log below and see which headers are different. You can use tcpflow or wireshark to capture the http traffic from your browser. Firefox also has some plugins for logging http headers to compare with your httpClient log below. Im not too familiar with httpClient auth, but this will show you what httpClient headers are different from your browser's headers.
hth. On Wed, Jan 5, 2011 at 8:08 AM, Claudio Martella <[email protected] > wrote: > Hello list, > > I'm using Apache Nutch to crawl my intranet which is under DIGEST > authentication (nutch is using httpclient 3.1). > As the client is failing the auth with the same credentials my browser > is succeeding, I wrote an example app to try figure out what's going wrong. > > Here it is: > > HttpClient client = new HttpClient(); > client.getParams().setAuthenticationPreemptive(true); > Credentials defaultcreds = new > UsernamePasswordCredentials("user", "*******"); > List authPrefs = new ArrayList(); > authPrefs.add(AuthPolicy.DIGEST); > client.getParams().setParameter(AuthPolicy.AUTH_SCHEME_PRIORITY, > authPrefs); > client.getState().setCredentials(AuthScope.ANY, defaultcreds); > HttpMethod method = new GetMethod("http://192.168.10.209:8090";); > > > What I can see from the logs is that the client is trying to > authenticate with Basic authentication but the server expects NTLM and > only NTLM. Am I reading it correctly? > Why isn't it trying to authenticate with Digest as requested? > > > Here are the logs: > > 2011/01/05 13:25:07:566 CET [DEBUG] HttpClient - Java version: 1.6.0_22 > 2011/01/05 13:25:07:574 CET [DEBUG] HttpClient - Java vendor: Apple Inc. > 2011/01/05 13:25:07:574 CET [DEBUG] HttpClient - Java class path: > > /Users/hammer/TIS/java-hacking/auth-test/target/classes:/Users/hammer/.m2/repository/commons-codec/commons-codec/1.2/commons-codec-1.2.jar:/Users/hammer/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/Users/hammer/.m2/repository/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar:/Users/hammer/.m2/repository/org/apache/httpcomponents/httpclient/4.0.3/httpclient-4.0.3.jar:/Users/hammer/.m2/repository/org/apache/httpcomponents/httpcore/4.0.1/httpcore-4.0.1.jar:/Users/hammer/.m2/repository/junit/junit/3.8.1/junit-3.8.1.jar:/Users/hammer/.m2/repository/log4j/log4j/1.2.14/log4j-1.2.14.jar > 2011/01/05 13:25:07:574 CET [DEBUG] HttpClient - Operating system name: > Mac OS X > 2011/01/05 13:25:07:574 CET [DEBUG] HttpClient - Operating system > architecture: x86_64 > 2011/01/05 13:25:07:574 CET [DEBUG] HttpClient - Operating system > version: 10.5.8 > 2011/01/05 13:25:07:697 CET [DEBUG] HttpClient - SUN 1.6: SUN (DSA > key/parameter generation; DSA signing; SHA-1, MD5 digests; SecureRandom; > X.509 certificates; JKS keystore; PKIX CertPathValidator; PKIX > CertPathBuilder; LDAP, Collection CertStores, JavaPolicy Policy; > JavaLoginConfig Configuration) > 2011/01/05 13:25:07:698 CET [DEBUG] HttpClient - Apple 1.0: Apple > Provider (implements DES, Triple DES, AES, Blowfish, PBE, > Diffie-Hellman, HMAC/MD5, HMAC/SHA1) > 2011/01/05 13:25:07:698 CET [DEBUG] HttpClient - SunRsaSign 1.5: Sun RSA > signature provider > 2011/01/05 13:25:07:698 CET [DEBUG] HttpClient - SunJSSE 1.6: Sun JSSE > provider(PKCS12, SunX509 key/trust factories, SSLv3, TLSv1) > 2011/01/05 13:25:07:698 CET [DEBUG] HttpClient - SunJCE 1.6: SunJCE > Provider (implements RSA, DES, Triple DES, AES, Blowfish, ARCFOUR, RC2, > PBE, Diffie-Hellman, HMAC) > 2011/01/05 13:25:07:698 CET [DEBUG] HttpClient - SunJGSS 1.0: Sun > (Kerberos v5, SPNEGO) > 2011/01/05 13:25:07:698 CET [DEBUG] HttpClient - SunSASL 1.5: Sun SASL > provider(implements client mechanisms for: DIGEST-MD5, GSSAPI, EXTERNAL, > PLAIN, CRAM-MD5; server mechanisms for: DIGEST-MD5, GSSAPI, CRAM-MD5) > 2011/01/05 13:25:07:698 CET [DEBUG] HttpClient - XMLDSig 1.0: XMLDSig > (DOM XMLSignatureFactory; DOM KeyInfoFactory) > 2011/01/05 13:25:07:698 CET [DEBUG] HttpClient - SunPCSC 1.6: Sun PC/SC > provider > 2011/01/05 13:25:07:703 CET [DEBUG] DefaultHttpParams - Set parameter > http.useragent = Jakarta Commons-HttpClient/3.1 > 2011/01/05 13:25:07:705 CET [DEBUG] DefaultHttpParams - Set parameter > http.protocol.version = HTTP/1.1 > 2011/01/05 13:25:07:706 CET [DEBUG] DefaultHttpParams - Set parameter > http.connection-manager.class = class > org.apache.commons.httpclient.SimpleHttpConnectionManager > 2011/01/05 13:25:07:706 CET [DEBUG] DefaultHttpParams - Set parameter > http.protocol.cookie-policy = default > 2011/01/05 13:25:07:706 CET [DEBUG] DefaultHttpParams - Set parameter > http.protocol.element-charset = US-ASCII > 2011/01/05 13:25:07:706 CET [DEBUG] DefaultHttpParams - Set parameter > http.protocol.content-charset = ISO-8859-1 > 2011/01/05 13:25:07:708 CET [DEBUG] DefaultHttpParams - Set parameter > http.method.retry-handler = > org.apache.commons.httpclient.defaulthttpmethodretryhand...@41fc2fb > 2011/01/05 13:25:07:708 CET [DEBUG] DefaultHttpParams - Set parameter > http.dateparser.patterns = [EEE, dd MMM yyyy HH:mm:ss zzz, EEEE, > dd-MMM-yy HH:mm:ss zzz, EEE MMM d HH:mm:ss yyyy, EEE, dd-MMM-yyyy > HH:mm:ss z, EEE, dd-MMM-yyyy HH-mm-ss z, EEE, dd MMM yy HH:mm:ss z, EEE > dd-MMM-yyyy HH:mm:ss z, EEE dd MMM yyyy HH:mm:ss z, EEE dd-MMM-yyyy > HH-mm-ss z, EEE dd-MMM-yy HH:mm:ss z, EEE dd MMM yy HH:mm:ss z, > EEE,dd-MMM-yy HH:mm:ss z, EEE,dd-MMM-yyyy HH:mm:ss z, EEE, dd-MM-yyyy > HH:mm:ss z] > 2011/01/05 13:25:07:713 CET [DEBUG] DefaultHttpParams - Set parameter > http.authentication.preemptive = true > 2011/01/05 13:25:07:715 CET [DEBUG] DefaultHttpParams - Set parameter > http.auth.scheme-priority = [Digest] > 2011/01/05 13:25:07:764 CET [DEBUG] DefaultHttpParams - Set parameter > http.method.retry-handler = > org.apache.commons.httpclient.defaulthttpmethodretryhand...@450e790c > 2011/01/05 13:25:07:774 CET [DEBUG] HttpMethodDirector - Preemptively > sending default basic credentials > 2011/01/05 13:25:07:783 CET [DEBUG] HttpMethodDirector - Authenticating > with BASIC <any realm>@192.168.10.210:8090 > 2011/01/05 13:25:07:783 CET [DEBUG] HttpMethodParams - Credential > charset not configured, using HTTP element charset > 2011/01/05 13:25:07:785 CET [DEBUG] HttpConnection - Open connection to > 192.168.10.210:8090 > 2011/01/05 13:25:07:818 CET [DEBUG] header - >> "GET / HTTP/1.1[\r][\n]" > 2011/01/05 13:25:07:819 CET [DEBUG] HttpMethodBase - Adding Host request > header > 2011/01/05 13:25:07:832 CET [DEBUG] header - >> "Authorization: Basic > ****************************[\r][\n]" > 2011/01/05 13:25:07:832 CET [DEBUG] header - >> "User-Agent: Jakarta > Commons-HttpClient/3.1[\r][\n]" > 2011/01/05 13:25:07:833 CET [DEBUG] header - >> "Host: > 192.168.10.210:8090[\r][\n]" > 2011/01/05 13:25:07:833 CET [DEBUG] header - >> "[\r][\n]" > 2011/01/05 13:25:07:835 CET [DEBUG] header - << "HTTP/1.1 401 > Unauthorized[\r][\n]" > 2011/01/05 13:25:07:835 CET [DEBUG] header - << "HTTP/1.1 401 > Unauthorized[\r][\n]" > 2011/01/05 13:25:07:836 CET [DEBUG] header - << "Content-Length: > 1656[\r][\n]" > 2011/01/05 13:25:07:836 CET [DEBUG] header - << "Content-Type: > text/html[\r][\n]" > 2011/01/05 13:25:07:836 CET [DEBUG] header - << "Server: > Microsoft-IIS/6.0[\r][\n]" > 2011/01/05 13:25:07:837 CET [DEBUG] header - << "WWW-Authenticate: > Negotiate[\r][\n]" > 2011/01/05 13:25:07:837 CET [DEBUG] header - << "WWW-Authenticate: > NTLM[\r][\n]" > 2011/01/05 13:25:07:837 CET [DEBUG] header - << "X-Powered-By: > ASP.NET[\r][\n]" > 2011/01/05 13:25:07:837 CET [DEBUG] header - << "Date: Wed, 05 Jan 2011 > 12:25:07 GMT[\r][\n]" > 2011/01/05 13:25:07:837 CET [DEBUG] header - << "[\r][\n]" > 2011/01/05 13:25:07:838 CET [DEBUG] HttpMethodDirector - Authorization > required > 2011/01/05 13:25:07:839 CET [DEBUG] AuthChallengeProcessor - Supported > authentication schemes in the order of preference: [Digest] > 2011/01/05 13:25:07:839 CET [DEBUG] AuthChallengeProcessor - Challenge > for Digest authentication scheme not available > 2011/01/05 13:25:07:840 CET [WARN] HttpMethodDirector - Unable to > respond to any of these challenges: {ntlm=NTLM, negotiate=Negotiate} > Method failed: HTTP/1.1 401 Unauthorized > 2011/01/05 13:25:07:840 CET [DEBUG] HttpMethodBase - Buffering response > body > 2011/01/05 13:25:07:841 CET [DEBUG] HttpMethodBase - Resorting to > protocol version default close connection policy > 2011/01/05 13:25:07:841 CET [DEBUG] HttpMethodBase - Should NOT close > connection, using HTTP/1.1 > 2011/01/05 13:25:07:841 CET [DEBUG] HttpConnection - Releasing > connection back to connection manager. > <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" > "http://www.w3.org/TR/html4/strict.dtd";> > <HTML><HEAD><TITLE>You are not authorized to view this page</TITLE> > <META HTTP-EQUIV="Content-Type" Content="text/html; charset=Windows-1252"> > <STYLE type="text/css"> > BODY { font: 8pt/12pt verdana } > H1 { font: 13pt/15pt verdana } > H2 { font: 8pt/12pt verdana } > A:link { color: red } > A:visited { color: maroon } > </STYLE> > </HEAD><BODY><TABLE width=500 border=0 cellspacing=10><TR><TD> > > <h1>You are not authorized to view this page</h1> > You do not have permission to view this directory or page using the > credentials that you supplied because your Web browser is sending a > WWW-Authenticate header field that the Web server is not configured to > accept. > <hr> > <p>Please try the following:</p> > <ul> > <li>Contact the Web site administrator if you believe you should be able > to view this directory or page.</li> > <li>Click the <a href="javascript:location.reload()">Refresh</a> button > to try again with different credentials.</li> > </ul> > <h2>HTTP Error 401.2 - Unauthorized: Access is denied due to server > configuration.<br>Internet Information Services (IIS)</h2> > <hr> > <p>Technical Information (for support personnel)</p> > <ul> > <li>Go to <a > href="http://go.microsoft.com/fwlink/?linkid=8180";>Microsoft Product > Support Services</a> and perform a title search for the words > <b>HTTP</b> and <b>401</b>.</li> > <li>Open <b>IIS Help</b>, which is accessible in IIS Manager (inetmgr), > and search for topics titled <b>About Security</b>, > <b>Authentication</b>, and <b>About Custom Error Messages</b>.</li> > </ul> > > </TD></TR></TABLE></BODY></HTML> > > > > > > -- > > Claudio Martella > Digital Technologies > Unit Research & Development - Analyst > > TIS innovation park > Via Siemens 19 | Siemensstr. 19 > 39100 Bolzano | 39100 Bozen > Tel. +39 0471 068 123 > Fax +39 0471 068 129 > [email protected] http://www.tis.bz.it > > Short information regarding use of personal data. According to Section 13 > of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we > process your personal data in order to fulfil contractual and fiscal > obligations and also to send you information regarding our services and > events. Your personal data are processed with and without electronic means > and by respecting data subjects' rights, fundamental freedoms and dignity, > particularly with regard to confidentiality, personal identity and the right > to personal data protection. At any time and without formalities you can > write an e-mail to [email protected] in order to object the processing of > your personal data for the purpose of sending advertising materials and also > to exercise the right to access personal data and other rights referred to > in Section 7 of Decree 196/2003. The data controller is TIS Techno > Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the > complete information on the web site www.tis.bz.it. > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
