On Mon, Jul 6, 2009 at 3:10 PM, youyou wu<wuyouyou...@hotmail.com> wrote: > hi Susam, > > I have read your work about HttpAuthenticationSchemes,and configed all the > file according to what you said , but no authentication occured . > I can find the word "Credentials" but no "auth.AuthChallengeProcessor " in > my logs. I have also tried it in different websites requiring > Authentication, no one succeeded. > Is there anything wrong with my Configuration that lead to the failure? > The sixth of "before asking for help" points out "probably something needs > to be fixed at the server side", would you mind telling me the details? > > The version of nutch running in my machine is 1.0. > My 'conf/httpclient-auth.xml' is like this: > <auth-configuration> > <credentials username="wendysky98" password="666666"> > <default/> > </credentials> > </auth-configuration> > > in the attachment is the complete hadoop.log > > > looking forward to your answer. > Thanks a lot! > > Yours > > Wuyu > > ________________________________ > What can you do with the new Windows Live? Find out
If you can not see "auth.AuthChallengeProcessor " lines in the log files, probably the website does not require authentication. What kind of authentication were you expecting? You can also verify whether the server really requires authentication using telnet, netcat (nc) or a sniffer. With telnet you would type this and press the enter key twice: telnet example.server.com 80 GET / HTTP/1.0 Host: example.server.com If the server requires authentication, you would see a "WWW-Authenticate" in the response. Please delete the current logs, enable DEBUG logs for log4j.logger.org.apache.nutch.protocol.httpclient, start a new crawl with only one site URL that you are unable to authenticate to and then send the following files: 1. conf/nutch-site.xml 2. conf/httpclient-auth.xml 3. logs/hadoop.log 4. Output from telnet, netcat, etc. Please go through "Need Help?" section of http://wiki.apache.org/nutch/HttpAuthenticationSchemes to make sure you haven't missed anything important. Regards, Susam Pal
