[
https://issues.apache.org/jira/browse/NUTCH-827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13918294#comment-13918294
]
yuanyun.cn commented on NUTCH-827:
----------------------------------
I was assigned a task to use nutch2 to crawla web site which uses form-based
authentication.
Based on Jasper's code, I made some improvement to make it work. Please view
the patch: http-client-form-authtication.patch.
To use it, first we try to figure it out how to use http client to do form
based login successfully, We can use Chrome Devtools to get the login formId,
username and password fields, get the exact post request; we may remove some
form fields, or add some headers.
{code:borderStyle=solid}
private static void authTestAspWebApp() throws Exception, IOException {
HttpFormAuthConfigurer authConfigurer = new HttpFormAuthConfigurer();
authConfigurer.setLoginUrl("http://localhost:44444/Account/Login.aspx";)
.setLoginFormId("ctl01").setLoginRedirect(true);
Map<String, String> loginPostData = new HashMap<String, String>();
loginPostData.put("ctl00$MainContent$LoginUser$UserName", "admin");
loginPostData.put("ctl00$MainContent$LoginUser$Password", "admin123");
authConfigurer.setLoginPostData(loginPostData);
Set<String> removedFormFields = new HashSet<String>();
removedFormFields.add("ctl00$MainContent$LoginUser$RememberMe");
authConfigurer.setRemovedFormFields(removedFormFields);
HttpFormAuthentication example = new HttpFormAuthentication(
authConfigurer);
example.login();
String result = example
.httpGetPageContent("http://localhost:44444/secret/needlogin.aspx";);
System.out.println(result);
}
{code}
After make the test code work, we define form authentication info in
httpclient-auth.xml:
{code:xml}
<?xml version="1.0"?>
<auth-configuration>
<credentials authMethod="formAuth"
loginUrl="http://localhost:44444/Account/Login.aspx"; loginFormId="ctl01"
loginRedirect="true">
<loginPostData>
<field name="ctl00$MainContent$LoginUser$UserName" value="admin"/>
<field name="ctl00$MainContent$LoginUser$Password" value="admin123"/>
</loginPostData>
<removedFormFields>
<field name="ctl00$MainContent$LoginUser$RememberMe"/>
</removedFormFields>
</credentials>
</auth-configuration>
{code}
Be sure to use protocol-httpclient plugin in nutch-site.xml: not protocol-http.
If you are interested, you may
read:http://lifelongprogrammer.blogspot.com/2014/02/part1-using-apache-http-client-to-do-http-post-form-authentication.html
> HTTP POST Authentication
> ------------------------
>
> Key: NUTCH-827
> URL: https://issues.apache.org/jira/browse/NUTCH-827
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 1.1, nutchgora
> Reporter: Jasper van Veghel
> Priority: Minor
> Labels: authentication
> Fix For: 1.9
>
> Attachments: http-client-form-authtication.patch,
> nutch-http-cookies.patch
>
>
> I've created a patch against the trunk which adds support for very
> rudimentary POST-based authentication support. It takes a link from
> nutch-site.xml with a site to POST to and its respective parameters
> (username, password, etc.). It then checks upon every request whether any
> cookies have been initialized, and if none have, it fetches them from the
> given link.
> This isn't perfect but Works For Me (TM) as I generally only need to retrieve
> results from a single domain and so have no cookie overlap (i.e. if the
> domain cookies expire, all cookies disappear from the HttpClient and I can
> simply re-fetch them). A natural improvement would be to be able to specify
> one particular cookie to check the expiration-date against. If anyone is
> interested in this beside me I'd be glad to put some more effort into making
> this more universally applicable.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
