[
https://issues.apache.org/jira/browse/NUTCH-827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464589#comment-13464589
]
Jasper van Veghel commented on NUTCH-827:
-----------------------------------------
Hey guys,
This has been some time back, but take a look at the patch:
nutch-default.xml ..
<name>http.cookie.login.page</name>
<description>URL of the login page to derive the cookies from. Cookies will be
stored upon initialization and re-initialized upon expiration. Any URL request
attributes will be [..] POSTed to the page. [..]</description>
Apologies for the poor grammar in the original. ;-) Basically:
- Whenever protocol-httpclient performs an HTTP request, it will first check if
there are cookies stored in the cookie jar.
- If there are cookies in the cookie jar AND none of the cookies have expired,
it will do nothing.
- If there are no cookies in the cookie jar OR at least one of the cookies has
expired, it will ..
- POST the URL / parameters provided in "http.cookie.login.page" property
- In the process of which, the cookie jar should get filled with the cookies
you need to perform subsequent (authenticated) requests
The "http.cookie.login.page" property could contain something like
"http://abc/def?username=foo&password=bar"; .. the 'username' and 'password'
properties will them be POSTed to 'http://abc/def', which should result in
cookies being added to the cookie jar which is used for each subsequent request.
This isn't exactly a fool-proof solution (what if other requests generate
expired cookies? what if the login fails? etc.), but for the project for which
I wrote the patch, it suited our needs. Hope it helps!
> HTTP POST Authentication
> ------------------------
>
> Key: NUTCH-827
> URL: https://issues.apache.org/jira/browse/NUTCH-827
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Affects Versions: 1.1, nutchgora
> Reporter: Jasper van Veghel
> Priority: Minor
> Labels: authentication
> Fix For: 1.6
>
> Attachments: nutch-http-cookies.patch
>
>
> I've created a patch against the trunk which adds support for very
> rudimentary POST-based authentication support. It takes a link from
> nutch-site.xml with a site to POST to and its respective parameters
> (username, password, etc.). It then checks upon every request whether any
> cookies have been initialized, and if none have, it fetches them from the
> given link.
> This isn't perfect but Works For Me (TM) as I generally only need to retrieve
> results from a single domain and so have no cookie overlap (i.e. if the
> domain cookies expire, all cookies disappear from the HttpClient and I can
> simply re-fetch them). A natural improvement would be to be able to specify
> one particular cookie to check the expiration-date against. If anyone is
> interested in this beside me I'd be glad to put some more effort into making
> this more universally applicable.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
