Dear Wiki user, You have subscribed to a wiki page or wiki category on "Nutch Wiki" for change notification.
The following page has been changed by susam: http://wiki.apache.org/nutch/protocol-http11 The comment on the change is: rough draft New page: == Introduction == 'protocol-http11' is a protocol plugin which supports retrieving documents via the HTTP 1.0, HTTP 1.1 and HTTPS protocols, optionally with Basic, Digest and NTLM authentication schemes for web server as well as proxy server. == Author == Susam Pal, Infosys Technologies Limited == Necessity == There were two plugins already present, viz. 'protocol-http' and 'protocol-httpclient'. However, 'protocol-http' could not support HTTP 1.1, HTTPS and NTLM, Basic and Digest authentication schemes. 'protocol-httpclient' supported HTTPS and had code for NTLM authentication but the NTLM authentication didn't work due to a bug. 'protocol-http11' was written to solve these problems, provide additional features like authentication support for proxy server and better inline documentation for the properties to be used in 'nutch-site.xml' to enable 'protocol-http11' and use its authentication features. This is an improvement on the previous two plugins. The author of this plugin has tested it in Infosys Technologies Limited by crawling the corporate intranet requiring NTLM authentication and this has been found to work well. == Quick Guide == This section is a quick guide to configure authentication related properties for 'protocol-http11'. 1. Include 'protocol-http11' in 'plugin.includes'. 1. For basic or digest authentication in proxy server, set 'http.proxy.username' and 'http.proxy.password'. Also, set 'http.proxy.realm' if you want to specify a realm as the authentication scope. 1. For NTLM authentication in proxy server, set 'http.proxy.username', 'http.proxy.password', 'http.proxy.realm' and 'http.auth.host'. 'http.proxy.realm' is the NTLM domain name. 'http.auth.host' is the host where the crawler is running. 1. For basic or digest authentication in web servers, set 'http.auth.username' and 'http.auth.password'. Also, set 'http.auth.realm' if you want to specify a realm as the authentication scope. 1. For NTLM authentication in proxy server, set 'http.auth.username', 'http.auth.password', 'http.auth.realm' and 'http.auth.host'. 'http.auth.realm' is the NTLM domain name. 'http.auth.host' is the host where the crawler is running. 1. It is recommended that 'http.useHttp11' be set to true. This is explained in a little more detail in the next section. == Nutch Configuration == To use 'protocol-http11', 'conf/nutch-site.xml has to be edited to include some properties which is explained in this section. First and foremost, to enable the plugin, this plugin must be added in the 'plugin.includes' of 'nutch-site.xml'. So, this property would typically look like:- {{{<property> <name>plugin.includes</name> <value>protocol-http11|urlfilter-regex|parse-(text|html|js|pdf|mp3|oo|msexcel|mspowerpoint|msword|pdf|rss|swf|zip)|index-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|regex|basic)</value> <description>...</description> </property>}}} It is recommended that HTTP 1.1 should be enabled. {{{<property> <name>http.useHttp11</name> <value>false</value> <description>...</description> </property>}}} Next, if authentication is required for proxy server, the following properties need to be set in 'conf/nutch-site.xml'. * http.proxy.username * http.proxy.password * http.proxy.realm (If a realm needs to be provided. In case of NTLM authentication, the domain name should be provided as its value.) * http.auth.host (This is required in case of NTLM authentication only. This is the host where the crawler would be running.) If the web servers of the intranet are in a particular domain or realm and requires authentication, these properties should be set in 'conf/nutch-site.xml'. * http.auth.username * http.auth.password * http.auth.realm * http.auth.host The explanation for these properties are similar to that of the proxy authentication properties. As you might have noticed, 'http.auth.host' is used both for proxy NTLM authentication and web server NTLM authentication. Since, the host at which the HTTP requests are originating are same for both, so the same property is used for both and two different properties were not created. Even though, the 'http.auth.host' property is required only for NTLM authentication, it is advisable to set this for all cases, because, in case the crawler comes across a server which requires NTLM authentication (which you might not have anticipated), the crawler can still fetch the page. == Underlying HttpClient Library == 'protocol-http11' is based on [http://jakarta.apache.org/httpcomponents/httpclient-3.x/ Jakarta Commons HttpClient]. Some servers support multiple schemes for authenticating users. Given that only one scheme may be used at a time for authenticating, HttpClient must choose which scheme to use. To accompish this, HttpClient uses an order of preference to select the correct authentication scheme. By default this order is: NTLM, Digest, Basic. For more information on the behavior during authentication, you might want to read the [http://jakarta.apache.org/httpcomponents/httpclient-3.x/authentication.html HttpClient Authentication Guide]. == Need Help? == If you need help, please feel free to post your question to the [http://lucene.apache.org/nutch/mailing_lists.html#Users nutch-user mailing list].
