[
https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295804#comment-13295804
]
Hudson commented on NUTCH-1319:
-------------------------------
Integrated in Nutch-trunk #1869 (See
[https://builds.apache.org/job/Nutch-trunk/1869/])
NUTCH-1319 HostNormalizer plugin (Revision 1349236)
Result = SUCCESS
markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349236
Files :
* /nutch/trunk/CHANGES.txt
* /nutch/trunk/conf/host-urlnormalizer.txt
* /nutch/trunk/src/plugin/urlnormalizer-host
* /nutch/trunk/src/plugin/urlnormalizer-host/build.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/data
* /nutch/trunk/src/plugin/urlnormalizer-host/data/hosts.txt
* /nutch/trunk/src/plugin/urlnormalizer-host/ivy.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/plugin.xml
* /nutch/trunk/src/plugin/urlnormalizer-host/src
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch
* /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net
*
/nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer
*
/nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host
*
/nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host/HostURLNormalizer.java
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch
* /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net
*
/nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer
*
/nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host
*
/nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host/TestHostURLNormalizer.java
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.6
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a
> given host to the desired host. A basic example is to map www.apache.org to
> apache.org. The Apache website is one of many on the internet that has a
> duplicate website on the same domain just because it allows both www and
> non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if
> there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these
> problems. It also leads to skewed scores in the webgraph as different
> websites link to different versions of the same duplicate website.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
