[ https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13401275#comment-13401275 ]
Hudson commented on NUTCH-1319: ------------------------------- Integrated in nutch-trunk-maven #331 (See [https://builds.apache.org/job/nutch-trunk-maven/331/]) NUTCH-1319 adding test to accomodate HostURLNormalizer (Revision 1353884) Result = SUCCESS markus : Files : * /nutch/trunk/src/test/org/apache/nutch/net/TestURLNormalizers.java > HostNormalizer > -------------- > > Key: NUTCH-1319 > URL: https://issues.apache.org/jira/browse/NUTCH-1319 > Project: Nutch > Issue Type: New Feature > Reporter: Markus Jelsma > Assignee: Markus Jelsma > Fix For: 1.6 > > Attachments: NUTCH-1319-1.5-1.patch > > > Nutch would benefit from having a host normalizer. A host normalizer maps a > given host to the desired host. A basic example is to map www.apache.org to > apache.org. The Apache website is one of many on the internet that has a > duplicate website on the same domain just because it allows both www and > non-www to return HTTP 200 and proper content. > It is also able to handle wildcards such as *.example.org to example.org if > there are multiple sub domains that actually point to the same website. > Large internet crawls tend to get polluted very quickly due to these > problems. It also leads to skewed scores in the webgraph as different > websites link to different versions of the same duplicate website. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira