[
https://issues.apache.org/jira/browse/NUTCH-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Markus Jelsma updated NUTCH-1319:
---------------------------------
Description:
Nutch would benefit from having a host normalizer. A host normalizer maps a
given host to the desired host. A basic example is to map www.apache.org to
apache.org. The Apache website is one of many on the internet that has a
duplicate website on the same domain just because it allows both www and
non-www to return HTTP 200 and proper content.
It is also able to handle wildcards such as *.example.org to example.org if
there are multiple sub domains that actually point to the same website.
Large internet crawls tend to get polluted very quickly due to these problems.
It also leads to skewed scores in the webgraph as different websites link to
different versions of the same duplicate website. An example:
{code}
# Force all sub domains to non-www.
*.example.com example.com
# Force www sub domain to non-www.
www.example.net example.net
# Force non-www. sub domain to www
example.org www.example.org
{code}
was:
Nutch would benefit from having a host normalizer. A host normalizer maps a
given host to the desired host. A basic example is to map www.apache.org to
apache.org. The Apache website is one of many on the internet that has a
duplicate website on the same domain just because it allows both www and
non-www to return HTTP 200 and proper content.
It is also able to handle wildcards such as *.example.org to example.org if
there are multiple sub domains that actually point to the same website.
Large internet crawls tend to get polluted very quickly due to these problems.
It also leads to skewed scores in the webgraph as different websites link to
different versions of the same duplicate website. An example:
{code}
# Force all sub domains to www.
*.example.com example.com
# Force no sub domain to www. URL's
www.example.net example.net
# Force www. sub domain when hitting link without sub domain
example.org www.example.org
{code}
> HostNormalizer
> --------------
>
> Key: NUTCH-1319
> URL: https://issues.apache.org/jira/browse/NUTCH-1319
> Project: Nutch
> Issue Type: New Feature
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.6
>
> Attachments: NUTCH-1319-1.5-1.patch
>
>
> Nutch would benefit from having a host normalizer. A host normalizer maps a
> given host to the desired host. A basic example is to map www.apache.org to
> apache.org. The Apache website is one of many on the internet that has a
> duplicate website on the same domain just because it allows both www and
> non-www to return HTTP 200 and proper content.
> It is also able to handle wildcards such as *.example.org to example.org if
> there are multiple sub domains that actually point to the same website.
> Large internet crawls tend to get polluted very quickly due to these
> problems. It also leads to skewed scores in the webgraph as different
> websites link to different versions of the same duplicate website. An example:
> {code}
> # Force all sub domains to non-www.
> *.example.com example.com
> # Force www sub domain to non-www.
> www.example.net example.net
> # Force non-www. sub domain to www
> example.org www.example.org
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
