Hi Markus, Right. I agree that a simple map implementation would help here.
Mathijs On Mar 21, 2012, at 22:54 , Markus Jelsma wrote: > Hi Mathijs, > > We use this in the fetcher (parse=true) and when updating the CrawlDB and > with the free generator. We use it in the fetcher because we follow outlinks > and make sure we follow the desired host and in the CrawlDB because there we > update records for recently added host normalizer rules. > > It is just an URL normalizer like the others but only changes the host part. > This is not covered in other standard normalizers. The BasicURLNormalizer > cannot do this and the RegexURLNormalizer is far too heavy to take 20MB of > expressions and harder to auto-generate. A simple map lookup is very fast. > > Cheers, > > On Wed, 21 Mar 2012 22:22:54 +0100, Mathijs Homminga > <[email protected]> wrote: >> Hi Markus, >> >> How (where in the process) do you like to use this normalizer. Isn't >> this functionality already covered by the URL normalizer(s)? >> >> Mathijs Homminga >> >> On Mar 21, 2012, at 22:06, "Markus Jelsma (Created) (JIRA)" >> <[email protected]> wrote: >> >>> HostNormalizer >>> -------------- >>> >>> Key: NUTCH-1319 >>> URL: https://issues.apache.org/jira/browse/NUTCH-1319 >>> Project: Nutch >>> Issue Type: New Feature >>> Reporter: Markus Jelsma >>> Assignee: Markus Jelsma >>> Fix For: 1.5 >>> >>> >>> Nutch would benefit from having a host normalizer. A host normalizer maps a >>> given host to the desired host. A basic example is to map www.apache.org to >>> apache.org. The Apache website is one of many on the internet that has a >>> duplicate website on the same domain just because it allows both www and >>> non-www to return HTTP 200 and proper content. >>> >>> It is also able to handle wildcards such as *.example.org to example.org if >>> there are multiple sub domains that actually point to the same website. >>> >>> Large internet crawls tend to get polluted very quickly due to these >>> problems. It also leads to skewed scores in the webgraph as different >>> websites link to different versions of the same duplicate website. >>> >>> -- >>> This message is automatically generated by JIRA. >>> If you think it was sent incorrectly, please contact your JIRA >>> administrators: >>> https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa >>> For more information on JIRA, see: http://www.atlassian.com/software/jira >>> >>> >
