Dennis Kubes wrote:
I have been working on improving the Generator for the last couple of
days and here are the discussion areas I have come up with so far:
1) Would resolving IP addresses inside of the generator be useful? If we
are limiting the number of urls to fetch then this would allow us to
remove UnknownHosts before hand, essentially giving us a better fetch
list. Cons are it could as much as double the DNS load as it is
happening once during generate and once during fetching. As it is I am
working on a patch that give the option to either resolve it or not.
We had a discussion on this in the past. IIRC here are the issues with
early IP resolution:
* hostname->ip mapping may change rapidly (round-robin DNS),
* resolving IPs in Generator practicaly enforces that even small
installations must use local caching DNS servers - otherwise the
cumulative DNS traffic created by Generator may be too high.
* fetcher would need a way to re-use this resolved IP so that we don't
do the same lookup twice, i.e. we would have to implement a DNS provider
that can use the resolved (and presumably saved) IPs during fetching.
All in all, it may be worth it, or perhaps it might not. ;)
2) Normalization of urls inside of generate. Currently in the reduce
method of Selector inside of generate there is a normalization call.
Personally I think this is in the wrong place. I think this should be in
selector map method. As it is currently, the normalizer doesn't have
any effect anyways because we are not collecting a changed url (a bug).
If we were to put the normalizer in the map method, then the possibility
for duplicate urls from normalization arises as well. If I am not
mistaken this would need another MR job at the end of generator to
remove duplicates.
Another options would be just to NOT have the normalization option
inside of Generator. Is there a good reason to have normalization in
Generator?
Well, I think the idea here was that the normalization rules could be
very dynamic, i.e. they could change dramatically between subsequent
runs of Generator - although I must say that I don't see this happening
in practice ...
Perhaps Nutch should work on the following assumption: urls that are
found in Crawldb are guaranteed to be normalized. If different
normalization rules are needed then the crawldb needs to be explicitly
filtered in a separate step, using CrawlDbMerger tool.
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
