Andrzej Bialecki wrote:
Dennis Kubes wrote:
I have been working on improving the Generator for the last couple of
days and here are the discussion areas I have come up with so far:
1) Would resolving IP addresses inside of the generator be useful? If
we are limiting the number of urls to fetch then this would allow us
to remove UnknownHosts before hand, essentially giving us a better
fetch list. Cons are it could as much as double the DNS load as it is
happening once during generate and once during fetching. As it is I
am working on a patch that give the option to either resolve it or not.
We had a discussion on this in the past. IIRC here are the issues with
early IP resolution:
* hostname->ip mapping may change rapidly (round-robin DNS),
True but in this instance the goal was just to avoid pulling urls with
unknown DNS into the fetchlist so we can get the maximum amount of good
urls to fetch. We are trading time for quality.
* resolving IPs in Generator practicaly enforces that even small
installations must use local caching DNS servers - otherwise the
cumulative DNS traffic created by Generator may be too high.
If this is an option that is set to false by default then it should
affect current behavior and it should give those of us with proper
resources the ability to filter out if desired. I guess on one hand I
am thinking that having the option is better than not having it, maybe
not. Do you think it would be bad to have the option?
* fetcher would need a way to re-use this resolved IP so that we don't
do the same lookup twice, i.e. we would have to implement a DNS provider
that can use the resolved (and presumably saved) IPs during fetching.
We could start with an option to have it there (not saved), then move it
along to a more developed solution later :)
All in all, it may be worth it, or perhaps it might not. ;)
2) Normalization of urls inside of generate. Currently in the reduce
method of Selector inside of generate there is a normalization call.
Personally I think this is in the wrong place. I think this should be
in selector map method. As it is currently, the normalizer doesn't
have any effect anyways because we are not collecting a changed url (a
bug).
If we were to put the normalizer in the map method, then the
possibility for duplicate urls from normalization arises as well. If
I am not mistaken this would need another MR job at the end of
generator to remove duplicates.
Another options would be just to NOT have the normalization option
inside of Generator. Is there a good reason to have normalization in
Generator?
Well, I think the idea here was that the normalization rules could be
very dynamic, i.e. they could change dramatically between subsequent
runs of Generator - although I must say that I don't see this happening
in practice ...
Currently there is a normalization in the selector reduce. But that
normalized url never updates the entry.url, which is a bug, so is it
better to just remove the normalization or leave it. Here is the kicker
though, if we leave it, it is possible that duplicate urls will make it
through to the fetchlist.
Perhaps Nutch should work on the following assumption: urls that are
found in Crawldb are guaranteed to be normalized. If different
normalization rules are needed then the crawldb needs to be explicitly
filtered in a separate step, using CrawlDbMerger tool.
We have been talking about this lately. It seems like there needs to be
two more tools or extension points.
One is a url translator. Something that would say url A is actually url
B. Anytime that url is seen it would be translated to its new form.
This would help with cases such as java.net vs www.java.net being two
separate urls. Jobs could be written to translate based on hash or
other values.
The second tool would be a tool that more completely manipulates the
crawldb (maybe this is the crawldbmerger). This would allow things like
resetting the crawl dates on all urls for the crawldb, normalizing
scores, things like that that are global operations on the entire
crawldb or a subset of the urls in the crawldb.
Dennis
