Hi,
after playing around to figure out the best place to resolve IP's of
freshly discovered ulrs I agree with Andrzej that the
Parseoutputformat isn't the best place.
The problem here, Parseoutputformat is not multithreaded and we
definitely need many threads for ip lookup.
I think in case we a ip Resolving MapRunnable to preprocess segment
data (after fetching) before crawldb updating would be may be a
better place.
+ less data to process (in opposite to process a complete crawldb)
+ good dns cache usage, since many new urls will point to the same
host (internal links)
- we may lookup urls we already have in the crawldb.
Any thoughts?
Stefan
