Enzo Michelangeli wrote:
Is there a way of parallelizing URLFiltering over multiple threads?
After all, the URLFilters themselves must already be thread-safe, or
else they would have problems during fetching.
The reason why I'm asking is I have a custom URLFilter that needs to
make calls to the DNS resolver, and multi-threading the URLFiltering
would greatly speed up some filtering procedures that, unlike fetching,
appear to be single-threaded: "mergedb -filter", inject, generate,
"updatedb -filter" etc. (The most important is of course "generate" or,
even better, "updatedb -filter" to prevent undesired URL's to reach the
crawldb in first place).
Are you running jobs in the "local" mode? In distributed mode filtering
is naturally parallel, because you have as many concurrent lookups as
there are map tasks.
In my experience, using multiple threads for DNS lookup doesn't help
that much. What helps A LOT (like several orders of magnitude) is using
a local DNS cache, or even two-level DNS cache (one cache per node, one
cache per cluster).
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
