[
https://issues.apache.org/jira/browse/NUTCH-2455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16247765#comment-16247765
]
Sebastian Nagel commented on NUTCH-2455:
----------------------------------------
Hi [[email protected]], thanks for the patch. The solution to cache the
entire HostDb in a HashMap is, however, not scalable:
* consider that the HostDb may contain 100 millions of hosts - how much RAM
would be required to hold the HashMap in memory?
* when running on a cluster even every parallel task would have to load the
HashMap (and hold it in memory): while the CrawlDb is read only once, the
HostDb is read many times and every read takes long because it is not
parallelized.
A proper solution could be:
- read HostDatums together with CrawlDatums (cf. MultipleInputFormat, depends
on NUTCH-2375) as input of the select step
- the map function then emits key-value pairs <host, score> ->
<url,crawldatum,score,...>
-* of course, the HostDatums must be wrapped into the value structure. It's
already a custom class (SelectorEntry), so that should be doable
- via partitioning and secondary sorting these arrive in the reduce function:
-* all keys with the same host in one call of the function
-* in the following order: first the HostDatum (just assign an artificially
high score), then the CrawlDatum items sorted by decreasing score
Have a look at this modified [Generator
variant|https://github.com/commoncrawl/nutch/blob/cc/src/java/org/apache/nutch/crawl/Generator2.java#L395],
for how secondary sorting can be implemented.
> Speed up the merging of HostDb entries for variable fetch delay
> ---------------------------------------------------------------
>
> Key: NUTCH-2455
> URL: https://issues.apache.org/jira/browse/NUTCH-2455
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Affects Versions: 1.13
> Reporter: Markus Jelsma
> Attachments: NUTCH-2455.patch
>
>
> Citing Sebastian at NUTCH-2420:
> ??The correct solution would be to use <host,score> pairs as keys in the
> Selector job, with a partitioner and secondary sorting so that all keys with
> same host end up in the same call of the reducer. If values can also hold a
> HostDb entry and the sort comparator guarantees that the HostDb entry
> (entries if partitioned by domain or IP) comes in front of all CrawlDb
> entries. But that would be a substantial improvement...??
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
