[
https://issues.apache.org/jira/browse/NUTCH-2368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16099986#comment-16099986
]
Sebastian Nagel commented on NUTCH-2368:
----------------------------------------
+1 A powerful feature! A few remarks:
- the instance variable {{maxCount}} is overwritten in the reduce method for a
given host: this leads to unpredictable behavior if some hosts are missing in
the hostdb. It's probably safer to keep {{maxCount}} untouched and use a local
variable which holds the per-host count or to the value of
{{maxCount}}/{{generate.max.count}} as fall-back.
- ignored catch blogs: should log something, esp. if errors are severe such as
a failure reading the hostdb
- git complained about trailing white space while applying the patch
- avoid string concatenations / use {{{}}} placeholders when calling
LOG.xxx(...)
> Variable generate.max.count and fetcher.server.delay
> ----------------------------------------------------
>
> Key: NUTCH-2368
> URL: https://issues.apache.org/jira/browse/NUTCH-2368
> Project: Nutch
> Issue Type: Improvement
> Components: generator
> Affects Versions: 1.12
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.14
>
> Attachments: NUTCH-2368.patch, NUTCH-2368.patch, NUTCH-2368.patch,
> NUTCH-2368.patch, NUTCH-2368.patch, NUTCH-2368.patch
>
>
> In some cases we need to use host specific characteristics in determining
> crawl speed and bulk sizes because with our (Openindex) settings we can just
> recrawl host with up to 800k urls.
> This patch solves the problem by introducing the HostDB to the Generator and
> providing powerful Jexl expressions. Check these two expressions added to the
> Generator:
> {code}
> -Dgenerate.max.count.expr='
> if (unfetched + fetched > 800000) {
> return (conf.getInt("fetcher.timelimit.mins", 12) * 60) / ((pct95._rs_ +
> 500) / 1000) * conf.getInt("fetcher.threads.per.queue", 1)
> } else {
> return conf.getDouble("generate.max.count", 300);
> }'
> -Dgenerate.fetch.delay.expr='
> if (unfetched + fetched > 800000) {
> return (pct95._rs_ + 500);
> } else {
> return conf.getDouble("fetcher.server.delay", 1000)
> }'
> {code}
> For each large host: select as many records as possible that are possible to
> fetch based on number of threads, 95th percentile response time of the fetch
> limit. Or: queueMaxCount = (timelimit / resonsetime) * numThreads.
> The second expression just follows up to that, settings the crawlDelay of the
> fetch queue.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
