Sebastian Nagel commented on NUTCH-2328:

Thanks, [~arthur-evozon]. Good catch!

In which environment was the problem observed? E.g.,
- running Nutch via bin/crawl or via Nutch server ?
- in local / pseudo-distributed / fully distributed Hadoop environment ?

The variable {{count}} should not be static, that's definitely a problem when 
running multiple Generator jobs from a long-running Nutch server in local mode 
where all tasks are run in the same JVM.

The variable {{limit}} is a per-task limit (see how it's initialized in 
{{setup(context)}}), comparing it with a global counter seems wrong, also 
retrieving the counter in every call of the reduce function may be too 
expensive. Why not make {{count}} an instance variable?

> GeneratorJob does not generate anything on second run
> -----------------------------------------------------
>                 Key: NUTCH-2328
>                 URL: https://issues.apache.org/jira/browse/NUTCH-2328
>             Project: Nutch
>          Issue Type: Bug
>          Components: generator
>    Affects Versions: 2.2, 2.3, 2.2.1, 2.3.1
>         Environment: Ubuntu 16.04 / Hadoop 2.7.1
>            Reporter: Arthur B
>              Labels: fails, generator, subsequent
>             Fix For: 2.4
>         Attachments: generator-issue-static-count.patch
>   Original Estimate: 24h
>  Remaining Estimate: 24h
> Given a topN parameter (ie 10) the GeneratorJob will fail to generate 
> anything new on the subsequent runs within the same process space.
> To reproduce the issue submit the GeneratorJob twice one after another to the 
> M/R framework. Second time will say it generated 0 URLs.
> This issue is due to the usage of the static count field 
> (org.apache.nutch.crawl.GeneratorReducer#count) to determine if the topN 
> value has been reached.

This message was sent by Atlassian JIRA

Reply via email to