[ https://issues.apache.org/jira/browse/NUTCH-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588759#comment-15588759 ]
Sebastian Nagel commented on NUTCH-2328: ---------------------------------------- Hi [~arthur-evozon], > Btw., I'm even not sure whether counters in the task context are updated with > global job values. The answer is: counters in task context *do not show aggregated values* over all tasks of a job. That's more or less clear from these definitions: "Counter is a facility for MapReduce applications to report its statistics." [[1|https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Counter]], more details in Tom White's "Hadoop, the definitive guide". Counter values are propagated and aggregated in this direction: task > task tracker > job tracker > job client. The aggregated values are not sent back. There are hacks to access the aggregated values from the tasks, cf. [[2|http://stackoverflow.com/questions/8009802/is-there-a-way-to-access-number-of-successful-map-tasks-from-a-reduce-task-in-an/8013573#8013573]]. But the central problem persists: the value of a job counter depends mostly on how many tasks are in progress or already done. If the programming logic relies on such a value it becomes largely indeterministic. > GeneratorJob does not generate anything on second run > ----------------------------------------------------- > > Key: NUTCH-2328 > URL: https://issues.apache.org/jira/browse/NUTCH-2328 > Project: Nutch > Issue Type: Bug > Components: generator > Affects Versions: 2.2, 2.3, 2.2.1, 2.3.1 > Environment: Ubuntu 16.04 / Hadoop 2.7.1 > Reporter: Arthur B > Labels: fails, generator, subsequent > Fix For: 2.4 > > Attachments: generator-issue-static-count.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Given a topN parameter (ie 10) the GeneratorJob will fail to generate > anything new on the subsequent runs within the same process space. > To reproduce the issue submit the GeneratorJob twice one after another to the > M/R framework. Second time will say it generated 0 URLs. > This issue is due to the usage of the static count field > (org.apache.nutch.crawl.GeneratorReducer#count) to determine if the topN > value has been reached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)