Lewis John McGibbney created NUTCH-3242:
-------------------------------------------

             Summary: Add status counters to the five core jobs that have none
                 Key: NUTCH-3242
                 URL: https://issues.apache.org/jira/browse/NUTCH-3242
             Project: Nutch
          Issue Type: Sub-task
            Reporter: Lewis John McGibbney


Five core jobs call \{{getCounter}} nowhere:

* \{{org.apache.nutch.crawl.LinkDb}}
* \{{org.apache.nutch.segment.SegmentMerger}}
* \{{org.apache.nutch.crawl.CrawlDbReader}}
* \{{org.apache.nutch.hostdb.ReadHostDb}}
* \{{org.apache.nutch.crawl.CrawlDbMerger}}

[NUTCH-3155|https://issues.apache.org/jira/browse/NUTCH-3155] adds 
\{{ErrorTracker}} to \{{LinkDb.LinkDbMapper}}, \{{LinkDbFilter}}, 
\{{CrawlDbReader.CrawlDbStatMapper}}, \{{SegmentMerger.SegmentMergerMapper}}, 
\{{ReadHostDb.ReadHostDbMapper}}, and \{{UpdateHostDbReducer}}. It does not add 
status counters. \{{UpdateHostDbReducer}} already has counters 
(\{{total_hosts_total}}, \{{skipped_not_eligible_total}}, 
\{{url_limit_not_reached_total}}) and is not part of this issue. 
\{{SegmentMerger}} already catches per record, at the two \{{catch 
(Exception)}} blocks in its mapper. \{{CrawlDbMerger}}'s reducer writes the 
merged datum and does not catch per record; its only catches are around job 
submission.

Other tools also have no counters (LinkDbMerger, LinkDbReader, FreeGenerator, 
the WebGraph tools, SegmentReader, and several dumpers). Those are a separate 
issue. This issue is the five classes above.

h2. Proposal

Add these counters, declared in \{{NutchMetrics}}, under a new group per job 
(\{{nutch_linkdb}}, \{{nutch_segment_merger}}, \{{nutch_crawldb_reader}}, 
\{{nutch_hostdb_reader}}, \{{nutch_crawldb_merger}}):

* \{{records_processed_total}}
* \{{records_skipped_total}}
* \{{records_failed_total}}

Cache the references in \{{setup}}, matching Generator and Injector.

Leave \{{ErrorTracker}} for the classes named in NUTCH-3155 to that issue. Do 
not add \{{ErrorTracker}} to \{{CrawlDbMerger}}. Its reducer does not catch per 
record.

h2. Acceptance

* Each of the five jobs emits the three status counters from \{{NutchMetrics}} 
constants, with the references cached.
* A run over a small fixture increments \{{records_processed_total}} by the 
number of records handled, and increments \{{records_skipped_total}} or 
\{{records_failed_total}} only for records that were actually skipped or failed.
* \{{UpdateHostDbReducer}} is unchanged.
* No class named in NUTCH-3155 gains a second \{{ErrorTracker}} from this issue.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to