Lewis John McGibbney created NUTCH-3248:
-------------------------------------------
Summary: Record categorized errors where record-level failures are
only logged
Key: NUTCH-3248
URL: https://issues.apache.org/jira/browse/NUTCH-3248
Project: Nutch
Issue Type: Sub-task
Reporter: Lewis John McGibbney
[NUTCH-3155|https://issues.apache.org/jira/browse/NUTCH-3155] adds
\{{ErrorTracker}} to \{{LinkDbMapper}}, \{{LinkDbFilter}},
\{{CrawlDbReader.CrawlDbStatMapper}}, \{{SegmentMergerMapper}},
\{{ReadHostDbMapper}}, and \{{UpdateHostDbReducer}}. Three further classes
catch a per-record failure, log it or drop it, and emit no error counter. They
already have other counters, so this is not the "no counters at all" issue.
* \{{WebGraph}} \{{normalizeUrl}} catches \{{Exception}}, logs \{{Skipping}},
and returns null. \{{filterUrl}} catches \{{Exception}} and returns null.
Neither increments a counter. Group for the new error counters:
\{{nutch_webgraph}}.
* \{{DomainStatistics}} mapper catches \{{MalformedURLException}}, logs, and
returns. A surrounding \{{catch (Exception)}} is empty, and
\{{fetchedCounter.increment(1)}} sits outside that try, so a failure still
counts the URL as fetched. Group: \{{nutch_domain_stats}}.
* \{{QueueFeeder.filterNormalize}} catches \{{MalformedURLException}} or
\{{URLFilterException}}, logs \{{Skipping}}, and drops the URL. Group:
\{{nutch_fetcher}}.
{\{ResolverThread}} is not part of this issue. It already increments
\{{errors_total}} and \{{errors_network_total}}, by hand rather than through an
\{{ErrorTracker}} instance. \{{DeduplicationJob}}'s \{{URLDecoder}} catch logs
and keeps the encoded URL, so the record is still processed. \{{CleaningJob}}'s
open failure is rethrown. Leave those three alone.
h2. Proposal
Construct an \{{ErrorTracker}} with the cached-counter constructor, matching
Generator, and call \{{incrementCounters}} from the catches above.
In \{{DomainStatistics}}, increment \{{fetched_total}} only when the record was
actually written. A malformed URL or an exception from the outer catch
increments the error counter and does not increment \{{fetched_total}}.
h2. Acceptance
* A URL that fails normalization or filtering in \{{WebGraph}} increments
\{{nutch_webgraph}} / \{{errors_total}} and exactly one category counter.
* A malformed URL in \{{DomainStatistics}} increments \{{errors_total}} and
does not increment \{{fetched_total}}.
* A URL \{{QueueFeeder}} skips because of a filter or normalizer exception
increments \{{nutch_fetcher}} / \{{errors_total}}.
* The classes named in NUTCH-3155 are unchanged.
* \{{ResolverThread}} is unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
