Lewis John McGibbney created NUTCH-3240:
-------------------------------------------
Summary: Add retry cause and exhaustion counters
Key: NUTCH-3240
URL: https://issues.apache.org/jira/browse/NUTCH-3240
Project: Nutch
Issue Type: Sub-task
Reporter: Lewis John McGibbney
The metrics report lists retry counts and patterns as a missing signal. Part of
that signal already exists, and the rest does not.
{\{FetcherThread}} increments \{{context.getCounter(NutchMetrics.GROUP_FETCHER,
status.getName())}} once per fetch outcome. \{{ProtocolStatus.getName()}}
returns \{{retry}} for \{{ProtocolStatus.RETRY}} and \{{exception}} for
\{{ProtocolStatus.EXCEPTION}}. So a \{{nutch_fetcher}} / \{{retry}} counter
already counts one of the retry paths. It is a dynamic name, not a
\{{NutchMetrics}} constant, and it does not say why the retry happened or when
the crawl gave up.
The paths that write \{{CrawlDatum.STATUS_FETCH_RETRY}} are:
* \{{ProtocolStatus.RETRY}}. No call to \{{ErrorTracker}}. The dynamic counter
name is \{{retry}}.
* \{{ProtocolStatus.EXCEPTION}}, which falls through into the same output.
\{{logError(String)}} records \{{ErrorTracker.ErrorType.OTHER}} always, not the
exception class. HTTP responses that are not 300-307, 400, 401, 404, or 410
come from \{{HttpBase}} as \{{ProtocolStatus.EXCEPTION}} with the message
\{{Http code=NNN}}. That includes 403, 429, and every 5xx. The dynamic counter
name is \{{exception}}.
* the \{{default}} branch, for an unknown protocol status.
* the \{{catch (Throwable)}} around the fetch. \{{logError(Throwable)}} does
categorize through \{{ErrorTracker}} (timeout, network, or other). The datum is
still \{{STATUS_FETCH_RETRY}}. \{{ProtocolStatus.STATUS_FAILED}} is what gets
passed to \{{output}}, and the dynamic status counter is only incremented if
execution already passed the \{{status.getName()}} line.
{\{CrawlDbReducer}} applies the retry limit.
\{{AbstractFetchSchedule.setPageRetrySchedule}} increments
\{{retriesSinceFetch}}, then the reducer marks the record \{{STATUS_DB_GONE}}
when the count is no longer below \{{db.fetch.retry.max}} (default 3). The
status counter for the resulting datum is \{{db_gone}}, the same counter used
for a permanent failure. Nothing counts "retries exhausted."
h2. Proposal
Add constants to \{{NutchMetrics}} and cache the references.
In \{{FetcherThread}}, whenever the datum is set to \{{STATUS_FETCH_RETRY}},
increment \{{retries_total}} in \{{nutch_fetcher}} and exactly one cause
counter:
* \{{catch (Throwable)}}: \{{retries_timeout_total}},
\{{retries_network_total}}, or \{{retries_other_total}}, from
\{{ErrorTracker.categorize}}
* \{{ProtocolStatus.EXCEPTION}} whose message matches \{{Http code=}} with a
code from 500 to 599: \{{retries_http_5xx_total}}. Parse the code from the
message. Do not treat any other HTTP code as a 5xx
* \{{ProtocolStatus.EXCEPTION}} otherwise, and the unknown-status branch:
\{{retries_other_total}}
* \{{ProtocolStatus.RETRY}}: \{{retries_protocol_retry_total}}. Leave the
existing dynamic \{{retry}} counter in place
In \{{CrawlDbReducer}}, when the record is marked gone because
\{{retriesSinceFetch >= retryMax}}, increment \{{retries_exhausted_total}} in
\{{nutch_crawldb}}. This is a CrawlDb fact. Do not put it in \{{nutch_fetcher}}.
Keep the cause set fixed. Do not add a counter per status code or per host.
h2. Acceptance
* \{{ProtocolStatus.RETRY}} increments \{{retries_total}} and
\{{retries_protocol_retry_total}}, and does not increment a timeout, network,
or 5xx counter.
* An HTTP 500 carried as \{{ProtocolStatus.EXCEPTION}} with message \{{Http
code=500}} increments \{{retries_total}} and \{{retries_http_5xx_total}}. An
HTTP 403 or 429 on the same path increments \{{retries_other_total}} instead.
* A \{{SocketTimeoutException}} on the throwable path increments
\{{retries_total}} and \{{retries_timeout_total}}.
* A record whose \{{retriesSinceFetch}} reaches \{{db.fetch.retry.max}}
increments \{{nutch_crawldb}} / \{{retries_exhausted_total}}.
* A successful first fetch increments none of these counters.
* A unit test covers each of those five cases.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)