Lewis John McGibbney created NUTCH-3241:
-------------------------------------------
Summary: Add a generated-URL counter and selection rates to the
Generator
Key: NUTCH-3241
URL: https://issues.apache.org/jira/browse/NUTCH-3241
Project: Nutch
Issue Type: Sub-task
Reporter: Lewis John McGibbney
{\{NutchMetrics}} defines nine \{{nutch_generator}} counters, and every one of
those constants is a rejection or an overflow:
* \{{url_filters_rejected_total}}
* \{{schedule_rejected_total}}
* \{{wait_for_update_total}}
* \{{expr_rejected_total}}
* \{{status_rejected_total}}
* \{{score_too_low_total}}
* \{{interval_rejected_total}}
* \{{urls_skipped_per_host_overflow_total}}
* \{{hosts_affected_per_host_overflow_total}}
The group is not rejection-only at runtime. The selector mapper and the reducer
each construct an \{{ErrorTracker}} on \{{nutch_generator}}, so the group also
contains \{{errors_total}} and the category counters. After the job,
\{{Generator}} logs every non-zero counter in the group, not just the nine.
Nothing counts the URLs actually written to a segment, so the selection rate
cannot be computed from the counters. The metrics report calls this out: many
rejection counters, no success rate.
h2. Proposal
* Add \{{urls_generated_total}} to \{{NutchMetrics.GROUP_GENERATOR}} and
increment it once per URL written to a segment.
* After the job completes, record the selection rate and each rejection
reason's share of the URLs considered. A ratio is a gauge, so set these with
\{{Counter.setValue}}, the same way \{{LatencyTracker}} publishes percentiles.
* Store each ratio as parts per 10,000 and document the scale on the constant.
Suggested names: \{{selection_rate_per_10000}} and one
\{{<reason>_rate_per_10000}} per existing rejection constant. Do not add rate
counters for the \{{ErrorTracker}} category counters.
{\{urls_considered}} for the rate is \{{urls_generated_total}} plus the nine
rejection counters. Do not add \{{hosts_affected_per_host_overflow_total}} into
that sum; it counts hosts, not URLs. \{{urls_skipped_per_host_overflow_total}}
does count URLs and belongs in the sum.
h2. Acceptance
* A run that selects URLs leaves \{{urls_generated_total}} equal to the number
of URLs in the generated segment.
* A run that rejects URLs for a known reason leaves that reason's rate counter
consistent with the rejection count and the number of URLs considered.
* A run that generates nothing leaves the rate counters at zero and does not
divide by zero.
* The nine existing rejection counters, and the \{{ErrorTracker}} counters, are
unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
