[
https://issues.apache.org/jira/browse/NUTCH-3241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lewis John McGibbney updated NUTCH-3241:
----------------------------------------
Description:
{{NutchMetrics}} defines nine {{nutch_generator}} counters, and every one of
those constants is a rejection or an overflow:
* {url_filters_rejected_total}
* {schedule_rejected_total}
* {wait_for_update_total}
* {expr_rejected_total}
* {status_rejected_total}
* {score_too_low_total}
* {interval_rejected_total}
* {urls_skipped_per_host_overflow_total}
* {hosts_affected_per_host_overflow_total}
The group is not rejection-only at runtime. The selector mapper and the reducer
each construct an {{ErrorTracker}} on {{{}nutch_generator{}}}, so the group
also contains {{errors_total}} and the category counters. After the job,
{{Generator}} logs every non-zero counter in the group, not just the nine.
Nothing counts the URLs actually written to a segment, so the selection rate
cannot be computed from the counters. The metrics report calls this out: many
rejection counters, no success rate.
h2. Proposal
* Add {{urls_generated_total}} to {{NutchMetrics.GROUP_GENERATOR}} and
increment it once per URL written to a segment.
* After the job completes, record the selection rate and each rejection
reason's share of the URLs considered. A ratio is a gauge, so set these with
{{{}Counter.setValue{}}}, the same way {{LatencyTracker}} publishes percentiles.
* Store each ratio as parts per 10,000 and document the scale on the constant.
Suggested names: {{selection_rate_per_10000}} and one
{{<reason>_rate_per_10000}} per existing rejection constant. Do not add rate
counters for the {{ErrorTracker}} category counters.
{{urls_considered}} for the rate is {{urls_generated_total}} plus the nine
rejection counters. Do not add {{hosts_affected_per_host_overflow_total}} into
that sum; it counts hosts, not URLs. {{urls_skipped_per_host_overflow_total}}
does count URLs and belongs in the sum.
h2. Acceptance
* A run that selects URLs leaves {{urls_generated_total}} equal to the number
of URLs in the generated segment.
* A run that rejects URLs for a known reason leaves that reason's rate counter
consistent with the rejection count and the number of URLs considered.
* A run that generates nothing leaves the rate counters at zero and does not
divide by zero.
* The nine existing rejection counters, and the {{ErrorTracker}} counters, are
unchanged.
was:
{\{NutchMetrics}} defines nine \{{nutch_generator}} counters, and every one of
those constants is a rejection or an overflow:
* \{{url_filters_rejected_total}}
* \{{schedule_rejected_total}}
* \{{wait_for_update_total}}
* \{{expr_rejected_total}}
* \{{status_rejected_total}}
* \{{score_too_low_total}}
* \{{interval_rejected_total}}
* \{{urls_skipped_per_host_overflow_total}}
* \{{hosts_affected_per_host_overflow_total}}
The group is not rejection-only at runtime. The selector mapper and the reducer
each construct an \{{ErrorTracker}} on \{{nutch_generator}}, so the group also
contains \{{errors_total}} and the category counters. After the job,
\{{Generator}} logs every non-zero counter in the group, not just the nine.
Nothing counts the URLs actually written to a segment, so the selection rate
cannot be computed from the counters. The metrics report calls this out: many
rejection counters, no success rate.
h2. Proposal
* Add \{{urls_generated_total}} to \{{NutchMetrics.GROUP_GENERATOR}} and
increment it once per URL written to a segment.
* After the job completes, record the selection rate and each rejection
reason's share of the URLs considered. A ratio is a gauge, so set these with
\{{Counter.setValue}}, the same way \{{LatencyTracker}} publishes percentiles.
* Store each ratio as parts per 10,000 and document the scale on the constant.
Suggested names: \{{selection_rate_per_10000}} and one
\{{<reason>_rate_per_10000}} per existing rejection constant. Do not add rate
counters for the \{{ErrorTracker}} category counters.
{\{urls_considered}} for the rate is \{{urls_generated_total}} plus the nine
rejection counters. Do not add \{{hosts_affected_per_host_overflow_total}} into
that sum; it counts hosts, not URLs. \{{urls_skipped_per_host_overflow_total}}
does count URLs and belongs in the sum.
h2. Acceptance
* A run that selects URLs leaves \{{urls_generated_total}} equal to the number
of URLs in the generated segment.
* A run that rejects URLs for a known reason leaves that reason's rate counter
consistent with the rejection count and the number of URLs considered.
* A run that generates nothing leaves the rate counters at zero and does not
divide by zero.
* The nine existing rejection counters, and the \{{ErrorTracker}} counters, are
unchanged.
> Add a generated-URL counter and selection rates to the Generator
> ----------------------------------------------------------------
>
> Key: NUTCH-3241
> URL: https://issues.apache.org/jira/browse/NUTCH-3241
> Project: Nutch
> Issue Type: Sub-task
> Reporter: Lewis John McGibbney
> Priority: Major
>
> {{NutchMetrics}} defines nine {{nutch_generator}} counters, and every one of
> those constants is a rejection or an overflow:
> * {url_filters_rejected_total}
> * {schedule_rejected_total}
> * {wait_for_update_total}
> * {expr_rejected_total}
> * {status_rejected_total}
> * {score_too_low_total}
> * {interval_rejected_total}
> * {urls_skipped_per_host_overflow_total}
> * {hosts_affected_per_host_overflow_total}
> The group is not rejection-only at runtime. The selector mapper and the
> reducer each construct an {{ErrorTracker}} on {{{}nutch_generator{}}}, so the
> group also contains {{errors_total}} and the category counters. After the
> job, {{Generator}} logs every non-zero counter in the group, not just the
> nine.
> Nothing counts the URLs actually written to a segment, so the selection rate
> cannot be computed from the counters. The metrics report calls this out: many
> rejection counters, no success rate.
> h2. Proposal
> * Add {{urls_generated_total}} to {{NutchMetrics.GROUP_GENERATOR}} and
> increment it once per URL written to a segment.
> * After the job completes, record the selection rate and each rejection
> reason's share of the URLs considered. A ratio is a gauge, so set these with
> {{{}Counter.setValue{}}}, the same way {{LatencyTracker}} publishes
> percentiles.
> * Store each ratio as parts per 10,000 and document the scale on the
> constant. Suggested names: {{selection_rate_per_10000}} and one
> {{<reason>_rate_per_10000}} per existing rejection constant. Do not add rate
> counters for the {{ErrorTracker}} category counters.
> {{urls_considered}} for the rate is {{urls_generated_total}} plus the nine
> rejection counters. Do not add {{hosts_affected_per_host_overflow_total}}
> into that sum; it counts hosts, not URLs.
> {{urls_skipped_per_host_overflow_total}} does count URLs and belongs in the
> sum.
> h2. Acceptance
> * A run that selects URLs leaves {{urls_generated_total}} equal to the
> number of URLs in the generated segment.
> * A run that rejects URLs for a known reason leaves that reason's rate
> counter consistent with the rejection count and the number of URLs considered.
> * A run that generates nothing leaves the rate counters at zero and does not
> divide by zero.
> * The nine existing rejection counters, and the {{ErrorTracker}} counters,
> are unchanged.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
