Lewis John McGibbney created NUTCH-3243:
-------------------------------------------
Summary: Add a pages-fetched counter to the Fetcher
Key: NUTCH-3243
URL: https://issues.apache.org/jira/browse/NUTCH-3243
Project: Nutch
Issue Type: Sub-task
Reporter: Lewis John McGibbney
{\{Fetcher.FetcherRun}} keeps these atomic fields
(\{{src/java/org/apache/nutch/fetcher/Fetcher.java}}):
* \{{AtomicInteger pages}}
* \{{AtomicLong bytes}}
* \{{AtomicInteger errors}}
* \{{AtomicInteger activeThreads}}
* \{{AtomicInteger spinWaiting}}
* \{{AtomicLong lastRequestStart}}
What each one actually is:
* \{{bytes}} is copied to \{{bytes_downloaded_total}} once a second inside the
fetch loop. The loop is a do/while. The last pass increments the counter after
the sleep, then sees \{{activeThreads == 0}} and exits. A successful task does
not drop the tail. A failed task loses every counter from that task, including
bytes already incremented, because Hadoop drops counters from a failed task.
Flushing earlier does not survive a failed task. \{{FetcherRun}} has no
\{{cleanup()}}.
* \{{errors}} is incremented in \{{FetcherThread.logError}}, and the same
methods call \{{errorTracker.recordError}}. \{{errorTracker.emitCounters}} runs
in the thread's \{{finally}} block, so a thread that finishes publishes
\{{errors_total}}. A second counter from the atomic would double-count.
* \{{pages}} is incremented only in \{{FetcherThread.updateStatus}}, which runs
after a successful fetch. It is read for the pages/sec status line and for the
throughput threshold (\{{pagesLastSec}}). It is never copied to a Hadoop
counter. \{{NutchMetrics}} has no pages counter.
* \{{activeThreads}} is the fetch-loop condition and the guard for spawning
threads. \{{spinWaiting}} and \{{activeThreads}} are read by the status line,
the bandwidth thread adjustment, and the hung-thread check.
\{{lastRequestStart}} is a timestamp used to detect hung threads. None of these
are totals waiting to be flushed.
[NUTCH-3146|https://issues.apache.org/jira/browse/NUTCH-3146] adds peak gauges
on top of the same fields. Replacing them with an accumulator flushed at the
end would break thread control.
The report's "consolidate the atomics into ThreadSafeMetrics" recommendation
does not match this code. The missing counter is pages fetched.
h2. Proposal
* Add \{{FETCHER_PAGES_FETCHED_TOTAL = "pages_fetched_total"}} to
\{{NutchMetrics}}, group \{{nutch_fetcher}}.
* Cache the counter on \{{FetcherThread}} and increment it by 1 in
\{{updateStatus}}, next to \{{pages.incrementAndGet()}}.
* Leave \{{pages}}, \{{bytes}}, \{{errors}}, \{{activeThreads}},
\{{spinWaiting}}, and \{{lastRequestStart}} as they are.
* Do not add a counter for the \{{errors}} atomic.
h2. Acceptance
* A fetch task that successfully fetches N pages leaves
\{{pages_fetched_total}} at N, including pages fetched in the last second.
* \{{bytes_downloaded_total}}, \{{errors_total}}, the throughput threshold, and
the status line are unchanged.
* \{{activeThreads}} is still the loop condition.
* A unit test covers a successful fetch and a fetch that does not call
\{{updateStatus}}.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
