Lewis John McGibbney created NUTCH-3238:
-------------------------------------------
Summary: Reject invalid latency samples and count them in
nutch_validation
Key: NUTCH-3238
URL: https://issues.apache.org/jira/browse/NUTCH-3238
Project: Nutch
Issue Type: Sub-task
Reporter: Lewis John McGibbney
The Hadoop metrics analysis (\{{nutch_metrics_report.md}}, suggestion 8) asks
for checks that recorded counters are plausible.
[NUTCH-3131|https://issues.apache.org/jira/browse/NUTCH-3131] lists "Basic
metrics validation is executed at the end of each job" as an acceptance
criterion. No code checks counter values today.
A post-job pass is the wrong shape. \{{bin/crawl}} runs inject, generate,
fetch, parse, updatedb and index as separate processes, and each one blocks on
\{{job.waitForCompletion(true)}} before the next starts. A check that runs
after that call sits on the critical path of every step. A counter incremented
on the client after the job finishes is also never written to the job's
counters, so it would not be visible anyway.
h2. What is checkable inside the task
Most invariants that look useful are true by construction inside one tracker,
so a runtime check on them cannot fail:
* \{{ErrorTracker.recordError}} and \{{incrementCounters}} each add one to
\{{errors_total}} and one to exactly one category, in the same call.
\{{emitCounters}} writes both from the same local state.
* \{{LatencyTracker.emitCounters}} writes count, sum and percentiles from the
same digest.
The useful form of these invariants compares values written by different code
paths, for example a raw \{{getCounter}} write to an \{{errors_*}} name in a
group that also has an \{{ErrorTracker}}. Those are properties of the code, so
they are covered by the tests in issue 8, not by a runtime check.
One input can actually be bad at runtime. \{{LatencyTracker.record(long)}} has
no guard, and all three callers compute the sample as a difference of
\{{System.currentTimeMillis()}}:
* \{{FetcherThread}}: \{{fetchLatencyTracker.record(System.currentTimeMillis()
- fetchStart)}}
* \{{IndexerMapReduce}}:
\{{indexLatencyTracker.record(System.currentTimeMillis() - indexStart)}}
* \{{ParseSegment}}: \{{parseLatencyTracker.record(parseTime)}}, where
\{{parseTime = end - start}} from two \{{System.currentTimeMillis()}} calls
If the wall clock steps backwards during a task, one of those differences is
negative. \{{record}} adds it to the digest and to \{{sum}}, and the percentile
counters and \{{*_sum_ms}} absorb it with no sign that anything went wrong.
h2. Proposal
In \{{LatencyTracker.record}}, reject a negative sample before it reaches the
digest, the count, or the sum. Keep a local count of rejected samples. Emit
that count from the same call that already emits count and sum, under
\{{nutch_validation}}, as \{{latency_negative_sample_total}}. Declare the group
and name in \{{NutchMetrics}}. Log one warning per task, the first time a
sample is rejected.
Reject the sample rather than clamping it to zero. Rejecting keeps
\{{count_total}}, \{{sum_ms}} and the percentiles consistent with each other. A
clamped zero would pull the percentiles down with no trace in the output.
{\{LatencyTracker.merge}} must carry the rejected count. Each task emits its
own count, so the job total is counted once.
The job is never failed by this check.
h2. Acceptance
* \{{record(-1)}} leaves \{{getCount()}}, \{{getSum()}} and the percentiles
unchanged, and increases the rejected count by one.
* A task that rejects samples emits \{{nutch_validation}} /
\{{latency_negative_sample_total}} with that count, and logs one warning.
* A run with no negative samples leaves \{{nutch_validation}} /
\{{latency_negative_sample_total}} at zero, and produces the same
\{{nutch_fetcher}}, \{{nutch_parser}} and \{{nutch_indexer}} latency counters
as before the change.
* A unit test covers \{{record}} with a negative value and \{{merge}} carrying
the rejected count.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)