Lewis John McGibbney created NUTCH-3238:
-------------------------------------------

             Summary: Reject invalid latency samples and count them in 
nutch_validation
                 Key: NUTCH-3238
                 URL: https://issues.apache.org/jira/browse/NUTCH-3238
             Project: Nutch
          Issue Type: Sub-task
            Reporter: Lewis John McGibbney


The Hadoop metrics analysis (\{{nutch_metrics_report.md}}, suggestion 8) asks 
for checks that recorded counters are plausible. 
[NUTCH-3131|https://issues.apache.org/jira/browse/NUTCH-3131] lists "Basic 
metrics validation is executed at the end of each job" as an acceptance 
criterion. No code checks counter values today.

A post-job pass is the wrong shape. \{{bin/crawl}} runs inject, generate, 
fetch, parse, updatedb and index as separate processes, and each one blocks on 
\{{job.waitForCompletion(true)}} before the next starts. A check that runs 
after that call sits on the critical path of every step. A counter incremented 
on the client after the job finishes is also never written to the job's 
counters, so it would not be visible anyway.

h2. What is checkable inside the task

Most invariants that look useful are true by construction inside one tracker, 
so a runtime check on them cannot fail:

* \{{ErrorTracker.recordError}} and \{{incrementCounters}} each add one to 
\{{errors_total}} and one to exactly one category, in the same call. 
\{{emitCounters}} writes both from the same local state.
* \{{LatencyTracker.emitCounters}} writes count, sum and percentiles from the 
same digest.

The useful form of these invariants compares values written by different code 
paths, for example a raw \{{getCounter}} write to an \{{errors_*}} name in a 
group that also has an \{{ErrorTracker}}. Those are properties of the code, so 
they are covered by the tests in issue 8, not by a runtime check.

One input can actually be bad at runtime. \{{LatencyTracker.record(long)}} has 
no guard, and all three callers compute the sample as a difference of 
\{{System.currentTimeMillis()}}:

* \{{FetcherThread}}: \{{fetchLatencyTracker.record(System.currentTimeMillis() 
- fetchStart)}}
* \{{IndexerMapReduce}}: 
\{{indexLatencyTracker.record(System.currentTimeMillis() - indexStart)}}
* \{{ParseSegment}}: \{{parseLatencyTracker.record(parseTime)}}, where 
\{{parseTime = end - start}} from two \{{System.currentTimeMillis()}} calls

If the wall clock steps backwards during a task, one of those differences is 
negative. \{{record}} adds it to the digest and to \{{sum}}, and the percentile 
counters and \{{*_sum_ms}} absorb it with no sign that anything went wrong.

h2. Proposal

In \{{LatencyTracker.record}}, reject a negative sample before it reaches the 
digest, the count, or the sum. Keep a local count of rejected samples. Emit 
that count from the same call that already emits count and sum, under 
\{{nutch_validation}}, as \{{latency_negative_sample_total}}. Declare the group 
and name in \{{NutchMetrics}}. Log one warning per task, the first time a 
sample is rejected.

Reject the sample rather than clamping it to zero. Rejecting keeps 
\{{count_total}}, \{{sum_ms}} and the percentiles consistent with each other. A 
clamped zero would pull the percentiles down with no trace in the output.

{\{LatencyTracker.merge}} must carry the rejected count. Each task emits its 
own count, so the job total is counted once.

The job is never failed by this check.

h2. Acceptance

* \{{record(-1)}} leaves \{{getCount()}}, \{{getSum()}} and the percentiles 
unchanged, and increases the rejected count by one.
* A task that rejects samples emits \{{nutch_validation}} / 
\{{latency_negative_sample_total}} with that count, and logs one warning.
* A run with no negative samples leaves \{{nutch_validation}} / 
\{{latency_negative_sample_total}} at zero, and produces the same 
\{{nutch_fetcher}}, \{{nutch_parser}} and \{{nutch_indexer}} latency counters 
as before the change.
* A unit test covers \{{record}} with a negative value and \{{merge}} carrying 
the rejected count.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

Reply via email to