[
https://issues.apache.org/jira/browse/NUTCH-3190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lewis John McGibbney updated NUTCH-3190:
----------------------------------------
Description:
{\{IndexerMapReduce}} records index latency on one \{{LatencyTracker}} per
reduce task and publishes it from \{{cleanup}} with \{{emitCounters}}. That
method calls \{{Counter.setValue}} for \{{index_latency_count_total}},
\{{index_latency_sum_ms}}, and the three percentile counters
(\{{IndexerMapReduce}} around the \{{cleanup}} method, and
\{{indexLatencyTracker.record}} at the end of \{{reduce}}).
Hadoop sums counter values across tasks. Summing is right for the count and the
sum. It is wrong for a percentile: with more than one reduce task,
\{{index_latency_p50_ms}} becomes the sum of the per-task medians, and the same
happens to p95 and p99. That is the second bug described in
[NUTCH-3162|https://issues.apache.org/jira/browse/NUTCH-3162]. The Fetcher and
the Parser were fixed there. The indexer was not.
{\{Fetcher.FetcherReducer}} and \{{ParseSegment}} now call
\{{emitCountAndSumOnly}} from each task, write the serialized TDigest under
\{{NutchMetrics.LATENCY_KEY}}, and have one reducer merge the digests and
\{{setValue}} the percentile counters once. \{{IndexerMapReduce}} does none of
this. It has no \{{LATENCY_KEY}} branch.
An earlier attempt to fix the indexer, in pull request 906, wrote the digests
to a temporary \{{_latency}} directory and added that path as job input. That
fails the index job when the path is absent. Reproduced with \{{bin/nutch index
-nocrawldb}} and on a single-node cluster: \{{InvalidInputException: Input path
does not exist: .../_latency}}. The review on that pull request
(https://github.com/apache/nutch/pull/906#pullrequestreview-4081904105) asked
for the indexer change to be taken out of NUTCH-3162 and done separately. Do
not bring the temporary directory back.
h2. Proposal
Follow the Fetcher and Parser pattern inside the existing indexing job.
* From each reduce task, call \{{emitCountAndSumOnly}} instead of
\{{emitCounters}}.
* Write the serialized digest as a \{{NutchWritable}} wrapping a
\{{BytesWritable}}, keyed by \{{NutchMetrics.LATENCY_KEY}}. Writing a bare
\{{BytesWritable}} fails the task. ParseSegment hit that during NUTCH-3162:
\{{Type mismatch in value from map: expected org.apache.hadoop.io.Writable,
received org.apache.hadoop.io.BytesWritable}}.
* Send every \{{LATENCY_KEY}} record to one reducer, the way
\{{ParseSegment}}'s partitioner does, and merge the digests there. Set
\{{index_latency_p50_ms}}, \{{index_latency_p95_ms}}, and
\{{index_latency_p99_ms}} once from the merged digest.
* Leave \{{index_latency_count_total}} and \{{index_latency_sum_ms}} to
Hadoop's summation of the per-task counters.
h2. Acceptance
* An index job with more than one reduce task leaves \{{index_latency_p50_ms}}
equal to the median of the merged samples, not the sum of the per-task medians.
The same holds for p95 and p99.
* \{{index_latency_count_total}} equals the number of documents passed to
\{{record}}, and \{{index_latency_sum_ms}} equals the sum of those samples.
* \{{bin/nutch index -nocrawldb}} against a segment completes, with no
\{{_latency}} input path.
* A unit test merges digests from two reduce tasks and checks the percentile
counters.
was:Similar to NUTCH-3162, all context is in that ticket.
> Indexer latency metrics to properly merge data from all threads and tasks
> -------------------------------------------------------------------------
>
> Key: NUTCH-3190
> URL: https://issues.apache.org/jira/browse/NUTCH-3190
> Project: Nutch
> Issue Type: Sub-task
> Components: indexer
> Affects Versions: 1.22
> Reporter: Lewis John McGibbney
> Assignee: Lewis John McGibbney
> Priority: Major
> Fix For: 1.24
>
> Attachments: NUTCH-3162-indexer-latency.patch
>
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
