[ https://issues.apache.org/jira/browse/NUTCH-3131 ]
Lewis John McGibbney deleted comment on NUTCH-3131:
---------------------------------------------
was (Author: lewismc):
I plan to add more to this issue but I won't be able to get around to it for a
little while.
> Nutch Metrics Refactoring & Enhancements
> ----------------------------------------
>
> Key: NUTCH-3131
> URL: https://issues.apache.org/jira/browse/NUTCH-3131
> Project: Nutch
> Issue Type: Improvement
> Components: metrics
> Reporter: Lewis John McGibbney
> Assignee: Lewis John McGibbney
> Priority: Major
> Fix For: 1.24
>
> Attachments: nutch_metrics_report.pdf
>
>
> The recent “Hadoop Metrics Analysis and Improvement Suggestions” report
> (attached PDF) identified 88 counter increment operations across 17 files
> with multiple anti-patterns and missing key observability signals
> (inconsistent naming, repeated counter lookups, missing latency/error
> context, cardinality risks, etc.).
> This ticket covers the implementation of all high and medium-priority
> improvements from the report (Phases 1–3). Phase 4 (external export,
> dashboard, testing) will be handled in separate tickets.
> Acceptance criteria for consideration
> * All Hadoop counter group and counter names are defined in a single source
> of truth (new class NutchMetricConstants or NutchMetrics).
> * No hardcoded counter group/name strings remain in the 17 affected files.
> * All frequently used counters (especially in hot paths – Fetcher,
> FetcherThread, Generator, Parser, Indexer) are cached in instance variables
> during setup(Context) / setup() and reused.
> * Latency metrics (fetch, parse, index) are added with proper timing and
> recorded via Hadoop counters (average + count).
> * Error counters include error type/context where feasible (at least
> class-level granularity).
> * Counter naming is fully standardized (camelCase counters, PascalCase
> groups).
> * LatencyTracker and ErrorTracker are the shared metrics utilities.
> LatencyTracker is used by Fetcher, ParseSegment, and IndexerMapReduce.
> ErrorTracker is used by the jobs that already construct it (FetcherThread,
> Generator, Injector, ParseSegment, CrawlDbReducer, CrawlDbFilter,
> IndexerMapReduce, UpdateHostDbMapper, SitemapProcessor, WARCExporter,
> CommonCrawlDataDumper). There is no MetricsHelper class. Jobs that still log
> a per-record failure without an error counter are tracked separately.
> *
> FetcherThread.updateStatus publishes pages_fetched_total alongside the
> existing pages atomic. bytes_downloaded_total continues to be updated from
> the fetch loop. errors_total continues to be emitted by ErrorTracker when the
> fetch thread finishes. activeThreads, spinWaiting, and lastRequestStart stay
> as live atomics, because the fetch loop and the thread controls read them
> during the run. There is no ThreadSafeMetrics class and no
> FetcherRun.cleanup().
> * Resource utilization metrics (queue sizes, depths) are added for Fetcher.
> * Basic metrics validation is executed at the end of each job (warn on
> impossible conditions).
> * No regression in existing counter values (verified via existing
> integration tests or new sanity job).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
