[
https://issues.apache.org/jira/browse/NUTCH-3150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18057194#comment-18057194
]
ASF GitHub Bot commented on NUTCH-3150:
---------------------------------------
lewismc opened a new pull request, #892:
URL: https://github.com/apache/nutch/pull/892
PR for [NUTCH-3150](https://issues.apache.org/jira/browse/NUTCH-3150) which
Implements comprehensive counter caching optimization across all MapReduce jobs
to eliminate repeated `context.getCounter()` lookups in hot paths.
Breaking this PR down...
* Counter caching is now implemented in 16 MapReduce classes using a
standardized `initCounters(Context context)` pattern which I think improves
code interpretation aallowing for more intuitive future development around
metrics. I saw @igiguere evolving metrics counters in
https://github.com/apache/nutch/pull/891 which is excellent :)
* Migrated `DomainStatistics.java` from custom enum to NutchMetrics
constants with cached counters.
* Refactored inline counter initialization to dedicated `initCounters()`
methods for consistency across:
* Core crawl jobs: Fetcher, Generator, Injector, CrawlDbFilter,
CrawlDbReducer
* Post-processing: DeduplicationJob, CleaningJob, ParseSegment
* Analytics: DomainStatistics, WebGraph, SitemapProcessor
* HostDB: UpdateHostDbMapper, UpdateHostDbReducer, ResolverThread
* Export: WARCExporter
* Indexing: IndexerMapReduce
... the metrics journey continues.
> Expand Caching Hadoop Counter References
> ----------------------------------------
>
> Key: NUTCH-3150
> URL: https://issues.apache.org/jira/browse/NUTCH-3150
> Project: Nutch
> Issue Type: Sub-task
> Components: metrics
> Reporter: Lewis John McGibbney
> Assignee: Lewis John McGibbney
> Priority: Major
> Fix For: 1.22
>
>
> This issue expands on initial work to cache counters in hot paths only. See
> NUTCH-3141.
> After some analysis this issue can be scoped as follows
> 9 files to modify (~45 counters total):
> * Fetcher.java - 5 counters (main fetch loop)
> * ResolverThread.java - 10 counters (DNS resolution)
> * Generator.java - 9 counters (URL generation)
> * CrawlDbReducer.java - Dynamic counters with Map
> * Injector.java - 6 counters (mapper + reducer)
> * CrawlDbFilter.java - 3 counters
> * WebGraph.java - 2 counters
> * DeduplicationJob.java - 1 counter
> * CleaningJob.java - 1 counter
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
