[
https://issues.apache.org/jira/browse/NUTCH-1201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13187904#comment-13187904
]
Edward Drapkin commented on NUTCH-1201:
---------------------------------------
I was thinking more of an approach of breaking Fetcher into these components:
interface Fetcher
class FetcherImpl
interface FetcherThread (extends Thread)
class FetcherThreadImpl
interface FetchItemQueue
class FetchItemQueueImpl
interface FetchQueueFeeder
class FetchQueueFeederImpl
Where all of the *Impl classes would be the current implementations of the
classes. I may be over-engineering here (I'm pretty prone to do that), but I
think that this would open up the potential to heavily profile fetching and
optimizing under various scenarios as I have a sneaky suspicion there's a lot
more lock contention and thread spinning that happens during fetching than
entirely necessary. It may be beneficial to offer several implementations out
of the box for various scenarios: single-threaded fetchers, lightweight queues
for short lists and/or small numbers of fetcher threads, heavyweight queues for
large workloads, etc.
> Allow for different FetcherThread impls
> ---------------------------------------
>
> Key: NUTCH-1201
> URL: https://issues.apache.org/jira/browse/NUTCH-1201
> Project: Nutch
> Issue Type: New Feature
> Components: fetcher
> Reporter: Markus Jelsma
> Assignee: Markus Jelsma
> Fix For: 1.5
>
>
> For certain cases we need to modify parts in FetcherThread and make it
> pluggable. This introduces a new config directive fetcher.impl that takes a
> FQCN and uses that setting Fetcher.fetch to load a class to use for
> job.setMapRunnerClass(). This new class has to extend Fetcher and and inner
> class FetcherThread. This allows for overriding methods in FetcherThread but
> also methods in Fetcher itself if required.
> A follow up on this issue would be to refactor parts of FetcherThread to make
> it easier to override small sections instead of copying the entire method
> body for a small change, which is now the case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
