Hello Joseph - on slow running fetchers and large segments. I would suggest to
use time bomb on the spot where most queues are crawled. That makes fetches
smaller (less time) and more efficient.
Markus
-----Original message-----
> From:Joseph Naegele <[email protected]>
> Sent: Tuesday 10th May 2016 18:08
> To: [email protected]
> Subject: RE: startUp/shutDown methods for plugins
>
> Markus, you're correct and I admit my use case is a bit exceptional. The one
> case I thought I needed a "shutdown" hook was when using adaptive threading
> and/or the fetch bandwidth threshold. When crawling large segments
> (multi-hour fetches) I assumed that if a slow fetcher is killed it would be
> nice to clean up the resources early for our app-specific logic. Anyway,
> relying on the JVM shutting down after each map/reduce is working fine for us.
>
> Thanks!
>
> -----Original Message-----
> From: Markus Jelsma [mailto:[email protected]]
> Sent: Monday, May 09, 2016 5:33 AM
> To: [email protected]
> Subject: RE: startUp/shutDown methods for plugins
>
> Hello Joseph, you can initialize things in the plugin's setConf() method, but
> there is no close(). Why do you want to clear resources? The plugin's
> lifetime is very short for most mappers and reducers, and Hadoop will kill
> those JVM's anyway.
>
> Markus
>
>
>
> -----Original message-----
> > From:Joseph Naegele <[email protected]>
> > Sent: Friday 6th May 2016 16:04
> > To: [email protected]
> > Subject: startUp/shutDown methods for plugins
> >
> > Hi folks,
> >
> >
> >
> > I'm using Nutch 1.11. Is it possible to implement plugin instance
> > startUp/shutDown methods for normal extension points? This would allow
> > for cleaning up resources at the end of a plugin instance's lifetime.
> >
> >
> >
> > Thanks,
> >
> > Joe
> >
> >
>
>
