-----Original message----- > From:Vikas Hazrati <[email protected]> > Sent: Mon 28-May-2012 13:55 > To: [email protected] > Subject: Re: Setting the Fetch time with a CustomFetchSchedule > > Thanks Markus, what I understand from the code is that I should be able to > extract and pass meta information from my ParsePlugin and access that as a > part of the custom fetch schedule which extends AbstractFetchSchedule. > > If I create a custom fetch class as > > class CustomEventFetchScheduler extends AbstractFetchSchedule { ...} > > how do i include this custom class a part of my crawl cycle? I understand > that there is no extension point for this?
Indeed, there is no extension point so you cannot make a nice plugin. What you can do is make sure it's on the classpath and simply tell the scheduler to use it via db.fetch.schedule.class, that should work just fine. > > I get this -> Caused by: java.lang.RuntimeException: Plugin > (12kdaggregator), extension point: org.apache.nutch.crawl.FetchSchedule > does not exist. > > Also I could not successfully plug it as a part of nutch-site.xml by > overriding the nutch-default.xml > > > <property> > <name>db.fetch.schedule.class</name> > <value>com.custom.CustomEventFetchScheduler</value> > </property> > > > How do I include my custom logic so that it gets picked as a part of the > crawl cycle. > > Regards | Vikas > > On Mon, May 21, 2012 at 6:14 PM, Markus Jelsma > <[email protected]>wrote: > > > Yes, you can pass ParseMeta keys to the FetchSchedule as part of the > > CrawlDatum's meta data as i did with: > > https://issues.apache.org/jira/browse/NUTCH-1024 > > > > > > -----Original message----- > > > From:Vikas Hazrati <[email protected]> > > > Sent: Mon 21-May-2012 13:44 > > > To: [email protected] > > > Subject: Setting the Fetch time with a CustomFetchSchedule > > > > > > Hi, > > > > > > I would like to implement a custom implementation of > > AbstractFetchSchedule > > > and would like to change the FetchTime on the basis of some parameters > > that > > > I get as a part of my parsing. > > > > > > // something like this > > > datum.setFetchTime(fetchTime + (long)datum.getFetchInterval() * 1000 + > > > customLogic); > > > > > > Right now I have a custom URLFilter and a custom parser which extends > > > HtmlParseFilter. At the time of custom parsing, I come across some > > > parameters which would help me define how should I define the fetchtime > > for > > > that URL. I would like to pass these values to my CustomFetchSchedule. > > > > > > Is there a way to do that? Can I pass them as a part of configuration? > > > > > > Since I would get the data that i need to make a decision only as a part > > of > > > Parse, would it be possible to pass this data to the FetchSchedule? > > > > > > Thoughts? > > > > > > Regards | Vikas > > > > > >
