The steps occur in this order 1) Inject 2) Generate 3) Fetcher 4) Parse 5) dbUpdate
I would suggest you to clean the database and start again. Please let me know your results. On Wed, Jan 30, 2013 at 2:52 PM, Weilei Zhang <[email protected]> wrote: > Hi > I am trying to use Nutch 2.x and have one question regarding Generator > and Injector: > Basically, I only have link as root to crawl and I see (by > instrumenting the code) that this one link was written to Context in > the last step of InjectorJob and that is the only link written to > Context from GeneratorJob. However, I saw multiple links sent to map > function in the first steps of GeneratorJob ( I instrumented setup > function). Those links seem to include all URLs referenced from the > original link. My question is where does fetch/parse happen? From the > Crawler code, it is straightforward to me that Injector is immediately > followed by Generator; I tried to scrub the code down to do the job > but failed. > > I ran crawl in the following way: > >/nutch crawl urlsDir > > There is only one link under a file in urlsDir. > >cat urlsDir/* > http://www.bmw.com > > The following is excerpt from the Generator map function > instrumentation output. Those are reversedURL. > al.com.bmw.www:http/ > al.com.bmw.www:http/al/en > am.bmw.www:http/ > am.bmw.www:http/am/en > ao.co.bmw:http/ > ao.co.bmw:http/ao/pt > ar.com.bmw.www:http/ > ar.com.bmw.www:http/ar/es/ > at.bmw.www:http/ > at.bmw.www:http/at/de/general/configurations_center/configure.html > at.bmw.www:http/de/index.html > > at.bmw.www:http/de/topics/services-angebote/connecteddrivedienste/connecteddrive-antrag/ueberblick.html > au.com.bmw.www:http/ > > au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/compare.html > > au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/configurator.html > > au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/driveawayprice.html > > au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/financecalculator.html > > au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/highlights/ > > au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/introduction.html > > au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/requestebrochure.html > > au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/requesttestdrive.html > > au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/compare.html > > au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/configurator.html > > au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/driveawayprice.html > > au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/financecalculator.html > > au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/highlights/ > > > Thanks for any hints! > -- > Best Regards > -Weilei > -- Kiran Chitturi
