Hi I am trying to use Nutch 2.x and have one question regarding Generator and Injector: Basically, I only have link as root to crawl and I see (by instrumenting the code) that this one link was written to Context in the last step of InjectorJob and that is the only link written to Context from GeneratorJob. However, I saw multiple links sent to map function in the first steps of GeneratorJob ( I instrumented setup function). Those links seem to include all URLs referenced from the original link. My question is where does fetch/parse happen? From the Crawler code, it is straightforward to me that Injector is immediately followed by Generator; I tried to scrub the code down to do the job but failed.
I ran crawl in the following way: >/nutch crawl urlsDir There is only one link under a file in urlsDir. >cat urlsDir/* http://www.bmw.com The following is excerpt from the Generator map function instrumentation output. Those are reversedURL. al.com.bmw.www:http/ al.com.bmw.www:http/al/en am.bmw.www:http/ am.bmw.www:http/am/en ao.co.bmw:http/ ao.co.bmw:http/ao/pt ar.com.bmw.www:http/ ar.com.bmw.www:http/ar/es/ at.bmw.www:http/ at.bmw.www:http/at/de/general/configurations_center/configure.html at.bmw.www:http/de/index.html at.bmw.www:http/de/topics/services-angebote/connecteddrivedienste/connecteddrive-antrag/ueberblick.html au.com.bmw.www:http/ au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/compare.html au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/configurator.html au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/driveawayprice.html au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/financecalculator.html au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/highlights/ au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/introduction.html au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/requestebrochure.html au.com.bmw.www:http/com/en/newvehicles/1series/5door/2011/showroom/requesttestdrive.html au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/compare.html au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/configurator.html au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/driveawayprice.html au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/financecalculator.html au.com.bmw.www:http/com/en/newvehicles/1series/convertible/2011/showroom/highlights/ Thanks for any hints! -- Best Regards -Weilei
