Half-way, it's clear in the log. Is your document a redirect, i've not yet seen such a log line before.
* haven't double-checked source code > Not sure where fetching starts... > > 2011-12-19 20:13:53,223 INFO crawl.FetchScheduleFactory - Using > FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule > 2011-12-19 20:13:53,223 INFO crawl.AbstractFetchSchedule - > defaultInterval=2592000 > 2011-12-19 20:13:53,223 INFO crawl.AbstractFetchSchedule - > maxInterval=7776000 2011-12-19 20:13:53,261 INFO regex.RegexURLNormalizer > - can't find rules for scope 'partition', using default > 2011-12-19 20:13:53,394 INFO crawl.FetchScheduleFactory - Using > FetchSchedule impl: org.apache.nutch.crawl.DefaultFetchSchedule > 2011-12-19 20:13:53,395 INFO crawl.AbstractFetchSchedule - > defaultInterval=2592000 > 2011-12-19 20:13:53,395 INFO crawl.AbstractFetchSchedule - > maxInterval=7776000 2011-12-19 20:13:53,399 INFO regex.RegexURLNormalizer > - can't find rules for scope 'generate_host_count', using default > 2011-12-19 20:13:54,474 INFO crawl.Generator - Generator: > Partitioning selected urls for politeness. > 2011-12-19 20:13:55,479 INFO crawl.Generator - Generator: segment: > /cdda/nutch/crawl/segments/20111219201355 > 2011-12-19 20:13:56,537 INFO regex.RegexURLNormalizer - can't find > rules for scope 'partition', using default > 2011-12-19 20:13:56,939 INFO crawl.Generator - Generator: finished at > 2011-12-19 20:13:56, elapsed: 00:00:05 > 2011-12-19 20:13:57,695 INFO fetcher.Fetcher - Fetcher: starting at > 2011-12-19 20:13:57 > 2011-12-19 20:13:57,695 INFO fetcher.Fetcher - Fetcher: segment: > /nutch/crawl/segments/20111219201355 > 2011-12-19 20:13:58,743 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:58,744 INFO fetcher.Fetcher - Fetcher: threads: 10 > 2011-12-19 20:13:58,744 INFO fetcher.Fetcher - Fetcher: time-out divisor: > 2 2011-12-19 20:13:58,749 DEBUG fetcher.Fetcher - -feeding 500 input urls > ... 2011-12-19 20:13:58,756 INFO plugin.PluginRepository - Plugins: > looking in: /nutch/plugins > 2011-12-19 20:13:58,774 INFO fetcher.Fetcher - QueueFeeder finished: > total 1 records + hit by time limit :0 > <cut plugin loader stuff, can push this if you need it> > 2011-12-19 20:13:59,036 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,036 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,036 INFO fetcher.Fetcher - fetching > http://wnspstg8o.imostg.intelink.gov/sites/mlogic/Shared > Documents/Alpha.docx > 2011-12-19 20:13:59,036 DEBUG fetcher.Fetcher - redirectCount=0 > 2011-12-19 20:13:59,038 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,039 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > 2011-12-19 20:13:59,039 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,039 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > 2011-12-19 20:13:59,040 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,040 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > 2011-12-19 20:13:59,040 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,041 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > 2011-12-19 20:13:59,041 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,041 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > 2011-12-19 20:13:59,041 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,042 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > 2011-12-19 20:13:59,042 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,042 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > 2011-12-19 20:13:59,042 INFO fetcher.Fetcher - Using queue mode : byHost > 2011-12-19 20:13:59,043 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > 2011-12-19 20:13:59,043 INFO fetcher.Fetcher - Fetcher: throughput > threshold: -1 > 2011-12-19 20:13:59,043 INFO fetcher.Fetcher - Fetcher: throughput > threshold retries: 5 > 2011-12-19 20:13:59,043 INFO http.Http - http.proxy.host = null > 2011-12-19 20:13:59,043 INFO http.Http - http.proxy.port = 8080 > 2011-12-19 20:13:59,043 INFO http.Http - http.timeout = 10000 > 2011-12-19 20:13:59,043 INFO http.Http - http.content.limit = -1 > 2011-12-19 20:13:59,043 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=1 > 2011-12-19 20:13:59,043 INFO http.Http - http.agent = > google-robot-intelink/Nutch-1.4 (CDDA Crawler; search.intelink.gov; > [email protected]) > 2011-12-19 20:13:59,043 INFO http.Http - http.accept.language = > en-us,en-gb,en;q=0.7,*;q=0.3 > 2011-12-19 20:13:59,380 INFO fetcher.Fetcher - -finishing thread > FetcherThread, activeThreads=0 > 2011-12-19 20:14:00,050 INFO fetcher.Fetcher - -activeThreads=0, > spinWaiting=0, fetchQueues.totalSize=0 > 2011-12-19 20:14:00,050 INFO fetcher.Fetcher - -activeThreads=0 > 2011-12-19 20:14:00,372 WARN util.NativeCodeLoader - Unable to load > native-hadoop library for your platform... using builtin-java classes > where applicable > 2011-12-19 20:14:01,451 INFO fetcher.Fetcher - Fetcher: finished at > 2011-12-19 20:14:01, elapsed: 00:00:03 > 2011-12-19 20:14:02,197 INFO parse.ParseSegment - ParseSegment: > starting at 2011-12-19 20:14:02 > 2011-12-19 20:14:02,198 INFO parse.ParseSegment - ParseSegment: > segment: /cdda/nutch/crawl/segments/20111219201355 > 2011-12-19 20:14:03,062 WARN util.NativeCodeLoader - Unable to load > native-hadoop library for your platform... using builtin-java classes > where applicable > 2 > ...is that enough for the fetch logs? It's all crawl/generator > messages after that. > > > I ran: > ./nutch freegen ../urls/ ./test-segments > ./nutch readseg -dump ./test-segments/ ./segment-output > > I got an error: > Exception in thread "main" > org.apache.hadoop.mapred.InvalidInputException: Input path does not > exist: file:/data/search/cdda/nutch-1.4/bin/test-segments/crawl_generate > Input path does not exist: > file:/data/search/cdda/nutch-1.4/bin/test-segments/crawl_fetch > Input path does not exist: > file:/data/search/cdda/nutch-1.4/bin/test-segments/crawl_parse > Input path does not exist: > file:/data/search/cdda/nutch-1.4/bin/test-segments/content > Input path does not exist: > file:/data/search/cdda/nutch-1.4/bin/test-segments/parse_data > Input path does not exist: > file:/data/search/cdda/nutch-1.4/bin/test-segments/parse_text > at > org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:1 > 90) at > org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileIn > putFormat.java:44) at > org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:20 > 1) at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:810) > at > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:781) > at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:730) at > org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1249) at > org.apache.nutch.segment.SegmentReader.dump(SegmentReader.java:225) at > org.apache.nutch.segment.SegmentReader.main(SegmentReader.java:564) > > So do I need to run the generator step in the middle? How is this > different than just doing a crawl? > > Thanks! > > -- Chris > > > > On Mon, Dec 19, 2011 at 3:22 PM, Markus Jelsma > > <[email protected]> wrote: > >> I'm a little confused -- should I set up a whole other instance of > >> nutch, crawldb, etc? > > > > Yes, i use clean instances for quick testing. Makes things easy > > sometimes. > > > >> Set the log to trace, I think this helps to tell why..... > >> > >> 2011-12-19 20:14:10,716 INFO crawl.FetchScheduleFactory - Using > >> FetchSchedule impl: > >> org.apache.nutch.crawl.DefaultFetchSchedule2011-12-19 20:14:10,716 > >> INFO crawl.AbstractFetchSchedule - defaultInterval=25920002011-12-19 > >> 20:14:10,716 INFO crawl.AbstractFetchSchedule - > >> maxInterval=77760002011-12-19 20:14:10,738 DEBUG crawl.Generator - > >> -shouldFetch rejected 'http://url/Alpha.docx', > >> fetchTime=1328213639379, curTime=13247576493782011-12-19 20:14:10,843 > >> INFO crawl.FetchScheduleFactory - Using FetchSchedule impl: > >> org.apache.nutch.crawl.DefaultFetchSchedule2011-12-19 20:14:10,843 > >> INFO crawl.AbstractFetchSchedule - defaultInterval=25920002011-12-19 > >> 20:14:10,843 INFO crawl.AbstractFetchSchedule - > >> maxInterval=77760002011-12-19 20:14:11,145 WARN crawl.Generator - > >> Generator: 0 records selected for fetching, exiting ... > > > > Now, this is the generator indeed but you need to fetcher logs. > > > >> Now, before I ran this I cleared the crawldb, linkdb & segments, but I > >> still got a rejected because it is before the next fetch time...why do > >> I get that? How do I set it up to always crawl all the docs? (Not > >> practical for production, but it's what I want when testing...) > > > > As i said, create segments using the freegen tool. It takes an input dir > > with seed files, just as your initial inject. Or can also inject files > > and give them meta data with a very low fetch interval so Nutch will > > crawl it each time, i usually take this approach in small tests. > > > > http://url<TAB>nutch.fetchInterval=10 > > > > The URL will be selected by the generator all the time because of this > > low fetch interval. > > > >> -- Chris > >> > >> > >> > >> On Mon, Dec 19, 2011 at 2:42 PM, Markus Jelsma > >> > >> <[email protected]> wrote: > >> >> > Hmm, the status db_gone prevents it from being indexed, of course. > >> >> > It is perfectly possible for the checkers to pass but that the > >> >> > fetcher will fail. There may have been an error and i remeber you > >> >> > using a proxy earlier, that's likely the problem here too. The > >> >> > checkers don't use proxy configurations. > >> >> > > >> >> > Check the logs to make sure. > >> >> > >> >> I cut out the proxy, and that let me get as far as I have now. > >> >> Having that in place prevents me from crawling the local > >> >> source...is there any way to be able to crawl both the inside & > >> >> outside networks? [separate issue, but something that I'll need this > >> >> to do] > >> > > >> > Not that i know of. You can use separate configs but this is tricky. > >> > Better use separate crawldb's configs etc. > >> > > >> >> > That's good. But remember, to pass it _must_ match regex prefixed > >> >> > by a +. This, however, is not your problem because in that case it > >> >> > wouldn't have ended up in the CrawlDB at all. > >> >> > >> >> I have two +'s that it should match on, including +.* > >> > > >> > That'll do. > >> > > >> >> > Check the fetcher output thoroughly. Grep around. You should find > >> >> > it. > >> >> > >> >> What exactly am I grepping for? > >> >> This is the block between the doc and the next one that it tries to > >> >> crawl.... > >> > > >> > Hmm, that looks fine but can still indicate a 404 because a 404 is not > >> > an error. Does debug say anything? You can set the level for the > >> > Fetcher in conf/log4j.properties. You can use freegen tool to > >> > generate a segments from some input text for tests. > >> > > >> >> 2011-12-19 18:42:19,538 INFO fetcher.Fetcher - fetching > >> >> http://url/Alpha.docx 2011-12-19 18:42:19,538 INFO fetcher.Fetcher - > >> >> Using queue mode : byHost 2011-12-19 18:42:19,539 INFO > >> >> fetcher.Fetcher - Using queue mode : byHost 2011-12-19 18:42:19,540 > >> >> INFO > >> >> fetcher.Fetcher - Using queue mode : byHost 2011-12-19 18:42:19,541 > >> >> INFO fetcher.Fetcher - Using queue mode : byHost 2011-12-19 > >> >> 18:42:19,541 INFO fetcher.Fetcher - Using queue mode : byHost > >> >> 2011-12-19 18:42:19,541 INFO fetcher.Fetcher - Using queue mode : > >> >> byHost 2011-12-19 18:42:19,542 INFO fetcher.Fetcher - Using queue > >> >> mode > >> >> > >> >> : byHost 2011-12-19 18:42:19,542 INFO fetcher.Fetcher - Using queue > >> >> > >> >> mode : byHost 2011-12-19 18:42:19,542 INFO fetcher.Fetcher - > >> >> Fetcher: throughput threshold: -1 > >> >> 2011-12-19 18:42:19,543 INFO fetcher.Fetcher - Fetcher: throughput > >> >> threshold retries: 5 > >> >> 2011-12-19 18:42:19,545 INFO http.Http - http.proxy.host = null > >> >> 2011-12-19 18:42:19,545 INFO http.Http - http.proxy.port = 8080 > >> >> 2011-12-19 18:42:19,545 INFO http.Http - http.timeout = 10000 > >> >> 2011-12-19 18:42:19,545 INFO http.Http - http.content.limit = -1 > >> >> 2011-12-19 18:42:19,545 INFO http.Http - http.agent = > >> >> crawler-nutch/Nutch-1.4 (Crawler; [email protected]) > >> >> 2011-12-19 18:42:19,545 INFO http.Http - http.accept.language = > >> >> en-us,en-gb,en;q=0.7,*;q=0.3 > >> >> 2011-12-19 18:42:20,545 INFO fetcher.Fetcher - -activeThreads=10, > >> >> spinWaiting=10, fetchQueues.totalSize=13 > >> >> 2011-12-19 18:42:21,548 INFO fetcher.Fetcher - -activeThreads=10, > >> >> spinWaiting=10, fetchQueues.totalSize=13 > >> >> 2011-12-19 18:42:22,550 INFO fetcher.Fetcher - -activeThreads=10, > >> >> spinWaiting=10, fetchQueues.totalSize=13 > >> >> 2011-12-19 18:42:23,552 INFO fetcher.Fetcher - -activeThreads=10, > >> >> spinWaiting=10, fetchQueues.totalSize=13 > >> >> 2011-12-19 18:42:24,554 INFO fetcher.Fetcher - -activeThreads=10, > >> >> spinWaiting=10, fetchQueues.totalSize=13 > >> >> > >> >> Thanks! > >> >> > >> >> --Chris
