You set a time limit for generate. These urls are hitted by time limit. Can you share your nutch-site.xml
If you change nutch-defualt.xml please change with orginal version before. 24 Mar 2014 20:25 tarihinde "glumet" <[email protected]> yazdı: > Hi, > I generated batchid 1111 with topN 100, in Hbase I have 100 rows with > generate mark with the same value, but fetch is not working, after run > nutch* fetch 1111* I got no progres for this generated urls, in database > is > no row with fetch mark and pages are unfetched. > I write debug prints in nutch source codes and build it. FetcherMapper > works > fine, if generate mark is 1111 writes it in cotext (/context.write(new > IntWritable(random.nextInt(65536)), > new > FetchEntry(context.getConfiguration(), key, page));)/, but in > FetcherReducer is nothing /*fit = fetchQueues.getFetchItem(); */ produces > null /(org.apache.nutch.fetcher.FetcherReducer$FetcherThread getFetchItem : > null/ > ) > It looks like the queues are empty, but why? What could be problem here? > Thanks in advance > > /FetcherJob: starting > FetcherJob: batchId: 1111 > FetcherJob: threads: 10 > FetcherJob: parsing: false > FetcherJob: resuming: false > FetcherJob : timelimit set for : 1395683736716 > Using queue mode : byHost > Fetcher: threads: 10 > QueueFeeder finished: total 0 records. Hit by time limit :100 > -finishing thread FetcherThread0, activeThreads=0 > -finishing thread FetcherThread2, activeThreads=1 > -finishing thread FetcherThread1, activeThreads=0 > -finishing thread FetcherThread4, activeThreads=0 > -finishing thread FetcherThread3, activeThreads=0 > -finishing thread FetcherThread5, activeThreads=0 > -finishing thread FetcherThread6, activeThreads=0 > -finishing thread FetcherThread7, activeThreads=0 > -finishing thread FetcherThread8, activeThreads=0 > Fetcher: throughput threshold: -1 > -finishing thread FetcherThread9, activeThreads=0 > Fetcher: throughput threshold sequence: 5 > 0/0 spinwaiting/active, 0 pages, 0 errors, 0.0 0 pages/s, 0 0 kb/s, 0 URLs > in 0 queues > -activeThreads=0 > FetcherJob: done/ > > HBase shell: return 100 rows only for generate mark > scan 'webpage_webpage', {COLUMNS => ['mk:_gnmrk_','mk:_ftcmrk_']} > > > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Nutch-2-1-fetching-is-not-working-maybe-broken-generate-tp4123813p4126652.html > Sent from the Nutch - User mailing list archive at Nabble.com. >
