Hmm...Why dont you try to git clone a new nutch and then use the nutch only to see if you can crawl or not?
On Thu, Feb 19, 2015 at 10:09 PM, Nagarjun Pola <[email protected]> wrote: > Hmm I don't think the crawler is being blocked of politeness because I am > using the default Nutch configuration which is 1 request per second. > > And when I try to crawl with the sample URL by disabling Nutch plugin in > the Nutch-site.xml I can retrieve some links. > > The problem seems to be in the selenium plugin. Though Firefox pops > nothing is fetched. > > Best, > Nagarjun Pola > > > On Thu, Feb 19, 2015 at 10:05 PM, Jiaxin Ye <[email protected]> wrote: > >> Hi, my teammate is also suffering from this situation now and I >> encountered this situation last night. But I am able to crawl now almost >> without doing anything. The reason I may guess is that your crawler is >> blocked by the website because not being polite. At least I believe that's >> the reason why I got the same *Could not initialize class >> org.apache.http.impl.conn.* last night. I don't how to solve it, >> though..... Fortunate enough I think I am unbanned now, I guess? Hope it >> helps...... >> >> On Thu, Feb 19, 2015 at 9:44 PM, Nagarjun Pola <[email protected]> wrote: >> >>> I get the following error when tried with selenium. Firefox pops up >>> couple of times but fetches nothing. >>> >>> Can anyone help me on this issue? >>> >>> *-activeThreads=50, spinWaiting=49, fetchQueues.totalSize=1, >>> fetchQueues.getQueueCount=1* >>> >>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>* >>> >>> * maxThreads = 1* >>> >>> * inProgress = 1* >>> >>> * crawlDelay = 5000* >>> >>> * minCrawlDelay = 0* >>> >>> * nextFetchTime = 1424410799976* >>> >>> * now = 1424410803146* >>> >>> * 0. http://gcmd.gsfc.nasa.gov/ <http://gcmd.gsfc.nasa.gov/>* >>> >>> *fetch of >>> http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0 >>> <http://gcmd.gsfc.nasa.gov/KeywordSearch/Home.do?Portal=amd&MetadataType=0> >>> failed with: java.lang.NoClassDefFoundError: Could not initialize class >>> org.apache.http.impl.conn.ManagedHttpClientConnectionFactory* >>> >>> *-activeThreads=50, spinWaiting=50, fetchQueues.totalSize=1, >>> fetchQueues.getQueueCount=1* >>> >>> ** queue: http://gcmd.gsfc.nasa.gov <http://gcmd.gsfc.nasa.gov>* >>> >>> * maxThreads = 1* >>> >>> * inProgress = 0* >>> >>> * crawlDelay = 5000* >>> >>> * minCrawlDelay = 0* >>> >>> * nextFetchTime = 1424410808305* >>> >>> * now = 1424410804147* >>> >> >> >
