there is a jira + a discussion on the mailing list on this. This is a synchronisation problem which has already been reported, patched but not yet committed. See https://issues.apache.org/jira/browse/NUTCH-719
J. 2009/11/27 MilleBii <mille...@gmail.com> > My fetch run is getting to the end now I have the following logs towards > the > end > > 2009-11-27 19:07:43,866 INFO fetcher.Fetcher - -activeThreads=100, > spinWaiting=100, fetchQueues.totalSize=12 > 2009-11-27 19:07:44,866 INFO fetcher.Fetcher - -activeThreads=100, > spinWaiting=100, fetchQueues.totalSize=12 > 2009-11-27 19:07:45,866 INFO fetcher.Fetcher - -activeThreads=100, > spinWaiting=100, fetchQueues.totalSize=12 > 2009-11-27 19:07:46,866 INFO fetcher.Fetcher - -activeThreads=100, > spinWaiting=100, fetchQueues.totalSize=12 > 2009-11-27 19:07:47,867 INFO fetcher.Fetcher - -activeThreads=100, > spinWaiting=100, fetchQueues.totalSize=12 > 2009-11-27 19:07:47,867 WARN fetcher.Fetcher - Aborting with 100 hung > threads. > > It was same on previous run, the fetchqueue is not "empty", what does it > mean ? Looks like there is 'problem' > > > 2009/11/27 Andrzej Bialecki <a...@getopt.org> > > > MilleBii wrote: > > > >> You mean map/reduce tasks ??? > >> > > > > Yes. > > > > > > Being in pseudo-distributed / single node I only have two maps during > the > >> fetch phase... so it would be back to the URLs distribution. > >> > > > > Well, yes, but my explanation is still valid. Which unfortunately doesn't > > change the situation. > > > > Next week I will be working on integrating the patches from Julien, and > if > > time permits I could perhaps start working on a speed monitoring to lock > out > > slow servers. > > > > > > -- > > Best regards, > > Andrzej Bialecki <>< > > ___. ___ ___ ___ _ _ __________________________________ > > [__ || __|__/|__||\/| Information Retrieval, Semantic Web > > ___|||__|| \| || | Embedded Unix, System Integration > > http://www.sigram.com Contact: info at sigram dot com > > > > > > > -- > -MilleBii- > -- DigitalPebble Ltd http://www.digitalpebble.com
