Markus, I am not very clear about the suggested route due to my unfamiliarity with Nutch. All I am doing in the Fetcher is intercepting the URL , extracting precise data and populating Solr using a Utility methos from a third party jar. What would I gain from doing this in a Reduce phase suggested by you instead of the apparent Map phase I am doing it in. I would really appreciate if you could clear that.
Also , after I started running Nutch1.7 in Hadoop 2.2 I started getting OOM exceptions , I have already increased my HADOOP_HEAPSIZE to 4GB on a 8GB laptop, this never happened when I was running in a lcoal mode and I was able to crawl large ( few 100Ks ) easily. Can you please also let me know what the issue might be ? Thanks again and much appreciated ! On Tue, Dec 17, 2013 at 4:09 AM, Markus Jelsma <[email protected]>wrote: > In Fetcher's protected ParseStatus output(Text key, CrawlDatum datum, > Content content, ProtocolStatus pstatus, int status, int outlinkDepth) you > need to output a NutchIndexAction object for every add or delete you get. > By comparing signature you can skip not_modified pages. You also need to > check for robots=noindex and 404 pages.You need to build up a > NutchDocument, pass it though indexing filters etc, just like the current > IndexerMapReduce. In FetcherOutputFormat you need to create an > IndexerOutputFormat just like the ParseOutputFormat that is already there, > and handle the incoming NutchIndexAction in RecordWriter.write(). > > -----Original message----- > > From:S.L <[email protected]> > > Sent: Monday 16th December 2013 18:00 > > To: [email protected] > > Subject: Re: Excessive HttpClient creation (Nutch 1.7 on Hadoop 2.2) > > > > Markus, > > > > Actually I started doing this after you replied to my queries few months > > ago , I did not face this issue when I was running Nutch in a local mode > , > > seems like this issue shows up when running in a deploy(single node > > cluster) mode. > > > > I will go ahead and change this to index the document in the > > FetcherOutputFormat class( would you please tell me the line number to > > insert th code at ?). > > > > However I was wondering if I would able to leverage the plugin mechanism > to > > do this and if there is any Solr plugin that takes the parsed text from > > the URL and indexes it based on some transformation that I do ? > > > > I really appreciate your help . > > > > Thanks. > > > > > > On Mon, Dec 16, 2013 at 10:08 AM, Markus Jelsma > > <[email protected]>wrote: > > > > > You have modified the Fetcher to index documents? In that case, you > should > > > index in the reducer (FetcherOutputFormat), not while mapping, and > reuse > > > the existing indexing code of SolrWriter. In any case, you should not > > > create a client per document. > > > > > > -----Original message----- > > > > From:S.L <[email protected]> > > > > Sent: Monday 16th December 2013 15:57 > > > > To: [email protected]; [email protected] > > > > Subject: Re: Excessive HttpClient creation (Nutch 1.7 on Hadoop 2.2) > > > > > > > > Markus, > > > > > > > > > > > > Yes you are right FetcherThread does not use SolrJ by itself ,I am > > > adding a call to Solr to save the data. I am concerned about the > number of > > > HttpClients being created,it.seems its creating a client per a > document i > > > am saving in Solr,thiscould be expected but I just want to confirm. > > > > > > > > Thanks. > > > > > > > > Sent from my HTC Inspire™ 4G on AT&T > > > > > > > > ----- Reply message ----- > > > > From: "Markus Jelsma" <[email protected]> > > > > To: "[email protected]" <[email protected]> > > > > Subject: Excessive HttpClient creation (Nutch 1.7 on Hadoop 2.2) > > > > Date: Mon, Dec 16, 2013 6:27 am > > > > > > > > > > > > Hi - How can this be in FetcherThread, Nutch does not use SolrJ in > > > Fetcher. Do you have the entire Fetcher log? > > > > > > > > -----Original message----- > > > > > From:S.L <[email protected]> > > > > > Sent: Monday 16th December 2013 6:40 > > > > > To: [email protected] > > > > > Subject: Excessive HttpClient creation (Nutch 1.7 on Hadoop 2.2) > > > > > > > > > > Hi Folks, > > > > > > > > > > I am running Nutch 1.7 on Hadoop 2.2 and in the Hadoop logs for > > > > > FetcherThread, I see the following statements , which tells me > that the > > > > > HttpCleints are being created per URL, is this correct assumption? > Also > > > > > after a few fetches I also notice that the Hadoop job throws a OOM > > > error , > > > > > please advise. > > > > > > > > > > 2013-12-15 23:47:31,921 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > 2013-12-15 23:47:31,931 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > 2013-12-15 23:47:31,932 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > 2013-12-15 23:47:32,034 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > 2013-12-15 23:47:32,034 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > 2013-12-15 23:47:32,040 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > 2013-12-15 23:47:32,187 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > 2013-12-15 23:47:32,214 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > 2013-12-15 23:47:32,250 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > 2013-12-15 23:47:32,264 INFO [FetcherThread] > > > > > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > > > > > client, > > > > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > > > > > > > > > > > > > > >
