Markus,
Yes you are right FetcherThread does not use SolrJ by itself ,I am adding a call to Solr to save the data. I am concerned about the number of HttpClients being created,it.seems its creating a client per a document i am saving in Solr,thiscould be expected but I just want to confirm. Thanks. Sent from my HTC Inspire™ 4G on AT&T ----- Reply message ----- From: "Markus Jelsma" <[email protected]> To: "[email protected]" <[email protected]> Subject: Excessive HttpClient creation (Nutch 1.7 on Hadoop 2.2) Date: Mon, Dec 16, 2013 6:27 am Hi - How can this be in FetcherThread, Nutch does not use SolrJ in Fetcher. Do you have the entire Fetcher log? -----Original message----- > From:S.L <[email protected]> > Sent: Monday 16th December 2013 6:40 > To: [email protected] > Subject: Excessive HttpClient creation (Nutch 1.7 on Hadoop 2.2) > > Hi Folks, > > I am running Nutch 1.7 on Hadoop 2.2 and in the Hadoop logs for > FetcherThread, I see the following statements , which tells me that the > HttpCleints are being created per URL, is this correct assumption? Also > after a few fetches I also notice that the Hadoop job throws a OOM error , > please advise. > > 2013-12-15 23:47:31,921 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > 2013-12-15 23:47:31,931 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > 2013-12-15 23:47:31,932 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > 2013-12-15 23:47:32,034 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > 2013-12-15 23:47:32,034 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > 2013-12-15 23:47:32,040 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > 2013-12-15 23:47:32,187 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > 2013-12-15 23:47:32,214 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > 2013-12-15 23:47:32,250 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false > 2013-12-15 23:47:32,264 INFO [FetcherThread] > org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http > client, > config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false >
