Hi - How can this be in FetcherThread, Nutch does not use SolrJ in Fetcher. Do
you have the entire Fetcher log?
-----Original message-----
> From:S.L <[email protected]>
> Sent: Monday 16th December 2013 6:40
> To: [email protected]
> Subject: Excessive HttpClient creation (Nutch 1.7 on Hadoop 2.2)
>
> Hi Folks,
>
> I am running Nutch 1.7 on Hadoop 2.2 and in the Hadoop logs for
> FetcherThread, I see the following statements , which tells me that the
> HttpCleints are being created per URL, is this correct assumption? Also
> after a few fetches I also notice that the Hadoop job throws a OOM error ,
> please advise.
>
> 2013-12-15 23:47:31,921 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> 2013-12-15 23:47:31,931 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> 2013-12-15 23:47:31,932 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> 2013-12-15 23:47:32,034 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> 2013-12-15 23:47:32,034 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> 2013-12-15 23:47:32,040 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> 2013-12-15 23:47:32,187 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> 2013-12-15 23:47:32,214 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> 2013-12-15 23:47:32,250 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
> 2013-12-15 23:47:32,264 INFO [FetcherThread]
> org.apache.solr.client.solrj.impl.HttpClientUtil: Creating new http
> client,
> config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
>
