Of course, forgot to mention that detail… :-) The client instanciation is done per thread as follow:
HttpClient client = new DefaultHttpClient( cm, params ); Where cm and params were initialized in the static block. dcheckoway wrote: > > You may want to pass cm to the DefaultHttpClient constructor... > > On Mon, Jan 23, 2012 at 2:36 PM, Dvora <[email protected]> wrote: > >> >> Hi, >> >> I would like to code an high performance web crawler using httpclient >> 4.1.2. >> In order to bring the machine to highest throughput, each crawling thread >> creating a DefaultHttpClient with a pool configured as follow (based on >> one >> of the examples): >> >> static >> { >> cm = new ThreadSafeClientConnManager(); >> cm.setMaxTotal( 50000 ); >> cm.setDefaultMaxPerRoute( Integer.MAX_VALUE ); >> >> HttpClient client = new DefaultHttpClient(); >> >> params = client.getParams(); >> >> HttpClientParams.setRedirecting( params, false ); >> HttpClientParams.setAuthenticating( params, true ); >> >> HttpConnectionParams.setSoTimeout( params, 30000 ); >> HttpConnectionParams.setConnectionTimeout( params, 30000 >> ); >> >> IdleConnectionEvictor connEvictor = new >> IdleConnectionEvictor( cm ); >> >> connEvictor.start(); >> } >> >> When running the application with lots of crawling threads, netstat show >> only 2k tcp connections in status ESTABLISHED. Is this expected >> considering >> maxTotsl = 50000? Are there other bottlenecks (OS level, etc.) blocking >> the >> application to reach more than 2k tcp connections? >> >> Thanks. >> >> >> -- >> View this message in context: >> http://old.nabble.com/Understanding-how-ThreadSafeClientConnManager-parameters-affect-number-of-tcp-connections-tp33190497p33190497.html >> Sent from the HttpClient-User mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > -- View this message in context: http://old.nabble.com/Understanding-how-ThreadSafeClientConnManager-parameters-affect-number-of-tcp-connections-tp33190497p33193107.html Sent from the HttpClient-User mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
