If I do netstat, I see thousands of TCP connections in TIME_WAIT state, which
means they are have been closed and are clearing up.
So to limit the no of connection, I use a single instance of HttpClient in
the following way. I think I would have to watch out for connections that
might have been stale, but overall does this sound about right?
public class MyHttpClientFactory {
private static MyHttpClientFactory instance = new
HttpClientFactory();
private MultiThreadedHttpConnectionManager connectionManager;
private HttpClient client;
private HttpClientFactory() {
init();
}
public static HttpClientFactory getInstance() {
return instance;
}
public void init() {
connectionManager = new MultiThreadedHttpConnectionManager();
HttpConnectionManagerParams managerParams = new
HttpConnectionManagerParams();
managerParams.setMaxTotalConnections(1000);
connectionManager.setParams(managerParams);
client = new HttpClient(connectionManager);
}
public HttpClient getHttpClient() {
if (client != null) {
return client;
} else {
init();
return client;
}
}
}
MyHttpClientFactory.getInstance().getHttpClient();
Ken Krugler wrote:
>
> From looking at this code, it seems that you're creating a new
> connection manager for each time the thread is run.
>
> That seems kind of crazy. Normally you'd create a single HttpClient
> instance, using the thread safe connection manager with support for
> 1000 threads.
>
> -- Ken
>
> On May 24, 2010, at 4:16pm, CrystalCracker wrote:
>
>>
>> I have got 1000 dedicated Java threads where each thread polls a
>> corresponding url every one second.
>>
>> Simplified version of the code is like this
>>
>> MyThread extends() {
>> Node node;
>> MyThread(Node node) {
>> this.node = node;
>> }
>> public void run() {
>> Poller.poll()
>> }
>> }
>>
>> public class Poller {
>> public static Node poll(Node node) {
>> GetMethod method = null;
>> try {
>> HttpClient client = new HttpClient(new
>> SimpleHttpConnectionManager(true));
>> client.getParams().setParameter("http.socket.timeout", 120000);
>>
>> method = new GetMethod(node.getUrl());
>> method.getParams().setSoTimeout(8000);
>>
>> client.executeMethod(method);
>>
>> method.getResponseHeaders();
>> method.getResponseBody();
>> } catch (IOException ex) {
>> ex.printStackTrace();
>> } finally {
>> method.releaseConnection();
>> }
>> }
>> }
>>
>> The threads are run one second:
>>
>> for (int i=0; i <1000; i++) {
>> MyThread thread = threads.get(i) // threads is a static field
>> if(thread.isAlive()) {
>> // If the previous thread is still running, let it run.
>> } else {
>> thread.start();
>> }
>> }
>>
>> The problem is if I run the job every one second I get random
>> exceptions
>> like these:
>>
>> java.net.BindException: Address already in use
>> INFO httpclient.HttpMethodDirector: I/O exception
>> (java.net.BindException)
>> caught when processing request: Address already in use
>> INFO httpclient.HttpMethodDirector: Retrying request
>>
>> But if I run the job every 2 seconds or more, everything runs fine.
>>
>> I even tried shutting down the instance of
>> SimpleHttpConnectionManager()
>> using shutDown() with no effect.
>>
>> It looks like if when myThread has finished running, the HttpClient it
>> started hasn't shutdown even though I told it to release conenction
>> and/or
>> shutdown the manager.
>>
>> Any ideas?
>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/BindException-while-using-HttpClient-under-load-tp28662810p28662810.html
>> Sent from the HttpClient-User mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c w e b m i n i n g
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
>
>
--
View this message in context:
http://old.nabble.com/BindException-while-using-HttpClient-under-load-tp28662810p28670237.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
