By the way, I am still on HttpClient 3.1
CrystalCracker wrote:
>
> If I do netstat, I see thousands of TCP connections in TIME_WAIT state,
> which means they are have been closed and are clearing up.
>
> So to limit the no of connection, I use a single instance of HttpClient in
> the following way. I think I would have to watch out for connections that
> might have been stale, but overall does this sound about right?
>
> public class MyHttpClientFactory {
> private static MyHttpClientFactory instance = new
> HttpClientFactory();
> private MultiThreadedHttpConnectionManager connectionManager;
> private HttpClient client;
>
> private HttpClientFactory() {
> init();
> }
>
> public static HttpClientFactory getInstance() {
> return instance;
> }
>
> public void init() {
> connectionManager = new MultiThreadedHttpConnectionManager();
> HttpConnectionManagerParams managerParams = new
> HttpConnectionManagerParams();
> managerParams.setMaxTotalConnections(1000);
> connectionManager.setParams(managerParams);
> client = new HttpClient(connectionManager);
> }
>
> public HttpClient getHttpClient() {
> if (client != null) {
> return client;
> } else {
> init();
> return client;
> }
> }
> }
>
> MyHttpClientFactory.getInstance().getHttpClient();
>
>
>
>
> Ken Krugler wrote:
>>
>> From looking at this code, it seems that you're creating a new
>> connection manager for each time the thread is run.
>>
>> That seems kind of crazy. Normally you'd create a single HttpClient
>> instance, using the thread safe connection manager with support for
>> 1000 threads.
>>
>> -- Ken
>>
>> On May 24, 2010, at 4:16pm, CrystalCracker wrote:
>>
>>>
>>> I have got 1000 dedicated Java threads where each thread polls a
>>> corresponding url every one second.
>>>
>>> Simplified version of the code is like this
>>>
>>> MyThread extends() {
>>> Node node;
>>> MyThread(Node node) {
>>> this.node = node;
>>> }
>>> public void run() {
>>> Poller.poll()
>>> }
>>> }
>>>
>>> public class Poller {
>>> public static Node poll(Node node) {
>>> GetMethod method = null;
>>> try {
>>> HttpClient client = new HttpClient(new
>>> SimpleHttpConnectionManager(true));
>>> client.getParams().setParameter("http.socket.timeout", 120000);
>>>
>>> method = new GetMethod(node.getUrl());
>>> method.getParams().setSoTimeout(8000);
>>>
>>> client.executeMethod(method);
>>>
>>> method.getResponseHeaders();
>>> method.getResponseBody();
>>> } catch (IOException ex) {
>>> ex.printStackTrace();
>>> } finally {
>>> method.releaseConnection();
>>> }
>>> }
>>> }
>>>
>>> The threads are run one second:
>>>
>>> for (int i=0; i <1000; i++) {
>>> MyThread thread = threads.get(i) // threads is a static field
>>> if(thread.isAlive()) {
>>> // If the previous thread is still running, let it run.
>>> } else {
>>> thread.start();
>>> }
>>> }
>>>
>>> The problem is if I run the job every one second I get random
>>> exceptions
>>> like these:
>>>
>>> java.net.BindException: Address already in use
>>> INFO httpclient.HttpMethodDirector: I/O exception
>>> (java.net.BindException)
>>> caught when processing request: Address already in use
>>> INFO httpclient.HttpMethodDirector: Retrying request
>>>
>>> But if I run the job every 2 seconds or more, everything runs fine.
>>>
>>> I even tried shutting down the instance of
>>> SimpleHttpConnectionManager()
>>> using shutDown() with no effect.
>>>
>>> It looks like if when myThread has finished running, the HttpClient it
>>> started hasn't shutdown even though I told it to release conenction
>>> and/or
>>> shutdown the manager.
>>>
>>> Any ideas?
>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/BindException-while-using-HttpClient-under-load-tp28662810p28662810.html
>>> Sent from the HttpClient-User mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [email protected]
>>> For additional commands, e-mail: [email protected]
>>>
>>
>> --------------------------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://bixolabs.com
>> e l a s t i c w e b m i n i n g
>>
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [email protected]
>> For additional commands, e-mail: [email protected]
>>
>>
>>
>
>
--
View this message in context:
http://old.nabble.com/BindException-while-using-HttpClient-under-load-tp28662810p28670261.html
Sent from the HttpClient-User mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
