Hello,
We have developed software that uses HttpClient 3.1.
It has been working for 2 months. Recently though, we have been getting
'Maximum number of simultaneous requests exceeded' errors.
We are using a multi-threaded pool with 100 connections. Our application
never has more than 10 simultaneous requests.
Doing a netstat -anp returns the following:
tcp 0 0 ::ffff:10.52.128.52:44439 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:44527 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:44317 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:44391 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:44217 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:44270 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:44148 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:43795 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:43884 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:43564 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:43562 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:47963 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:47816 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:46347 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:46399 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:46823 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:46683 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59804 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59800 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59796 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59792 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59788 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59785 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59783 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59782 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59781 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59777 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59715 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59774 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59772 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59768 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59764 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59760 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59756 ::ffff:10.52.128.66:7003
TIME_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59673 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59704 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59699 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59640 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59641 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59639 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:59548 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 1 0 ::ffff:10.52.128.52:59485 ::ffff:10.52.128.66:7003
CLOSE_WAIT -
tcp 0 0 ::ffff:10.52.128.52:59409 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 1 0 ::ffff:10.52.128.52:59438 ::ffff:10.52.128.66:7003
CLOSE_WAIT -
tcp 0 0 ::ffff:10.52.128.52:58841 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:58683 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:58617 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:57806 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:57852 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:57665 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:57663 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:57441 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:57393 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:58294 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:58289 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:58192 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:57911 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:53029 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:53030 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:52758 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:52174 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:52117 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:52033 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:51880 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:50915 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:56746 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:56466 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:57177 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:57114 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:56968 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:56883 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:55717 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:55701 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:55495 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:55442 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:55848 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:54783 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:54596 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:54436 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:54453 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:55185 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:55022 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:54905 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:53440 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:54247 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:54200 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:53960 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:53961 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:53980 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:53783 ::ffff:10.52.128.66:7003
ESTABLISHED -
tcp 0 0 ::ffff:10.52.128.52:53784 ::ffff:10.52.128.66:7003
ESTABLISHED -
There are dozens of connections established with ip address
10.52.128.66:7003.
When we stop our program (thus shutting down the pool), the following errors
appear in our log:
2008-07-14 17:35:16 [172.19.2.17] MICO1 (Hotel) : HyM :
searchHotelByProvidersRequest Error genérico de protocolo HTTP
Socket closed
2008-07-14 17:40:44 [172.19.2.17] MICO1 (Hotel) : HyM :
searchHotelByProvidersRequest Error genérico de protocolo HTTP
Socket closed
2008-07-14 19:08:30 [172.19.2.16] MICO1 (Hotel) : HyM :
searchHotelByProvidersRequest Error genérico de protocolo HTTP
Socket closed
2008-07-14 18:16:48 [172.19.2.16] MICO1 (Hotel) : HyM :
searchHotelByProvidersRequest Error genérico de protocolo HTTP
Socket closed
2008-07-14 18:17:59 [172.19.2.16] MICO1 (Hotel) : HyM :
searchHotelByProvidersRequest Error genérico de protocolo HTTP
Socket closed
2008-07-14 17:58:34 [172.19.2.17] MICO1 (Hotel) : HyM :
searchHotelByProvidersRequest Error genérico de protocolo HTTP
Socket closed
2008-07-14 16:02:18 [172.19.2.16] MICO1 (Hotel) : HyM :
searchHotelByProvidersRequest Error genérico de protocolo HTTP
Socket closed
2008-07-14 13:52:46 [172.19.2.17] MICO1 (Hotel) : HyM :
searchHotelByProvidersRequest Error genérico de protocolo HTTP
Socket closed
2008-07-14 13:52:51 [172.19.2.16] MICO1 (Hotel) : HyM :
searchHotelByProvidersRequest Error genérico de protocolo HTTP
Socket closed
The error is Socked closed. These connections were established on the 14th
of July. The shutdown was done on the 15th of July (in other words, these
connections were open for more than a day). What we don't understand is how
it is possible that these connections can be open for so long when they have
a timeout of 26 seconds (which we assign in the PostMethod).
This is the code::
PostMethod objPost = null;
try
{
do something ...
objPost = new PostMethod(request.getRequestURL(objCall));
if (objCall.responseTimeout != 0)
objPost.getParams().setSoTimeout(objCall.responseTimeout);
objHttp.executeMethod(objPost);
}
catch (Exception e) { bla bla bla }
finally
{
if (objPost != null) objPost.releaseConnection();
}
It's as if the connections have been established, but the response timeout
is being ignored (objCall.responseTimeout = 26000)
The only thing out of the ordinary that we have seen is that on the 14th of
July there are many more Connection reset errors than there normally are
when compared with other days. But that's it. The number of requests is more
or less the same as it always is, as is the server response time where
requests are sent.
Has anyone ever seen anything like this? We're not sure where to look for
the problem. Can anyone give us a clue?
Thanks in advance,
Joan.
