I'm trying to track down a performance/resource consumption problem which seems
to involve HttpClient. After running for few days with low to medium traffic
(with low concurrency), my application stops responding. A thread dump reveals
many thousands of threads (between 3K and 4K) waiting for the following
condition (see below). The VM eventually recovers, the thread count goes wayyy
down (around 100 or so, which what I'd expect with the kind of traffic we see.)
"search" prio=10 tid=0x0000002ae20f6000 nid=0x38d4 waiting on condition
[0x0000002b35504000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000002aa9aa4dc0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
at
org.apache.http.impl.conn.tsccm.WaitingThread.await(WaitingThread.java:159)
at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:339)
at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:238)
at
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:175)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:324)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:555)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:465)
... (truncated)
The connection parameters are like so:
ConnManagerParams.setMaxTotalConnections(params, 400);
ConnManagerParams.setMaxConnectionsPerRoute(params, new
ConnPerRouteBean(400));
HttpProtocolParams.setVersion(params, HttpVersion.HTTP_1_1);
ConnManagerParams.setTimeout(params, 10);
sharedConnectionManager = new ThreadSafeClientConnManager(params,
schemeRegistry);
I've turned on logging, and I've seen the following counters go pretty high:
org.apache.http.impl.conn.tsccm.ConnPoolByRoute Total issued connections:
300
org.apache.http.impl.conn.tsccm.ConnPoolByRoute Total allocated connection:
300 out of 400
org.apache.http.impl.conn.tsccm.ConnPoolByRoute Total connections kept
alive: 46
My application uses multiple instances of HttpClient, about a dozen or so, each
configured as above.
Question: what is "total issued connections", and how does it compare to "total
allocated connections"?
Environment:
HttpClient 4.0-beta1
HttpCore 4.0-beta2
Java HotSpot(TM) 64-Bit Server VM (16.3-b01 mixed mode):
Linux 2.6.9-78.ELsmp
I know this isn't much to go on, and I apologize. But I'm hoping it may ring a
bell with someone who's got an idea why this is happening. Kind of a reality
check too. Does any of this smell funny?
Thanks!
--Ren
