On Sat, 2012-01-07 at 13:19 +0200, Vasile Alin wrote: > How about something like a BlockingQueue made for each route, with a > per-route limit? >
Just one Queue instance may be not enough to contain all the state information of a per route connection pool. The best way to find it out, though, is to put it in code. Fork HttpCore at GitHub and hack away ;-) Oleg > On 7 January 2012 00:20, sebb <seb...@gmail.com> wrote: > > On 6 January 2012 22:07, Ken Krugler <kkrugler_li...@transpac.com> wrote: > >> > >> On Jan 6, 2012, at 1:01pm, Oleg Kalnichevski wrote: > >> > >>> On Fri, 2012-01-06 at 11:06 -0500, Dan Checkoway wrote: > >>>> Hello, > >>>> > >>>> I have an app that needs to make concurrent HTTP requests to a web > >>>> service > >>>> using persistent (keepalive) connections. I'm using > >>>> ThreadSafeClientConnManager. I ran into a performance bottleneck, and I > >>>> believe I've pinpointed the issue... > >>>> > >>>> Affects Version(s): HttpCore 4.1.3, HttpClient 4.1.2 > >>>> > >>>> I construct my connection manager and client like this: > >>>> > >>>> connMgr = new > >>>> ThreadSafeClientConnManager(SchemeRegistryFactory.createDefault(), -1, > >>>> TimeUnit.MILLISECONDS); > >>>> connMgr.setMaxTotal(400); > >>>> connMgr.setDefaultMaxPerRoute(400); > >>>> > >>>> httpClient = new DefaultHttpClient(connMgr); > >>>> > >>>> Note that this app only talks to a single URI on a single server -- thus > >>>> defaultMaxPerRoute == maxTotal, which I think is correct...please let me > >>>> know if that's bad! > >>>> > >>>> Anyway, my app has a pool of 400 threads and generally performs quite > >>>> well. But when all 400 threads need a connection concurrently, > >>>> performance > >>>> suffers. I've narrowed it down to contention caused by blocking calls in > >>>> the connection manager. For example...a thread dump shows... > >>>> > >>>> About half my threads are "stuck" (well, not stuck, but slow & waiting) > >>>> here: > >>>> > >>>> "catalina-exec-347" daemon prio=10 tid=0x00007f3a54065000 nid=0x6b73 > >>>> waiting on condition [0x00007f3a29b9a000] > >>>> java.lang.Thread.State: WAITING (parking) > >>>> at sun.misc.Unsafe.park(Native Method) > >>>> - parking to wait for <0x00000006147c8318> (a > >>>> java.util.concurrent.locks.ReentrantLock$NonfairSync) > >>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > >>>> at > >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) > >>>> at > >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) > >>>> at > >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) > >>>> at > >>>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) > >>>> at > >>>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) > >>>> at > >>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.freeEntry(ConnPoolByRoute.java:438) > >>>> at > >>>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager.releaseConnection(ThreadSafeClientConnManager.java:276) > >>>> - locked <0x000000062048ebc8> (a > >>>> org.apache.http.impl.conn.tsccm.BasicPooledConnAdapter) > >>>> at > >>>> org.apache.http.impl.conn.AbstractClientConnAdapter.releaseConnection(AbstractClientConnAdapter.java:308) > >>>> - locked <0x000000062048ebc8> (a > >>>> org.apache.http.impl.conn.tsccm.BasicPooledConnAdapter) > >>>> at > >>>> org.apache.http.conn.BasicManagedEntity.releaseManagedConnection(BasicManagedEntity.java:181) > >>>> at > >>>> org.apache.http.conn.BasicManagedEntity.eofDetected(BasicManagedEntity.java:142) > >>>> at > >>>> org.apache.http.conn.EofSensorInputStream.checkEOF(EofSensorInputStream.java:211) > >>>> at > >>>> org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:139) > >>>> ... > >>>> > >>>> While the other half are "stuck" here: > >>>> > >>>> "catalina-exec-346" daemon prio=10 tid=0x00007f3a4c05d000 nid=0x6b72 > >>>> waiting on condition [0x00007f3a29c9b000] > >>>> java.lang.Thread.State: WAITING (parking) > >>>> at sun.misc.Unsafe.park(Native Method) > >>>> - parking to wait for <0x00000006147c8318> (a > >>>> java.util.concurrent.locks.ReentrantLock$NonfairSync) > >>>> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) > >>>> at > >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811) > >>>> at > >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842) > >>>> at > >>>> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178) > >>>> at > >>>> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186) > >>>> at > >>>> java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262) > >>>> at > >>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:337) > >>>> at > >>>> org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:300) > >>>> at > >>>> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:224) > >>>> at > >>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401) > >>>> at > >>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820) > >>>> at > >>>> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:941) > >>>> ... > >>>> > >>>> It's not a deadlock per se. It's just a bottleneck, and it is causing > >>>> very > >>>> high latency in my app. Below a certain threshold, i.e. when not all 400 > >>>> threads need a connection concurrently, things are fine. But when all > >>>> 400 > >>>> need a connection at once, that's when it gets painful. > >>>> > >>>> I'm wondering if it might be feasible to switch to using non-blocking > >>>> calls > >>>> for this, i.e. with ConcurrentHashMap and/or ConcurrentLinkedQueue, or > >>>> something of that nature? I haven't dived into the source code yet, so > >>>> don't slap me too hard if that suggestion was way out of line. :-) > >>>> > >>>> Do you have any suggestions, in terms of ways I might be able to work > >>>> around this bottleneck otherwise? > >>>> > >>>> Thanks! > >>>> > >>>> Dan Checkoway > >>> > >>> Hi Dan > >>> > >>> Yes, your observation is correct. The problem is that the connection > >>> pool is guarded by a global lock. Naturally if you have 400 threads > >>> trying to obtain a connection at about the same time all of them end up > >>> contending for one lock. The problem is that I can't think of a > >>> different way to ensure the max limits (per route and total) are > >>> guaranteed not to be exceeded. If anyone can think of a better algorithm > >>> please do let me know. What might be a possibility is creating a more > >>> lenient and less prone to lock contention issues implementation that may > >>> under stress occasionally allocate a few more connections than the max > >>> limits. > >> > >> I'd also run into a similar situation during web crawling, when I had 300+ > >> threads sharing one connection pool. > >> > >> Would it work to go for finer-grained locking, by using atomic counters to > >> track & enforce limits on per route/total connections? > > > > If the per-route limit is likely to be reached, it might help to have > > a lock per route. > > If the route limit has not been reached, only then grab the global lock. > > > > However this won't help unless the per-route limits are reached > > sufficiently often. > > > >> -- Ken > >> > >> -------------------------- > >> Ken Krugler > >> http://www.scaleunlimited.com > >> custom big data solutions & training > >> Hadoop, Cascading, Mahout & Solr > >> > >> > >> > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
