On Thu, Nov 28, 2013 at 10:18:06AM +0100, Oleg Kalnichevski wrote: > On Wed, 2013-11-27 at 22:09 +0400, Dmitry Potapov wrote: > > Hello everyone, > > > > I've got OOM today and trying to understand if I'm using HttpAsyncClient > > wrong. > > > > So. I have HttpAsyncClient sending 100-150 requests per second to single > > remote > > host. Connection request timeout is 50 ms, connect timeout is 25 ms, socket > > timeout is 100 ms. > > In heap dump I've found about 200,000 LeaseRequest object. Since > > completedRequests is cleaned in the end of each lease(...) function call, > > I'm > > pretty sure that this is a leasingRequests. > > > > Looks like that remote host went down for a while and stopped requests > > processing. So, CPool started to accumulate lease requests and after several > > hours tons of accumulated LeaseRequest objects caused OOM. > > I've tried to find where leasingRequests must be cleaned and found that only > > function which performs cleanup by request deadline is > > AbstractNIOConnPool.processPendingRequests() which is called only by > > PoolingNHttpClientConnectionManager.closeIdleConnections() or > > .closeExpiredConnections(). > > > > So, my question is, should I call this functions periodically on my side, or > > there is another leasingRequests clean up mechanism which I didn't take into > > account? > > > > Dmitry, > > The only way to force the pooling connection manager to do a full linear > scan on the connection request queue is by calling > #closeExpiredConnections every once in a while (usually from a monitor > thread). > > However, the queue _should_ not get clogged with requests as long as > both connect and connection request timeouts are set to a positive > value. A connect timeout always causes eviction of expired requests from > the queue until a non-expired request is found. This strategy generally > should be good enough. > > Would it be possible for you to reproduce the issue with the context > logging for connection management / request execution turned on? > > Oleg > Oleg,
Thank you for your reply. Unfortunately I was unable to reproduce this issue on synthetic test. If I would be so (un)lucky to encounter this issue once again, I'll try to debug this on live system. -- Thanks, Dmitry Potapov > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
