date:20161017

RE: Memory leak in CloseableHttpAsyncClient?

2016-10-17 Thread Rob Griffin

I've raised the issue https://issues.apache.org/jira/browse/HTTPASYNC-116 

Regards,

Rob Griffin
Software Analyst, Spotlight on SQL Server 
Dell Software | R & D 
office +613 9811 8021 



-Original Message-
From: Oleg Kalnichevski [mailto:ol...@apache.org] 
Sent: Monday, 17 October 2016 11:58 PM
To: HttpClient User Discussion 
Subject: Re: Memory leak in CloseableHttpAsyncClient?

On Sun, 2016-10-16 at 21:54 +, Rob Griffin wrote:
> We are using HTTPAsynchClient to send data to our web site from a Java 
> client. 
> 
> We call CloseableHttpAsyncClient.execute() to execute HTTP PUTs at the rate 
> of several hundred per minute. Sometimes our web site slows down and does not 
> respond quickly enough and when this occurs the requests back up. We have 
> code that detects this and cancels the Future returned from the execute 
> method when the request has waited too long. If this happens too often the 
> application crashes with an out of memory error. 
> 
> Analysis of a dump showed that there were more 108,000 instances of 
> org.apache.http.nio.pool.LeaseRequest along with a similar number of 
> instances of other HTTP Client classes. Inspecting one of these objects 
> showed that its future variable is not cancelled but that by tracing though 
> the callback variables there is a cancelled Future further up the chain. That 
> cancelled Future object is one returned by execute because its callback is 
> one of our classes. To me it appears that the library is unaware that cancel 
> has been called on the Future returned by execute() and so keeps a reference 
> to it.
> 
> See 
> [url=https://postimg.org/image/j6zfdrquf/][img]https://s15.postimg.org
> /qa7atdwa3/Screen_Shot010.jpg[/img][/url][url=https://postimage.org/]i
> mage url[/url]
> 
> Regards,
> 
> Rob Griffin

Hi Rob

Currently cancellation of the response future does not immediately results in 
cancellation pending connection lease request or termination of the request 
execution.

Feel free to raise an improvement request in Jira.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

AW: Controlling releaseConnection

2016-10-17 Thread ecki

Hello,

Can younspecify why you need to delay it? Do wou want to make some kind of rate 
limit with this or optimize pipelining?

Gruss
Bernd
-- 
http://bernd.eckenfels.net
>From Win 10 Mobile

Von: Pellerin, Clement
Gesendet: Montag, 17. Oktober 2016 23:23
An: httpclient-users@hc.apache.org
Betreff: Controlling releaseConnection

We are using HttpClient 4.5.2

Our customer needs to delay the release of the connection until the response is 
fully processed.
They want to turn off the early automatic release of the connection and do it 
manually later.

This is the problematic code in MainClientExec
// check for entity, release connection if possible
final HttpEntity entity = response.getEntity();
if (entity == null || !entity.isStreaming()) {
// connection not needed and (assumed to be) in re-usable state
connHolder.releaseConnection();
return new HttpResponseProxy(response, null);
} else {
return new HttpResponseProxy(response, connHolder);
}

Can you suggest an approach to do this without duplicating all of 
MainClientExec.execute()?
What acceptable changes can we make to the HttpClient source code to make this 
easier?
I am willing to implement the change if there is interest.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Controlling releaseConnection

2016-10-17 Thread Shawn Heisey

On 10/17/2016 3:22 PM, Pellerin, Clement wrote:
> Our customer needs to delay the release of the connection until the response 
> is fully processed.
> They want to turn off the early automatic release of the connection and do it 
> manually later.
>
> This is the problematic code in MainClientExec
> // check for entity, release connection if possible
> final HttpEntity entity = response.getEntity();
> if (entity == null || !entity.isStreaming()) {
> // connection not needed and (assumed to be) in re-usable 
> state
> connHolder.releaseConnection();
> return new HttpResponseProxy(response, null);
> } else {
> return new HttpResponseProxy(response, connHolder);
> }

Mostly an end-user here, with no status to speak of in this project.  I
do have status on another Apache project that utilizes HttpClient, but I
don't know much about that part of the code.  I have written some
HttpClient code for a completely unrelated project of my own, but that
code is VERY simple.

When I read the code above, what I see is this: It only releases the
connection if the entity is nonexistent (null) or the entity is NOT a
type that uses streaming.

I will fully admit that my experience with HttpClient is limited, but I
think the chance is very small that the HttpComponents committers have
made a mistake here.  I think this particular code has probably been
discussed and examined, then ultimately validated as correct.  Here's
why I think they didn't make a mistake:

If the entity object is null, then the response probably doesn't HAVE an
entity (response body), so it will be entirely self-contained,
consisting of headers only, and the connection doesn't have anything
further to send.  If the entity exists but doesn't utilize streaming,
then I think it's likely that the entity was received in its entirety
and has been incorporated into the response object already, and once
again, the connection isn't needed.  If my limited understanding of
non-streaming entities is correct, they have the potential to be very
dangerous from a memory consumption perspective, and my own usage of
HttpClient (where I did not set anything related to the entity type)
suggests that streaming entities are used by default.

Restating in another way:  In the first situation that results in a
released connection, there's nothing to consume, you just need the
response object that you already have.  In the second situation, the
entity you will consume is probably already available within the
response object and doesn't need the connection.  The comment on the
release call in the code quoted above implies that this is how things work.

In these situations, why do you need the connection to stick around?  I
think it can't do anything else that's useful for that request.  I would
imagine that if the connection utilizes keepalive/pipelining, that it
will typically remain open after release and can be utilized again for a
different request.

Someone with more direct knowledge of HttpClient's internal
implementation will need to confirm whether or not I'm correct in what
I've written.  My understanding could be wrong.

Thanks,
Shawn

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Disabling pooling of PoolingConnectionManager

2016-10-17 Thread Murat Balkan

I see. I think that also means that I cannot share the ClosableHttpClient
instance among multiple threads as each client can refer to one connection
manager instance.

Can connectionreusestrategy be used so that the pooling connection manager
will always return a new connection regardless of the route provided?

Regards.
Murat

On Mon, Oct 17, 2016 at 5:05 PM, Bhowmik, Bindul 
wrote:

> Murat,
>
> On Mon, Oct 17, 2016 at 12:58 PM, Murat Balkan  wrote:
> > Hi Bindul,
> > Thanks for the answer.
> > I was thinking that using a shared connection manager will increase the
> > performance. What will be the implications of reusing the same
> > BasicHttpClientConnectionManager instance?
>
> If you see the documentation for the BasicHttpClientConnectionManager
> [1], you will see that it only maintains one active connection. If you
> share the the instance, your requests will be waiting for the
> connection to be available and that will be your bottleneck.
>
> I would also recommend reading the connection management section of
> the Http Client documentation [2]
>
> - Bindul
>
> [1] http://hc.apache.org/httpcomponents-client-ga/
> httpclient/apidocs/org/apache/http/impl/conn/
> BasicHttpClientConnectionManager.html
> [2] http://hc.apache.org/httpcomponents-client-4.5.x/
> tutorial/html/connmgmt.html
>
> > Regards,
> > Murat
> >
> > On Mon, Oct 17, 2016 at 2:31 PM, Bhowmik, Bindul <
> bindulbhow...@gmail.com>
> > wrote:
> >
> >> Murat,
> >>
> >> On Mon, Oct 17, 2016 at 11:12 AM, Murat Balkan 
> wrote:
> >> > Hi,
> >> >
> >> > We are using PoolingHttpClientConnectionManager to open up
> connections
> >> to
> >> > multiple URL's in different threads (via different HttpGet objects).
> >> >
> >> > The only reason we are using the PoolingHttpClientConnectionManager
> is
> >> its'
> >> > performance in multi-thread environments (as suggested by the
> >> > documentation).
> >> >
> >> > However, we are not interested in the actual "pooling" functionality.
> >> > That's to say, we want to open up a brand new connection even if the
> >> route
> >> > is the same.
> >>
> >> The performance enhancements you achieve from
> >> PoolingHttpClientConnectionManager are due to its connection pooling
> >> feature, that saves you to cost of establishing the connection when
> >> another request goes to the same route.
> >>
> >> >
> >> > How can we achieve this? We tried to set maxPerroute to 1 but it
> seems it
> >> > is not the correct way.
> >>
> >> I have not tested, but setting maxPerRoute to 1 would degrade
> >> performance for you as you will have a number of Http clients waiting
> >> for the single connection.
> >>
> >> If you do not want to use pooled connections, you can use
> >> BasicHttpClientConnectionManager and not share it.
> >>
> >> >
> >> > Regards,
> >> > Murat
> >>
> >> -
> >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >>
> >>
> >
> >
> > --
> > Murat Balkan
>
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
>


-- 
Murat Balkan

Re: Disabling pooling of PoolingConnectionManager

2016-10-17 Thread Bhowmik, Bindul

Murat,

On Mon, Oct 17, 2016 at 8:11 PM, Murat Balkan  wrote:
> I see. I think that also means that I cannot share the ClosableHttpClient
> instance among multiple threads as each client can refer to one connection
> manager instance.
>
> Can connectionreusestrategy be used so that the pooling connection manager
> will always return a new connection regardless of the route provided?

I did not think about that, guess you could use the NoConnectionReuseStrategy

- Bindul

>
> Regards.
> Murat
>
> On Mon, Oct 17, 2016 at 5:05 PM, Bhowmik, Bindul 
> wrote:
>
>> Murat,
>>
>> On Mon, Oct 17, 2016 at 12:58 PM, Murat Balkan  wrote:
>> > Hi Bindul,
>> > Thanks for the answer.
>> > I was thinking that using a shared connection manager will increase the
>> > performance. What will be the implications of reusing the same
>> > BasicHttpClientConnectionManager instance?
>>
>> If you see the documentation for the BasicHttpClientConnectionManager
>> [1], you will see that it only maintains one active connection. If you
>> share the the instance, your requests will be waiting for the
>> connection to be available and that will be your bottleneck.
>>
>> I would also recommend reading the connection management section of
>> the Http Client documentation [2]
>>
>> - Bindul
>>
>> [1] http://hc.apache.org/httpcomponents-client-ga/
>> httpclient/apidocs/org/apache/http/impl/conn/
>> BasicHttpClientConnectionManager.html
>> [2] http://hc.apache.org/httpcomponents-client-4.5.x/
>> tutorial/html/connmgmt.html
>>
>> > Regards,
>> > Murat
>> >
>> > On Mon, Oct 17, 2016 at 2:31 PM, Bhowmik, Bindul <
>> bindulbhow...@gmail.com>
>> > wrote:
>> >
>> >> Murat,
>> >>
>> >> On Mon, Oct 17, 2016 at 11:12 AM, Murat Balkan 
>> wrote:
>> >> > Hi,
>> >> >
>> >> > We are using PoolingHttpClientConnectionManager to open up
>> connections
>> >> to
>> >> > multiple URL's in different threads (via different HttpGet objects).
>> >> >
>> >> > The only reason we are using the PoolingHttpClientConnectionManager
>> is
>> >> its'
>> >> > performance in multi-thread environments (as suggested by the
>> >> > documentation).
>> >> >
>> >> > However, we are not interested in the actual "pooling" functionality.
>> >> > That's to say, we want to open up a brand new connection even if the
>> >> route
>> >> > is the same.
>> >>
>> >> The performance enhancements you achieve from
>> >> PoolingHttpClientConnectionManager are due to its connection pooling
>> >> feature, that saves you to cost of establishing the connection when
>> >> another request goes to the same route.
>> >>
>> >> >
>> >> > How can we achieve this? We tried to set maxPerroute to 1 but it
>> seems it
>> >> > is not the correct way.
>> >>
>> >> I have not tested, but setting maxPerRoute to 1 would degrade
>> >> performance for you as you will have a number of Http clients waiting
>> >> for the single connection.
>> >>
>> >> If you do not want to use pooled connections, you can use
>> >> BasicHttpClientConnectionManager and not share it.
>> >>
>> >> >
>> >> > Regards,
>> >> > Murat
>> >>
>> >> -
>> >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
>> >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> > Murat Balkan
>>
>> -
>> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
>> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>>
>>
>
>
> --
> Murat Balkan

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Disabling pooling of PoolingConnectionManager

2016-10-17 Thread Bhowmik, Bindul

Murat,

On Mon, Oct 17, 2016 at 11:12 AM, Murat Balkan  wrote:
> Hi,
>
> We are using PoolingHttpClientConnectionManager to open up connections to
> multiple URL's in different threads (via different HttpGet objects).
>
> The only reason we are using the PoolingHttpClientConnectionManager is its'
> performance in multi-thread environments (as suggested by the
> documentation).
>
> However, we are not interested in the actual "pooling" functionality.
> That's to say, we want to open up a brand new connection even if the route
> is the same.

The performance enhancements you achieve from
PoolingHttpClientConnectionManager are due to its connection pooling
feature, that saves you to cost of establishing the connection when
another request goes to the same route.

>
> How can we achieve this? We tried to set maxPerroute to 1 but it seems it
> is not the correct way.

I have not tested, but setting maxPerRoute to 1 would degrade
performance for you as you will have a number of Http clients waiting
for the single connection.

If you do not want to use pooled connections, you can use
BasicHttpClientConnectionManager and not share it.

>
> Regards,
> Murat

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Memory leak in CloseableHttpAsyncClient?

2016-10-17 Thread Oleg Kalnichevski

On Sun, 2016-10-16 at 21:54 +, Rob Griffin wrote:
> We are using HTTPAsynchClient to send data to our web site from a Java 
> client. 
> 
> We call CloseableHttpAsyncClient.execute() to execute HTTP PUTs at the rate 
> of several hundred per minute. Sometimes our web site slows down and does not 
> respond quickly enough and when this occurs the requests back up. We have 
> code that detects this and cancels the Future returned from the execute 
> method when the request has waited too long. If this happens too often the 
> application crashes with an out of memory error. 
> 
> Analysis of a dump showed that there were more 108,000 instances of 
> org.apache.http.nio.pool.LeaseRequest along with a similar number of 
> instances of other HTTP Client classes. Inspecting one of these objects 
> showed that its future variable is not cancelled but that by tracing though 
> the callback variables there is a cancelled Future further up the chain. That 
> cancelled Future object is one returned by execute because its callback is 
> one of our classes. To me it appears that the library is unaware that cancel 
> has been called on the Future returned by execute() and so keeps a reference 
> to it.
> 
> See 
> [url=https://postimg.org/image/j6zfdrquf/][img]https://s15.postimg.org/qa7atdwa3/Screen_Shot010.jpg[/img][/url][url=https://postimage.org/]image
>  url[/url] 
> 
> Regards,
> 
> Rob Griffin

Hi Rob

Currently cancellation of the response future does not immediately
results in cancellation pending connection lease request or termination
of the request execution.

Feel free to raise an improvement request in Jira.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Disabling pooling of PoolingConnectionManager

2016-10-17 Thread Murat Balkan

Hi,

We are using PoolingHttpClientConnectionManager to open up connections to
multiple URL's in different threads (via different HttpGet objects).

The only reason we are using the PoolingHttpClientConnectionManager is its'
performance in multi-thread environments (as suggested by the
documentation).

However, we are not interested in the actual "pooling" functionality.
That's to say, we want to open up a brand new connection even if the route
is the same.

How can we achieve this? We tried to set maxPerroute to 1 but it seems it
is not the correct way.

Regards,
Murat

Re: Disabling pooling of PoolingConnectionManager

2016-10-17 Thread Murat Balkan

Hi Bindul,
Thanks for the answer.
I was thinking that using a shared connection manager will increase the
performance. What will be the implications of reusing the same
BasicHttpClientConnectionManager instance?
Regards,
Murat

On Mon, Oct 17, 2016 at 2:31 PM, Bhowmik, Bindul 
wrote:

> Murat,
>
> On Mon, Oct 17, 2016 at 11:12 AM, Murat Balkan  wrote:
> > Hi,
> >
> > We are using PoolingHttpClientConnectionManager to open up connections
> to
> > multiple URL's in different threads (via different HttpGet objects).
> >
> > The only reason we are using the PoolingHttpClientConnectionManager is
> its'
> > performance in multi-thread environments (as suggested by the
> > documentation).
> >
> > However, we are not interested in the actual "pooling" functionality.
> > That's to say, we want to open up a brand new connection even if the
> route
> > is the same.
>
> The performance enhancements you achieve from
> PoolingHttpClientConnectionManager are due to its connection pooling
> feature, that saves you to cost of establishing the connection when
> another request goes to the same route.
>
> >
> > How can we achieve this? We tried to set maxPerroute to 1 but it seems it
> > is not the correct way.
>
> I have not tested, but setting maxPerRoute to 1 would degrade
> performance for you as you will have a number of Http clients waiting
> for the single connection.
>
> If you do not want to use pooled connections, you can use
> BasicHttpClientConnectionManager and not share it.
>
> >
> > Regards,
> > Murat
>
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>
>


-- 
Murat Balkan

Re: Disabling pooling of PoolingConnectionManager

2016-10-17 Thread Bhowmik, Bindul

Murat,

On Mon, Oct 17, 2016 at 12:58 PM, Murat Balkan  wrote:
> Hi Bindul,
> Thanks for the answer.
> I was thinking that using a shared connection manager will increase the
> performance. What will be the implications of reusing the same
> BasicHttpClientConnectionManager instance?

If you see the documentation for the BasicHttpClientConnectionManager
[1], you will see that it only maintains one active connection. If you
share the the instance, your requests will be waiting for the
connection to be available and that will be your bottleneck.

I would also recommend reading the connection management section of
the Http Client documentation [2]

- Bindul

[1] 
http://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/conn/BasicHttpClientConnectionManager.html
[2] http://hc.apache.org/httpcomponents-client-4.5.x/tutorial/html/connmgmt.html

> Regards,
> Murat
>
> On Mon, Oct 17, 2016 at 2:31 PM, Bhowmik, Bindul 
> wrote:
>
>> Murat,
>>
>> On Mon, Oct 17, 2016 at 11:12 AM, Murat Balkan  wrote:
>> > Hi,
>> >
>> > We are using PoolingHttpClientConnectionManager to open up connections
>> to
>> > multiple URL's in different threads (via different HttpGet objects).
>> >
>> > The only reason we are using the PoolingHttpClientConnectionManager is
>> its'
>> > performance in multi-thread environments (as suggested by the
>> > documentation).
>> >
>> > However, we are not interested in the actual "pooling" functionality.
>> > That's to say, we want to open up a brand new connection even if the
>> route
>> > is the same.
>>
>> The performance enhancements you achieve from
>> PoolingHttpClientConnectionManager are due to its connection pooling
>> feature, that saves you to cost of establishing the connection when
>> another request goes to the same route.
>>
>> >
>> > How can we achieve this? We tried to set maxPerroute to 1 but it seems it
>> > is not the correct way.
>>
>> I have not tested, but setting maxPerRoute to 1 would degrade
>> performance for you as you will have a number of Http clients waiting
>> for the single connection.
>>
>> If you do not want to use pooled connections, you can use
>> BasicHttpClientConnectionManager and not share it.
>>
>> >
>> > Regards,
>> > Murat
>>
>> -
>> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
>> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
>>
>>
>
>
> --
> Murat Balkan

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Controlling releaseConnection

2016-10-17 Thread Pellerin, Clement

We are using HttpClient 4.5.2

Our customer needs to delay the release of the connection until the response is 
fully processed.
They want to turn off the early automatic release of the connection and do it 
manually later.

This is the problematic code in MainClientExec
// check for entity, release connection if possible
final HttpEntity entity = response.getEntity();
if (entity == null || !entity.isStreaming()) {
// connection not needed and (assumed to be) in re-usable state
connHolder.releaseConnection();
return new HttpResponseProxy(response, null);
} else {
return new HttpResponseProxy(response, connHolder);
}

Can you suggest an approach to do this without duplicating all of 
MainClientExec.execute()?
What acceptable changes can we make to the HttpClient source code to make this 
easier?
I am willing to implement the change if there is interest.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: Memory leak in CloseableHttpAsyncClient?

AW: Controlling releaseConnection

Re: Controlling releaseConnection

Re: Disabling pooling of PoolingConnectionManager

Re: Disabling pooling of PoolingConnectionManager

Re: Disabling pooling of PoolingConnectionManager

Re: Memory leak in CloseableHttpAsyncClient?

Disabling pooling of PoolingConnectionManager

Re: Disabling pooling of PoolingConnectionManager

Re: Disabling pooling of PoolingConnectionManager

Controlling releaseConnection

11 matches

Site Navigation

Mail list logo

Footer information