QUESTION ABOUT COOKIES

[Joan Balagueró]

Hello,

 

I’ve a little question about cookies.

 

I’ve a MultiThreadedHttpConnectionManager, and I get an HttpClient instance
from it. This httpclient object IS SHARED by all threads of my servlet (it
means, I only have 1 instance of httpclient). I set the RFC-2109 in this
way:

 

HttpClient objHttp = new HttpClient(new
MultiThreadedHttpConnectionManager());

objHttp.getParams().setCookiePolicy(CookiePolicy.RFC_2109);  

 

 

Now I want to set cookies to this httpclient using httpstate. The following
piece of code is executed by every thread (every request) of my servlet:

 

HttpState initialState = new HttpState();

initialState.addCookie(new
org.apache.commons.httpclient.Cookie(value_for_this_request, ……..));

objHttp.setState(initialState);

 

 

The question is: doing this, am I setting this cookie for all threads or
just for the current thread that is executing this code? Obviously, I want
to set this cookie just for the current thread, not for all. Is this
correct? And if not, how should I do it? Maybe set the cookies at method
level using method.setRequestHeader and setting the “Cookie” header and its
value as string?

 

And the following question is: this request is sent to another servlet, that
reads it. But if I examine the HttpServletRequest, I always get null for
cookies attribute. It’s like httpclient is not including the cookie in the
request I’m sending to the remote servlet. But why? Is it not enough
specifying the cookie policity at httclient params level?

 

 

Thanks a lot,

 

Joan. 

 

 

RE: Question About Cookies

[Joan Balagueró]

Hello,

Thanks for your response.

Now I know how to send a cookie just related to one thread. But how to
receive a cookie from my remote server?

I was using : Cookie[] auxCookies = objHttp.getState().getCookies();

Now I know this is incorrect. I suppose that I must get cookies from the
Post method. But how? Should I use the
method.getResponseHeader(Set-Cookie)? Or there is another way?


Thanks,

Joan.

 

-Mensaje original-
De: Roland Weber [mailto:[EMAIL PROTECTED] 
Enviado el: miércoles, 05 de marzo de 2008 19:38
Para: HttpClient User Discussion
Asunto: Re: Question About Cookies

Hello Joan,

there is no need to shout.

 The question is: doing this, am I setting this cookie for all threads or
 just for the current thread that is executing this code?

For all threads.

 Obviously, I want
 to set this cookie just for the current thread, not for all. Is this
 correct? And if not, how should I do it?

You should have a separate HttpState object for each thread
instead of using the default HttpState object for all of them:
http://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/Ht
tpClient.html#executeMethod(org.apache.commons.httpclient.HostConfiguration,
%20org.apache.commons.httpclient.HttpMethod,%20org.apache.commons.httpclient
.HttpState)

 And the following question is: this request is sent to another servlet,
that
 reads it. But if I examine the HttpServletRequest, I always get null for
 cookies attribute. It’s like httpclient is not including the cookie in the
 request I’m sending to the remote servlet.

Check the domain and path attributes of your cookie.
Enable wire logging to see wether the cookie is sent or not.
http://hc.apache.org/httpclient-3.x/logging.html

hope that helps,
   Roland


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RE: Pool running out of connections

[Joan Balagueró]

Hello,

Thanks for your response. I suppose that this eviction policy that you are
saying is achieved by using IdleConnectionTimeoutThread. Am I right?

Joan.
 
 

-Mensaje original-
De: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] 
Enviado el: miércoles, 16 de julio de 2008 18:03
Para: HttpClient User Discussion
Asunto: Re: Pool running out of connections

On Wed, 2008-07-16 at 09:03 +0200, Joan Balagueró wrote:
 Hello,
 
  
 
 We have developed software that uses HttpClient 3.1. 
 
  
 
 It has been working for 2 months. Recently though, we have been getting
 'Maximum number of simultaneous requests exceeded' errors.
 
  
 
 We are using a multi-threaded pool with 100 connections. Our application
 never has more than 10 simultaneous requests.
 
  
 
 The error is “Socked closed”. These connections were established on the
14th
 of July. The shutdown was done on the 15th of July (in other words, these
 connections were open for more than a day). What we don't understand is
how
 it is possible that these connections can be open for so long when they
have
 a timeout of 26 seconds (which we assign in the PostMethod).
 
  

Socket timeout applies to _active_ read operations only. Idle
connections cannot time out, as they are not blocked in a read
operation.  

You should implement an eviction policy to close connections that have
been idle for too long.

Hope this helps

Oleg


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RE: Pool running out of connections

[Joan Balagueró]

Hello,

We're using IdleConnectionTimeoutThread.

The code is the following:

this.objHttp = new HttpClient(new MultiThreadedHttpConnectionManager());
this.ictt = new IdleConnectionTimeoutThread();
this.ictt.addConnectionManager(this.objHttp.getHttpConnectionManager());
this.ictt.setTimeoutInterval(1);
this.ictt.setConnectionTimeout(2);

If I'm right, thread that checks for idle connections starts every 10 seconds 
and removes all idle connections with no activity for more than 20 seconds.

Is it OK? And if it's, how can I find out that idle connections are really 
being removed?

I could see when we send a request with httpclient, the connection goes to 
ESTABLISHED state, and when we release this connection, it goes to CLOSE_WAIT 
state, and remains in this state for several minutes.

Is an idle connection a connection in CLOSE_WAIT state? If httpclient removes 
one idle connection, must I suppose that the number of CLOSE_WAIT connections 
are decrement in 1? (looking at them with the netstat linux statement).

Basicly, now my problem is I need to check that IdleConnectionTimeoutThread  
is really removing idle connections, but I'm not sure how to do this.


Another thing you told me was that I could use the method 
closeIdleConnections(timeout). I suppose that the right way to do this is 
calling this method after releaseConnection. Is this the correct way? And if 
it's, is it optimal to call closeIdleConnections after every 
releaseConnection?


Thanks in advance,

Joan.


-Mensaje original-
De: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] 
Enviado el: viernes, 18 de julio de 2008 0:06
Para: HttpClient User Discussion
Asunto: RE: Pool running out of connections

On Thu, 2008-07-17 at 17:41 +0200, Joan Balagueró wrote:
 Hello,
 
 Thanks for your response. I suppose that this eviction policy that you are
 saying is achieved by using IdleConnectionTimeoutThread. Am I right?
 
 Joan.

Yes, you are. You do not necessarily need a special thread to achieve
that. Just call HttpConnectionManager#closeIdleConnections method after
a period inactivity to purge stale connections:

http://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/HttpConnectionManager.html#closeIdleConnections(long)

Oleg

  
 
 
 -Mensaje original-
 De: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] 
 Enviado el: miércoles, 16 de julio de 2008 18:03
 Para: HttpClient User Discussion
 Asunto: Re: Pool running out of connections
 
 On Wed, 2008-07-16 at 09:03 +0200, Joan Balagueró wrote:
  Hello,
  
   
  
  We have developed software that uses HttpClient 3.1. 
  
   
  
  It has been working for 2 months. Recently though, we have been getting
  'Maximum number of simultaneous requests exceeded' errors.
  
   
  
  We are using a multi-threaded pool with 100 connections. Our application
  never has more than 10 simultaneous requests.
  
   
  
  The error is “Socked closed”. These connections were established on the
 14th
  of July. The shutdown was done on the 15th of July (in other words, these
  connections were open for more than a day). What we don't understand is
 how
  it is possible that these connections can be open for so long when they
 have
  a timeout of 26 seconds (which we assign in the PostMethod).
  
   
 
 Socket timeout applies to _active_ read operations only. Idle
 connections cannot time out, as they are not blocked in a read
 operation.  
 
 You should implement an eviction policy to close connections that have
 been idle for too long.
 
 Hope this helps
 
 Oleg
 
 
 -
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]
 
 
 
 -
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]
 


-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



-
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

RE: Pool running out of connections

[Joan Balagueró]

Hello,

Then, I understand that idle connections are in ESTABLISHED state.

I could try:

this.ictt.setTimeoutInterval(1000);
this.ictt.setConnectionTimeout(1);

This should eliminate all established connections every second.

As I told you our problem is that, with our app, we don't know why connections 
remain in ESTABLISHED state for hours and hours.

Do you think this test could help me?

 Thanks,

Joan.
 
 
 
-Mensaje original-
De: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] 
Enviado el: martes, 29 de julio de 2008 13:12
Para: HttpClient User Discussion
Asunto: Re: Pool running out of connections

Joan Balagueró wrote:
 Hello,

 We're using IdleConnectionTimeoutThread.

 The code is the following:

 this.objHttp = new HttpClient(new MultiThreadedHttpConnectionManager());
 this.ictt = new IdleConnectionTimeoutThread();
 this.ictt.addConnectionManager(this.objHttp.getHttpConnectionManager());
 this.ictt.setTimeoutInterval(1);
 this.ictt.setConnectionTimeout(2);

 If I'm right, thread that checks for idle connections starts every 10 seconds 
 and removes all idle connections with no activity for more than 20 seconds.

 Is it OK? And if it's, how can I find out that idle connections are really 
 being removed?

 I could see when we send a request with httpclient, the connection goes to 
 ESTABLISHED state, and when we release this connection, it goes to CLOSE_WAIT 
 state, and remains in this state for several minutes.

   

Are you sure the connection goes to CLOSE_WAIT state and not TIME_WAIT?

 Is an idle connection a connection in CLOSE_WAIT state? If httpclient removes 
 one idle connection, must I suppose that the number of CLOSE_WAIT connections 
 are decrement in 1? (looking at them with the netstat linux statement).

 Basicly, now my problem is I need to check that IdleConnectionTimeoutThread 
  is really removing idle connections, but I'm not sure how to do this.

   

Reusable connections are in the ESTABLISHED state, stale connections are 
in the CLOSE_WAIT state (closed on the server side but still open on the 
client side), closed connections should temporarily go into the 
TIME_WAIT state and then be removed from the connection list

 Another thing you told me was that I could use the method 
 closeIdleConnections(timeout). I suppose that the right way to do this is 
 calling this method after releaseConnection. Is this the correct way? And 
 if it's, is it optimal to call closeIdleConnections after every 
 releaseConnection?

   
The best way is to call #closeIdleConnections _before_ executing a new 
request after a period of inactivity.

Oleg


 Thanks in advance,

 Joan.


 -Mensaje original-
 De: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] 
 Enviado el: viernes, 18 de julio de 2008 0:06
 Para: HttpClient User Discussion
 Asunto: RE: Pool running out of connections

 On Thu, 2008-07-17 at 17:41 +0200, Joan Balagueró wrote:
   
 Hello,

 Thanks for your response. I suppose that this eviction policy that you are
 saying is achieved by using IdleConnectionTimeoutThread. Am I right?

 Joan.
 

 Yes, you are. You do not necessarily need a special thread to achieve
 that. Just call HttpConnectionManager#closeIdleConnections method after
 a period inactivity to purge stale connections:

 http://hc.apache.org/httpclient-3.x/apidocs/org/apache/commons/httpclient/HttpConnectionManager.html#closeIdleConnections(long)

 Oleg

   
  


 -Mensaje original-
 De: Oleg Kalnichevski [mailto:[EMAIL PROTECTED] 
 Enviado el: miércoles, 16 de julio de 2008 18:03
 Para: HttpClient User Discussion
 Asunto: Re: Pool running out of connections

 On Wed, 2008-07-16 at 09:03 +0200, Joan Balagueró wrote:
 
 Hello,

  

 We have developed software that uses HttpClient 3.1. 

  

 It has been working for 2 months. Recently though, we have been getting
 'Maximum number of simultaneous requests exceeded' errors.

  

 We are using a multi-threaded pool with 100 connections. Our application
 never has more than 10 simultaneous requests.

  

 The error is “Socked closed”. These connections were established on the
   
 14th
 
 of July. The shutdown was done on the 15th of July (in other words, these
 connections were open for more than a day). What we don't understand is
   
 how
 
 it is possible that these connections can be open for so long when they
   
 have
 
 a timeout of 26 seconds (which we assign in the PostMethod).

  
   
 Socket timeout applies to _active_ read operations only. Idle
 connections cannot time out, as they are not blocked in a read
 operation.  

 You should implement an eviction policy to close connections that have
 been idle for too long.

 Hope this helps

 Oleg


 -
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]



 -
 To unsubscribe, e-mail

TRANSLATING CODE FROM HTTPCLIENT3 TO HTTPCLIENT4

[Joan Balagueró]

Hello,

 

I’m migrating code from httpclient3 to httpcllient4, and there are some
things I do with httpclient3 that I’m not sure how to do with httpclient4. I
think my problems are in how to enable/disable cookies at connection manager
level, and how to manage idle connections (IdleConnectionHanfdler) at
connection manager level too.

 

 

 The code for httpclient3 is:

--

 

this.objHttp = new HttpClient(new MultiThreadedHttpConnectionManager());

this.objHttp.getHttpConnectionManager().getParams().setMaxTotalConnections(m
axConnections);  // maxConnections

this.objHttp.getHttpConnectionManager().getParams().setConnectionTimeout(con
nectionTimeout);  // connectionTimeout

this.objHttp.getParams().setSoTimeout(responseTimeout);//
responseTimeout

  

// Manage ‘idle’ connections.

if (idleTimeout  0) 

{

 this.ictt = new IdleConnectionsHandler();

 this.ictt.addConnectionManager(this.objHttp.getHttpConnectionManager());

 this.ictt.setTimeoutInterval(idleInterval);

 this.ictt.setConnectionTimeout(idleTimeout);

 this.ictt.start();

}

   

// Timeout for taking connections from pool (1 ms -- we don’t wait)

this.objHttp.getParams().setConnectionManagerTimeout(1);

   

// stale connections and enable/disable cookies.

this.objHttp.getHttpConnectionManager().getParams().setStaleCheckingEnabled(
true);

this.objHttp.getParams().setCookiePolicy(useCookies ? CookiePolicy.RFC_2109
: CookiePolicy.IGNORE_COOKIES); 

 

 

 Now the code for httpclient4:

--

 

this.objHttpParams = new BasicHttpParams();

HttpProtocolParams.setVersion(this.objHttpParams, HttpVersion.HTTP_1_1);

  

SchemeRegistry schemeRegistry = new SchemeRegistry();

schemeRegistry.register(new Scheme(http,
PlainSocketFactory.getSocketFactory(), 80));

schemeRegistry.register(new Scheme(https,
SSLSocketFactory.getSocketFactory(), 443));

  

ConnManagerParams.setMaxTotalConnections(this.objHttpParams,
maxConnections); // maxConnections

HttpConnectionParams.setConnectionTimeout(this.objHttpParams,
connectionTimeout); // connectionTimeout

HttpConnectionParams.setSoTimeout(this.objHttpParams, responseTimeout);
// responseTimeout

   

// Manage ‘idle’ connections.

if (idleTimeout  0)

{

 NO IDEA. I could see ‘IdleConnectionHandler’ and the method ‘add’, but at
connection level and not at connection manager level.

}   

 

// Timeout for taking connections from pool (1 ms -- we don’t wait)

ConnManagerParams.setTimeout(this.objHttpParams, 1);

   

// stale connections and cookies.

HttpConnectionParams.setStaleCheckingEnabled(true);

if (useCookies) new RFC2109SpecFactory().newInstance(this.objHttpParams);
-- is this OK? And how to disable cookies??

 

// And create the 'HttpClient' with the 'ThreadSafeClientConnManager'.

ClientConnectionManager cm = new
ThreadSafeClientConnManager(this.objHttpParams, schemeRegistry);

this.objHttp = new DefaultHttpClient(cm, this.objHttpParams);

 

 

 

Thanks in advance,

 

Joan.

 

SOME QUESTIONS ABOUT HTTPCLIENT 4

[Joan Balagueró]

Hello,

 

I’ve just migrated the code from HttpClient3 to HttpClient4, and before
starting tests I’d like to make you some questions:

 

1.  When I set the Schema “new Scheme(http,
PlainSocketFactory.getSocketFactory(), 80)”, what does it mean? It means
that the “ThreadSafeClientConnManager” can only send http requests, or it
can only send http requests to destination port 80?

 

 

2.  With HttpClient3, I set the ‘retryHandler’ at method level with:

  objPost.getParams().setParameter(HttpMethodParams.RETRY_HANDLER, new
HttpRetryHandler(3));

 

  With HttpClient4, this is similar but at connection level:

this.objHttp.setHttpRequestRetryHandler(new
DefaultHttpRequestRetryHandler(3, false));

 

Is it right? For every get/post HttpClient will retry the request 3 times
for any non-fatal error? Or should I set the retry handler at method level
like I used to do with HttpClient3? (I’ve not seen a method to do this).

 

 

3.  With HttpClient3, I used ‘HttpState’ for managing cookies. 

With HttpClient4, I set a cookieStore to a HttpContext in this way: 

 

  HttpContext localContext = new BasicHttpContext();

  CookieStore cookieStore  = new BasicCookieStore();

 

  // Set cookies.

  for (int j = 0; j  cookieSize; j++) 

  {

   BasicClientCookie bcc = new
BasicClientCookie(this.arrCookies[j].getName(),
this.arrCookies[j].getValue());

   cookieStore.addCookie(bcc);

  }

 

  localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);

  this.objHttp.execute(objGet, localContext) 

 

And, when I receive the server response, I get cookies using the same
HttpContext:

ListCookie auxCookies =
((CookieStore)localContext.getAttribute(ClientContext.COOKIE_STORE)).getCook
ies();

 

Is this the right way to do this?

 

 

4.  In HttpClient3, the documentation says that a small performance can
be gained by disabling cookies. The same for HttpClient4? If I want to
disable cookies, what I should to do? I saw this piece of code on Internet,
but I’m not sure if it’s right: 

 

this.objHttp.setCookieStore(null); 

this.objHttp.setCookieSpecs(null);

 

 

5.  Receive a null HttpEntity means that the server sent an empty
response?

 

 

Thanks for your time. 




JOAN BALAGUERÓ 

GRUPO VENTUS

C/ Balmes 195, 4º 4ª // 08006 BCN
Telf.: 902 43 05 75 // Fax: 93 292 01 49

 http://www.grupoventus.com www.grupoventus.com

 

 

 

 

HTTPCLIENT4 EXCEPTIONS

[Joan Balagueró]

Hello,

 

Just a question about exceptions sending http requests. 

 

If I set a very low value to
HttpConnectionParams.setConnectionTimeout(this.objHttpParams,
connectionTimeout), httpclient throws “SocketTimeoutException” with the
message “connect timed out”, but I expected to receive a
“ConnectTimeoutException” (as HttpClient3).  Is this right?

 

The problem is that a response timeout also throws “SocketTimeoutException”
(with the message “Read timed out”, as expected). Then, should I read the
exception message to know which exception was thrown?

 

Thanks,

 

Joan. 

 

HTTP RETRY HANDLER

[Joan Balagueró]

Hello,

 

To set a retryHandler:

this.objHttp.setHttpRequestRetryHandler(new
DefaultHttpRequestRetryHandler(retryHttp, false));

 

But if I want to disable this retry handler (because the user doesn’t want
to retry anything), how can I do this?

 

Thanks,

Joan. 

 

 

 

NO_HTTP_RESPONSE_EXCEPTION

[Joan Balagueró]

Hello,

 

I’m sending requests to one server using HttpClient3 and HttpClient4.

 

HttpClient3 works well, but HttpClient4 fails randomly with an
‘HttpNoResponseException’.

This is the trace:

 

org.apache.http.NoHttpResponseException: The target server failed to respond

at
org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponsePar
ser.java:85)

at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.ja
va:174)

at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(Abst
ractHttpClientConnection.java:182)

at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(Defa
ultClientConnection.java:235)

at
org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(Ab
stractClientConnAdapter.java:259)

at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestEx
ecutor.java:279)

at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.jav
a:121)

at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDir
ector.java:410)

at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:555)

at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:487)

at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:465)

at
com.vpfw.proxy.services.http.HttpService.sendRequest(HttpService.java:562)

 

Do you know if any other people has had a problem like this?

 

Thanks,

 

Joan.

MAX CONNECTIONS

[Joan Balagueró]

Hello,

 

I’m testing HttpClient4, and I’ve found the following problem.

 

I set MaxTotalConnections to Integer.MAX_VALUE:

ConnManagerParams.setMaxTotalConnections(this.objHttpParams,
Integer.MAX_VALUE);

 

But when I send 5 simultaneous requests, I get 2 “maxConnections” errors.
The exception is:

 

org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for
connection

at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByR
oute.java:353)

at
org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRou
te.java:238)

at
org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(
ThreadSafeClientConnManager.java:175)

at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDir
ector.java:324)

at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:555)

at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:487)

at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
va:465)

at
com.vpfw.proxy.services.http.HttpService.sendRequest(HttpService.java:565)

at
com.vpfw.proxy.services.http.HttpService.connectServer(HttpService.java:301)

at
com.vpfw.proxy.services.http.HttpService.connect(HttpService.java:259)

at com.vpfw.proxy.servlet.ProxyUtils.byPass(ProxyUtils.java:216)

at
com.vpfw.proxy.servlet.ProxyServlet.service(ProxyServlet.java:449)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)

 

 

If I print the “maxConnections” value inside the exception
(System.out.println(ConnManagerParams.getMaxTotalConnections(this.objHttpPar
ams))), I get: 2147483647. That’s OK.

 

I suppose I’m forgetting something. Maybe
“ConnManagerParams.setMaxConnectionsPerRoute()”. Any help?

 

Thanks,

 

Joan.

 

 

 

 

RE: NO_HTTP_RESPONSE_EXCEPTION

[Joan Balagueró]

Thanks Oleg. It seems strange to me because HttpClient3 (sending requests to
the same server) never fails. I'll take a look at the code again and I'll
comment you something else.

Just another question. When I modify a connection or response timeout
setting another value (for example, from 5000 to 2000) HttpClient4 applies
this new value correctly (as expected, obviously). But not the same with
maxConnections. If I set MaxConnections from 100 to 1 using
ConnManagerParams.setMaxTotalConnections, then:

DefaultHttpClient object -- DefaultParams -- parameters --
http.conn-manager.max-total=1 -- OK!

But DefaultHttpClient object -- connManager -- connectionPool --
maxTotalConnections still to 100 -- BAD!

And if I send 50 simultaneous request, HttpClient processes all of them even
with maxTotalConnections to 1.

Is this a bug? Or after ConnManagerParams.setMaxTotalConnections I must
set the new param values to the HttpClient object with
this.objHttp.setParams(set the params object with the new value of
MaxConnections)?

Thanks,

Joan.




-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: viernes, 06 de febrero de 2009 17:34
Para: HttpClient User Discussion
Asunto: RE: NO_HTTP_RESPONSE_EXCEPTION

On Fri, 2009-02-06 at 13:44 +0100, Joan Balagueró wrote:
 Hello Oleg,
 
 
 This is the log. I sent 10 concurrent requests, and all of them failed
with
 HttpNoResponseException.
 
 
 Thanks for you help,
 
 Joan.
  

There is nothing wrong with packets generated by HttpClient. This
appears to be a server side issue. The target server fails to send any
valid response back and simply closes the connection.

Oleg


 -Mensaje original-
 De: Oleg Kalnichevski [mailto:ol...@apache.org] 
 Enviado el: jueves, 05 de febrero de 2009 21:44
 Para: HttpClient User Discussion
 Asunto: Re: NO_HTTP_RESPONSE_EXCEPTION
 
 Joan Balagueró wrote:
  Hello,
  
   
  
  I’m sending requests to one server using HttpClient3 and HttpClient4.
  
   
  
  HttpClient3 works well, but HttpClient4 fails randomly with an
  ‘HttpNoResponseException’.
  
  This is the trace:
  
   
  
  org.apache.http.NoHttpResponseException: The target server failed to
 respond
  
  at
 

org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponsePar
  ser.java:85)
  
  at
 

org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.ja
  va:174)
  
  at
 

org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(Abst
  ractHttpClientConnection.java:182)
  
  at
 

org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(Defa
  ultClientConnection.java:235)
  
  at
 

org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(Ab
  stractClientConnAdapter.java:259)
  
  at
 

org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestEx
  ecutor.java:279)
  
  at
 

org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.jav
  a:121)
  
  at
 

org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDir
  ector.java:410)
  
  at
 

org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
  va:555)
  
  at
 

org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
  va:487)
  
  at
 

org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.ja
  va:465)
  
  at
 
com.vpfw.proxy.services.http.HttpService.sendRequest(HttpService.java:562)
  
   
  
  Do you know if any other people has had a problem like this?
  
 
 
 Not to my best knowledge.
 
 
 Set 'org.apache.http' logging category to DEBUG and post the resulting 
 wire / context log to the list. I'll have a look.
 
 Oleg
 
  
  Thanks,
  
   
  
  Joan.
  
  
 
 -
 To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
 For additional commands, e-mail: httpclient-users-h...@hc.apache.org


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: NO_HTTP_RESPONSE_EXCEPTION

[Joan Balagueró]

Hello Oleg,

I've been making some tests about the NoHttpResponseException problem, and I
can't find any reason why HC4 gives some random errors with this exception.

But I need your opinion about the following. I've made a very simple test
sending a request to this problematic server. This is the code
(simplified):

long start;

try
{
 (... Some operations ...)

  start = Calendar.getInstance().getTimeInMillis();

  this.objHttp.execute(objPost);

 (... Some operations ...)
}
catch (NoHttpResponseException nhre)  
{
 System.out.println(Time (ms) =  +
(Calendar.getInstance().getTimeInMillis() - start));

 nhre.printStackTrace(); 
}


And this is the result in console:

Time (ms) = 2
org.apache.http.NoHttpResponseException: The target server failed to respond
(...)


As you can see, the process just takes 2 ms. since we send the request until
we receive the error. With this time of 2ms, my opinion is that the request
is not sent and the execute method fails with this exception.

Keep in mind that I'm in Barcelona and I'm sending the request to a server
located in Madrid, and I'm using an ADSL line shared with many other users.

But I'm not really sure. What do you think about this?

Thanks,
Joan.


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: sábado, 07 de febrero de 2009 14:08
Para: HttpClient User Discussion
Asunto: Re: NO_HTTP_RESPONSE_EXCEPTION

sebb wrote:
 On 06/02/2009, Oleg Kalnichevski ol...@apache.org wrote:
 Joan Balagueró wrote:

 Thanks Oleg. It seems strange to me because HttpClient3 (sending
requests
 to
 the same server) never fails. I'll take a look at the code again and
I'll
 comment you something else.


  I do not know why the server chokes on packets generated by HttpClient
4,
 but this is clearly a server side problem. The server should return a non
 2xx response if something is not okay, not just drop the connection.

  HTTP/1.0 can be a problem or absence of User-Agent header. Whatever. At
any
 rate the server is broken.



 Just another question. When I modify a connection or response timeout
 setting another value (for example, from 5000 to 2000) HttpClient4
applies
 this new value correctly (as expected, obviously). But not the same with
 maxConnections. If I set MaxConnections from 100 to 1 using
 ConnManagerParams.setMaxTotalConnections, then:

 DefaultHttpClient object -- DefaultParams -- parameters --
 http.conn-manager.max-total=1 -- OK!

 But DefaultHttpClient object -- connManager -- connectionPool --
 maxTotalConnections still to 100 -- BAD!

 And if I send 50 simultaneous request, HttpClient processes all of them
 even
 with maxTotalConnections to 1.

 Is this a bug? Or after
 ConnManagerParams.setMaxTotalConnections I must
 set the new param values to the HttpClient object with
 this.objHttp.setParams(set the params object with the new value of
 MaxConnections)?


  This happens because the connection manager does not reset the existing
 connection pools for established routes. One can see this as a bug or a
 'feature' as one can look at it from different perspectives. To me this
is
 an expected behavior. MaxTotalConnections parameter is expected to be set
at
 the start up time and to not mutate afterwards.
 
 Maybe this could be added to the Javadoc?
 

Opened a JIRA for this issue

https://issues.apache.org/jira/browse/HTTPCLIENT-823

Oleg

  Oleg



 Thanks,

 Joan.



 -
  To unsubscribe, e-mail:
 httpclient-users-unsubscr...@hc.apache.org
  For additional commands, e-mail:
 httpclient-users-h...@hc.apache.org


 
 -
 To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
 For additional commands, e-mail: httpclient-users-h...@hc.apache.org
 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

EntityUtils.consume question

[Joan Balagueró]

Hello,

 

Is there any overhead in executing a ‘EntityUtils.consume(httpEntity)’ if
the response has been completely consumed before? (reading and closing the
‘entity.getContent()’ inputStream)

 

Thanks,

 

Joan.

 

Repeated cookies with same name and value

[Joan Balagueró]

Hello,

 

I’m querying an url (with head method) and printing all the response headers
received:

 

HttpResponse response = objHttp.execute(objHead);

 

HeaderIterator it = response.headerIterator();

   

while (it.hasNext())

{

Header h = it.nextHeader();

System.out.println(h.getName() +  =  + h.getValue());

}

 

These are the headers:

 

Date = Sun, 14 Apr 2013 16:43:04 GMT

Server = Microsoft-IIS/6.0

Etag = 

X-XSS-Protection = 0

SRV = 18

X-Powered-By = ASP.NET

X-AspNet-Version = 4.0.30319

Set-Cookie = purgado=1; expires=Tue, 14-May-2013 16:43:04 GMT; path=/

Set-Cookie = devType=0; expires=Tue, 14-May-2013 16:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = ck_idioma=es; expires=Mon, 14-Apr-2014 16:43:04 GMT; path=/;
HttpOnly

Set-Cookie = ck_sesion=2128718291; expires=Mon, 14-Apr-2014 16:43:04 GMT;
path=/; HttpOnly

Set-Cookie = USW_codigo=1615631414; expires=Mon, 14-Apr-2014 16:43:04 GMT;
path=/; HttpOnly

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Set-Cookie = preferencia=-1; expires=Sun, 14-Apr-2013 17:43:04 GMT; path=/

Cache-Control = private

Content-Type = text/html; charset=iso-8859-1

Content-Length = 172813

 

 

Obviously this is not a problem with HttpClient, but is it posible to
receive the same cookie 12 times? Does this complain the http specification?
Should HttpResponse remove the repeated headers with exactly the same name
and value?

 

Thanks,

Joan.

IP transparency

[Joan Balagueró]

Hello,

 

I have developed a proxy servlet with an xml cache, running in a Tomcat 6. 

 

When the incoming xml request is not found in the proxy cache, I use
HttpClient 4.2.5 to create a new http request (PoolingConnection) and
redirect it to the client application servers to get the xml response. 

 

So far, everything worked ok with all our clients. But now we’ve a client
that needs ip transparency. Then, the request created by httpclient needs to
include the origin ip address, not the proxy ip (the client app reads this
IP using request.getRemoteAddr(), and this is something that they cannot
change now. Therefore, things like adding a ‘X-Forwarded-for’ header cannot
be implemented in this scenario).

 

Is this possible? Has anyone found an scenario like this?

 

Thanks in advance,

 

Joan.

 

 

 

SSL connection

[Joan Balagueró]

Hello,

 

I have an application (servlet running on tomcat) that must send a https
request to a server that requires client authentication. 

 

Tomcat has correctly installed the truststore and keystore. But I understand
that when our app sends the https request, I have to attach the client
authentication required by the server. 

 

Can anyone address to any doc where I can see how to do this?

 

Thanks,

 

J. 

 

 

 

RE: SSL connection

[Joan Balagueró]

Hello Oleg,

Thanks. I've been seeing some HttpClient samples. Some of them set the 
trustStore/keyStore directly to the SSLSocketFactory. And others create an 
SSLContext with them and then set this SSLContext to the SSLSocketFactory. Any 
advantage from one respect to the other?

Furthermore, when using SSLContext we need to create an instance using the 
secure socket protocol. Is there any way to accept all secure protocols?

Thanks,

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: jueves, 28 de noviembre de 2013 10:24
Para: HttpClient User Discussion
Asunto: Re: SSL connection

On Wed, 2013-11-27 at 19:24 +0100, Joan Balagueró wrote:
 Hello,
 
  
 
 I have an application (servlet running on tomcat) that must send a 
 https request to a server that requires client authentication.
 
  
 
 Tomcat has correctly installed the truststore and keystore. But I 
 understand that when our app sends the https request, I have to attach 
 the client authentication required by the server.
 
  
 
 Can anyone address to any doc where I can see how to do this?
 
  
 
 Thanks,
 
  
 
 J. 
 

There is enough good material on SSL fundamentals on the web. Just google it 
out. 

As far as HC APIs are concerned SSLContextBuilder should help you set up the 
correct SSL context for your application. Most likely you will need to load the 
private key and add it to the context using this method [1].

Oleg 

[1]
http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/apidocs/org/apache/http/conn/ssl/SSLContextBuilder.html#loadKeyMaterial%28java.security.KeyStore,%20char[],%20org.apache.http.conn.ssl.PrivateKeyStrategy%29

  
 
 
 
 
 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: SSL connection

[Joan Balagueró]

Oleg,

I close the connection pool by using  
this.objHttp.getConnectionManager().shutdown();

About the expired connections, I have an idleConnectionsHandler that every 5 
seconds removes expired connections and those that take idle longer than 5 
seconds:

   private ClientConnectionManager cm;
   (...)
   this.cm.closeExpiredConnections();
   this.cm.closeIdleConnections(5, TimeUnit.SECONDS);
  (...)

That's why I though that the pool should be empty after 1 minute.

Regards,

Joan.


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: lunes, 2 de diciembre de 2013 10:36
Para: HttpClient User Discussion
Asunto: Re: SSL connection

On Sat, 2013-11-30 at 23:52 +0100, Joan Balagueró wrote:
 Hello Oleg,
 
 Thanks for you help. Everything works fine now.
 
 Just one more question: when I shutdown Tomcat, I see this message in 
 catalina.out (ssl debug enabled):
 
 main, called close()
 main, called closeInternal(true)
 main, SEND TLSv1 ALERT:  warning, description = close_notify main, 
 WRITE: TLSv1 Alert, length = 18 main, called 
 closeSocket(selfInitiated)
 
 
 If I send 8 https requests, this message appears 8 times when shutting down 
 tomcat. It seems that HttpClient is closing the http connection pool (in 
 fact, our app closes it). But I have a keep-alive of 20 seconds, and I'm 
 waiting more than 1 minute (from the last request sent) before shutting down 
 tomcat (so I understand that all connections should be expired and removed 
 from the pool).
 
 I suppose I'm missing something. Could you clarify me this point, please?
 
 Thanks,
 
 Joan.
 

Joan

I do not know SSL protocol that intimately, but it looks like this message 
basically means that the server had to initiate connection shutdown and notify 
the client. I do not think there is anything wrong with that. 

Please note that expired connections in the client connection pool do not get 
evicted automatically if the pool is inactive. One needs to explicitly call 
#closeExpired to make it happen.

How exactly do you close the connection pool on the client side?

Oleg 

 
 -Mensaje original-
 De: Oleg Kalnichevski [mailto:o...@ok2consulting.com] Enviado el: 
 jueves, 28 de noviembre de 2013 22:12
 Para: HttpClient User Discussion
 Asunto: Re: SSL connection
 
 On Thu, 2013-11-28 at 20:11 +0100, Joan Balagueró wrote:
  Hello Oleg,
  
  Thanks. I've been seeing some HttpClient samples. Some of them set the 
  trustStore/keyStore directly to the SSLSocketFactory.
 
 SSLSocketFactory constructors internally create an SSLContext instance and 
 initialize it with the trust / key material passed as parameters. 
 
   And others create an SSLContext with them and then set this SSLContext to 
  the SSLSocketFactory. Any advantage from one respect to the other?
  
 
 No, not really. Simply a matter of convenience.
 
  Furthermore, when using SSLContext we need to create an instance using the 
  secure socket protocol. Is there any way to accept all secure protocols?
  
 
 I am not sure what you mean by that. Exactly wha
 
  Thanks,
  
  Joan.
  
  -Mensaje original-
  De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: jueves, 
  28 de noviembre de 2013 10:24
  Para: HttpClient User Discussion
  Asunto: Re: SSL connection
  
  On Wed, 2013-11-27 at 19:24 +0100, Joan Balagueró wrote:
   Hello,
   

   
   I have an application (servlet running on tomcat) that must send a 
   https request to a server that requires client authentication.
   

   
   Tomcat has correctly installed the truststore and keystore. But I 
   understand that when our app sends the https request, I have to 
   attach the client authentication required by the server.
   

   
   Can anyone address to any doc where I can see how to do this?
   

   
   Thanks,
   

   
   J. 
   
  
  There is enough good material on SSL fundamentals on the web. Just google 
  it out. 
  
  As far as HC APIs are concerned SSLContextBuilder should help you set up 
  the correct SSL context for your application. Most likely you will need to 
  load the private key and add it to the context using this method [1].
  
  Oleg
  
  [1]
  http://hc.apache.org/httpcomponents-client-4.3.x/httpclient/apidocs/
  or 
  g/apache/http/conn/ssl/SSLContextBuilder.html#loadKeyMaterial%28java
  .s 
  ecurity.KeyStore,%20char[],%20org.apache.http.conn.ssl.PrivateKeyStr
  at
  egy%29
  

   
   
   
   
   
  
  
  
  
  - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
  For additional commands, e-mail: httpclient-users-h...@hc.apache.org
  
  
  
  
  - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
  For additional commands, e-mail: httpclient-users-h...@hc.apache.org
  
 
 
 
 -
 To unsubscribe

RE: SSL connection

[Joan Balagueró]

Oleg,

What I would mean was:

1. I send 8 https requests, then I have 8 connections in the pool.

2. I have a keep-alive of 20 seconds. Then, I wait 1 min.

3. After 1 min, all connections should be removed from the pool (due to my 
IdleConnectionHandler, that works fine)

4. Then, I shutdown tomcat.

5. Then, I expect no messages because there are no connections in pool. But I 
see exactly 8 messages like this:
  main, called close()
  main, called closeInternal(true)
  main, SEND TLSv1 ALERT:  warning, description = close_notify main,
  WRITE: TLSv1 Alert, length = 18 main, called
  closeSocket(selfInitiated)

6. If I repeat the process sending 5 requests, then I see 5 messages. One per 
connection.


That's why I though this was something related to Http pool, as if the 
connections were not expired. But if you say that you don't print these 
messages, then I am a bit lost (because tomcat guys also say that they don't 
print these messages, and me neither).

Anyway, thanks,

Joan.


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: lunes, 2 de diciembre de 2013 16:12
Para: HttpClient User Discussion
Asunto: Re: SSL connection

On Mon, 2013-12-02 at 11:16 +0100, Joan Balagueró wrote:
 Oleg,
 
 I close the connection pool by using  
 this.objHttp.getConnectionManager().shutdown();
 
 About the expired connections, I have an idleConnectionsHandler that every 
 5 seconds removes expired connections and those that take idle longer than 5 
 seconds:
 
private ClientConnectionManager cm;
(...)
this.cm.closeExpiredConnections();
this.cm.closeIdleConnections(5, TimeUnit.SECONDS);
   (...)
 
 That's why I though that the pool should be empty after 1 minute.
 
 Regards,
 
 Joan.
 

Nothing in the connection pool can survive a call to #shutdown(). Can it be 
that you have some other processes connected to the same Tomcat instance?

Oleg  

 
 -Mensaje original-
 De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: lunes, 2 
 de diciembre de 2013 10:36
 Para: HttpClient User Discussion
 Asunto: Re: SSL connection
 
 On Sat, 2013-11-30 at 23:52 +0100, Joan Balagueró wrote:
  Hello Oleg,
  
  Thanks for you help. Everything works fine now.
  
  Just one more question: when I shutdown Tomcat, I see this message in 
  catalina.out (ssl debug enabled):
  
  main, called close()
  main, called closeInternal(true)
  main, SEND TLSv1 ALERT:  warning, description = close_notify main,
  WRITE: TLSv1 Alert, length = 18 main, called
  closeSocket(selfInitiated)
  
  
  If I send 8 https requests, this message appears 8 times when shutting down 
  tomcat. It seems that HttpClient is closing the http connection pool (in 
  fact, our app closes it). But I have a keep-alive of 20 seconds, and I'm 
  waiting more than 1 minute (from the last request sent) before shutting 
  down tomcat (so I understand that all connections should be expired and 
  removed from the pool).
  
  I suppose I'm missing something. Could you clarify me this point, please?
  
  Thanks,
  
  Joan.
  
 
 Joan
 
 I do not know SSL protocol that intimately, but it looks like this message 
 basically means that the server had to initiate connection shutdown and 
 notify the client. I do not think there is anything wrong with that. 
 
 Please note that expired connections in the client connection pool do not get 
 evicted automatically if the pool is inactive. One needs to explicitly call 
 #closeExpired to make it happen.
 
 How exactly do you close the connection pool on the client side?
 
 Oleg
 
  
  -Mensaje original-
  De: Oleg Kalnichevski [mailto:o...@ok2consulting.com] Enviado el: 
  jueves, 28 de noviembre de 2013 22:12
  Para: HttpClient User Discussion
  Asunto: Re: SSL connection
  
  On Thu, 2013-11-28 at 20:11 +0100, Joan Balagueró wrote:
   Hello Oleg,
   
   Thanks. I've been seeing some HttpClient samples. Some of them set the 
   trustStore/keyStore directly to the SSLSocketFactory.
  
  SSLSocketFactory constructors internally create an SSLContext instance and 
  initialize it with the trust / key material passed as parameters. 
  
And others create an SSLContext with them and then set this SSLContext 
   to the SSLSocketFactory. Any advantage from one respect to the other?
   
  
  No, not really. Simply a matter of convenience.
  
   Furthermore, when using SSLContext we need to create an instance using 
   the secure socket protocol. Is there any way to accept all secure 
   protocols?
   
  
  I am not sure what you mean by that. Exactly wha
  
   Thanks,
   
   Joan.
   
   -Mensaje original-
   De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: 
   jueves,
   28 de noviembre de 2013 10:24
   Para: HttpClient User Discussion
   Asunto: Re: SSL connection
   
   On Wed, 2013-11-27 at 19:24 +0100, Joan Balagueró wrote:
Hello,

 

I have an application (servlet running on tomcat) that must send 
a https request

RE: SSL connection

[Joan Balagueró]

Hello,

Yes, ssl debug is enabled.

But what we are discussing now is why 'someone' is closing 8 http connections 
that should be already closed.

I'll do what Oleg proposes, inspect the pool before shutting down tomcat.

Thanks,

Joan.

-Mensaje original-
De: thc...@gmail.com [mailto:thc...@gmail.com] 
Enviado el: lunes, 2 de diciembre de 2013 19:40
Para: HttpClient User Discussion
Asunto: Re: SSL connection

Hi.

 That's why I though this was something related to Http pool, as if the 
 connections were not expired. But if you say that you don't print these 
 messages, then I am a bit lost (because tomcat guys also say that they don't 
 print these messages, and me neither).

Those messages looks a lot like JSSE debugging messages [1]. Do you know if the 
System property javax.net.debug [2] is activated?


[1] 
http://docs.oracle.com/javase/6/docs/technotes/guides/security/jsse/ReadDebug.html
[2] 
http://docs.oracle.com/javase/6/docs/technotes/guides/security/jsse/JSSERefGuide.html#Debug

Best regards.

On 2 December 2013 17:40, Joan Balagueró joan.balagu...@grupoventus.com wrote:
 Oleg,

 What I would mean was:

 1. I send 8 https requests, then I have 8 connections in the pool.

 2. I have a keep-alive of 20 seconds. Then, I wait 1 min.

 3. After 1 min, all connections should be removed from the pool (due 
 to my IdleConnectionHandler, that works fine)

 4. Then, I shutdown tomcat.

 5. Then, I expect no messages because there are no connections in pool. But I 
 see exactly 8 messages like this:
  main, called close()
  main, called closeInternal(true)
  main, SEND TLSv1 ALERT:  warning, description = close_notify main,
  WRITE: TLSv1 Alert, length = 18 main, called
  closeSocket(selfInitiated)

 6. If I repeat the process sending 5 requests, then I see 5 messages. One per 
 connection.


 That's why I though this was something related to Http pool, as if the 
 connections were not expired. But if you say that you don't print these 
 messages, then I am a bit lost (because tomcat guys also say that they don't 
 print these messages, and me neither).

 Anyway, thanks,

 Joan.


 -Mensaje original-
 De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: lunes, 2 
 de diciembre de 2013 16:12
 Para: HttpClient User Discussion
 Asunto: Re: SSL connection

 On Mon, 2013-12-02 at 11:16 +0100, Joan Balagueró wrote:
 Oleg,

 I close the connection pool by using  
 this.objHttp.getConnectionManager().shutdown();

 About the expired connections, I have an idleConnectionsHandler that every 
 5 seconds removes expired connections and those that take idle longer than 5 
 seconds:

private ClientConnectionManager cm;
(...)
this.cm.closeExpiredConnections();
this.cm.closeIdleConnections(5, TimeUnit.SECONDS);
   (...)

 That's why I though that the pool should be empty after 1 minute.

 Regards,

 Joan.


 Nothing in the connection pool can survive a call to #shutdown(). Can it be 
 that you have some other processes connected to the same Tomcat instance?

 Oleg


 -Mensaje original-
 De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: lunes, 2 
 de diciembre de 2013 10:36
 Para: HttpClient User Discussion
 Asunto: Re: SSL connection

 On Sat, 2013-11-30 at 23:52 +0100, Joan Balagueró wrote:
  Hello Oleg,
 
  Thanks for you help. Everything works fine now.
 
  Just one more question: when I shutdown Tomcat, I see this message in 
  catalina.out (ssl debug enabled):
 
  main, called close()
  main, called closeInternal(true)
  main, SEND TLSv1 ALERT:  warning, description = close_notify main,
  WRITE: TLSv1 Alert, length = 18 main, called
  closeSocket(selfInitiated)
 
 
  If I send 8 https requests, this message appears 8 times when shutting 
  down tomcat. It seems that HttpClient is closing the http connection pool 
  (in fact, our app closes it). But I have a keep-alive of 20 seconds, and 
  I'm waiting more than 1 minute (from the last request sent) before 
  shutting down tomcat (so I understand that all connections should be 
  expired and removed from the pool).
 
  I suppose I'm missing something. Could you clarify me this point, please?
 
  Thanks,
 
  Joan.
 

 Joan

 I do not know SSL protocol that intimately, but it looks like this message 
 basically means that the server had to initiate connection shutdown and 
 notify the client. I do not think there is anything wrong with that.

 Please note that expired connections in the client connection pool do not 
 get evicted automatically if the pool is inactive. One needs to explicitly 
 call #closeExpired to make it happen.

 How exactly do you close the connection pool on the client side?

 Oleg

 
  -Mensaje original-
  De: Oleg Kalnichevski [mailto:o...@ok2consulting.com] Enviado el:
  jueves, 28 de noviembre de 2013 22:12
  Para: HttpClient User Discussion
  Asunto: Re: SSL connection
 
  On Thu, 2013-11-28 at 20:11 +0100, Joan Balagueró wrote:
   Hello Oleg,
  
   Thanks. I've been seeing some

RE: Migrating from 4.1 to 4.3

[Joan Balagueró]

Hello Oleg,

Thanks. But taking this example:

RequestConfig.Builder builder = RequestConfig.custom(); if (proxyPort != 0) {
builder.setProxy(new HttpHost(proxyHost, proxyPort)) } RequestConfig config 
= builder.build();

The question is this can be modified once the CloseableHttpClient is already 
created. This means that I had a proxy but not now, thus I want to remove this 
proxy from the HttpClient.

Must I create another RequestConfig.Builder, reconfigure it with all options  
(without the setProxy, because now I don't have proxy) and set it again to the 
HttpClient?

Thanks,
Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: miércoles, 12 de marzo de 2014 15:35
Para: HttpClient User Discussion
Asunto: Re: Migrating from 4.1 to 4.3

On Tue, 2014-03-11 at 12:43 +0100, Joan Balagueró wrote:
 Hello,
 

  
 
 1.   Where can I set the manager connection timeout?
 
 HttpClientParams.setConnectionManagerTimeout(this.objHttpParams, 1);

RequestConfig config = RequestConfig.custom()
.setConnectionRequestTimeout(1)
.build();

 
 2.   Where can I set the default connection timeout?
 
 HttpConnectionParams.setConnectionTimeout(this.objHttpParams,
 connectionTimeout);
 

RequestConfig config = RequestConfig.custom()
.setConnectTimeout(n)
.build();

 3.   I see how to add a proxy to the HttpClientBuilder, but how can I
 remove it?
 
 public void setProxy(String proxyHost, int proxyPort)
 
 {
 
   if (proxyPort == 0)
 this.objHttpParams.removeParameter(ConnRoutePNames.DEFAULT_PROXY);
 
   else 
 this.objHttp.getParams().setParameter(ConnRoutePNames.DEFAULT_PROXY,
 new HttpHost(proxyHost, proxyPort));
 
 }
 

RequestConfig.Builder builder = RequestConfig.custom(); if (proxyPort != 0) {
builder.setProxy(new HttpHost(proxyHost, proxyPort)) } RequestConfig config 
= builder.build();


  
 
 4.   Similar, for example, a retry handler:
 
 public void setRetryHandler(int retryHttp)
 
 {
 
   if (retryHttp  0) this.objHttp.setHttpRequestRetryHandler(new
 DefaultHttpRequestRetryHandler(retryHttp, false));
 
   else this.objHttp.setHttpRequestRetryHandler(new
 DefaultHttpRequestRetryHandler(0, false));
 
 }
 

HttpClientBuilder builder = HttpClients.custom(); if (retryHttp  0) {
builder.setRetryHandler(new
DefaultHttpRequestRetryHandler(retryHttp, false)); } else {
builder.setRetryHandler(new DefaultHttpRequestRetryHandler(0, false)); } 
CloseableHttpClient client = builder.build();


Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: Migrating from 4.1 to 4.3

[Joan Balagueró]

Hi,

What I'm trying to say is: is there any way to remove or change something that 
you have already configured with a  builder? It's not only proxy, it's whatever 
you can config. 

For example, I have a HttpClient with a KeepAlivestreategy, proxy, retryhandler 
and a tcp_nodely to true.

With this existing HttpClient, now I want to change (on the fly) this values, 
removing the proxy, keepAlive and retryHandler and changing the tcp_nodelay to 
false.

For example, with 4.2 I could remove a proxy by using:
this.objHttpParams.removeParameter(ConnRoutePNames.DEFAULT_PROXY);

Or I could change tcp_nodelay by using:
HttpConnectionParams.setTcpNoDelay(this.objHttpParams, Boolean.TRUE);


With 4.3, it seems that the only way is to recreate the Httpclient with a 
builder/s with all the new values.


Thanks,

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: miércoles, 12 de marzo de 2014 17:27
Para: HttpClient User Discussion
Asunto: Re: Migrating from 4.1 to 4.3


On Wed, 2014-03-12 at 15:46 +0100, Joan Balagueró wrote:
 Hello Oleg,
 
 Thanks. But taking this example:
 
 RequestConfig.Builder builder = RequestConfig.custom(); if (proxyPort != 0) {
 builder.setProxy(new HttpHost(proxyHost, proxyPort)) } 
 RequestConfig config = builder.build();
 
 The question is this can be modified once the CloseableHttpClient is already 
 created. This means that I had a proxy but not now, thus I want to remove 
 this proxy from the HttpClient.
 
 Must I create another RequestConfig.Builder, reconfigure it with all options  
 (without the setProxy, because now I don't have proxy) and set it again to 
 the HttpClient?
 
 Thanks,
 Joan.
 

Use a custom route planner if you want more control over request routing.

http://hc.apache.org/httpcomponents-client-4.3.x/tutorial/html/connmgmt.html#d5e342

Oleg 




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Keep alive

[Joan Balagueró]

Hello,

 

After migrating to 4.3 from 4.1, I don’t get keep-alive connections working.

 

My code is:

 

(…)

if (this.vkas == null) this.vkas = new KeepAliveStrategy(keepAliveDuration);

else this.vkas.setKeepAliveDuration(keepAliveDuration);

 

(...)

HttpClientBuilder hcBuilder =
HttpClients.custom().setConnectionManager(this.phccm).

disableCookieManagement().

 
setDefaultRequestConfig(this.rc).

setRetryHandler(this.rrh).

 
setKeepAliveStrategy(this.vkas);

// Add proxy.

if (proxyPort  0) hcBuilder.setProxy(new HttpHost(proxyHost, proxyPort));

  

// Build 'HttpClient'.

this.objHttp = hcBuilder.build();

 

 

When I send a request, http log shows the following (with a keep alive of 20
seconds):

 

Auth cache not set in the context

Connection request: [route: {}-http://ws.rhodasol.es:80][total kept alive:
0; route allocated: 0 of 100; total allocated: 0 of 100]

Connection leased: [id: 148][route: {}-http://ws.rhodasol.es:80][total kept
alive: 0; route allocated: 1 of 100; total allocated: 1 of 100]

Connecting to ws.rhodasol.es/188.165.133.226:80

http-outgoing-148: Shutdown connection

http-outgoing-148: Close connection

Connection released: [id: 148][route: {}-http://ws.rhodasol.es:80][total
kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 100]

 

 

In fact, my KeepAliveStrategy class is not called, because none of the
traces I put there are logged.

 

Should I enable keep alive in any way before calling the
setKeepAliveStrategy method?

 

Thanks,

 

Joan.

RE: Keep alive

[Joan Balagueró]

Hello,

Forget it ...

Thanks.

Joan.

-Mensaje original-
De: Joan Balagueró [mailto:joan.balagu...@grupoventus.com] 
Enviado el: jueves, 13 de marzo de 2014 19:23
Para: httpclient-users@hc.apache.org
Asunto: Keep alive

Hello,

 

After migrating to 4.3 from 4.1, I don’t get keep-alive connections working.

 

My code is:

 

(…)

if (this.vkas == null) this.vkas = new KeepAliveStrategy(keepAliveDuration);

else this.vkas.setKeepAliveDuration(keepAliveDuration);

 

(...)

HttpClientBuilder hcBuilder =
HttpClients.custom().setConnectionManager(this.phccm).

disableCookieManagement().

 
setDefaultRequestConfig(this.rc).

setRetryHandler(this.rrh).

 
setKeepAliveStrategy(this.vkas);

// Add proxy.

if (proxyPort  0) hcBuilder.setProxy(new HttpHost(proxyHost, proxyPort));

  

// Build 'HttpClient'.

this.objHttp = hcBuilder.build();

 

 

When I send a request, http log shows the following (with a keep alive of 20
seconds):

 

Auth cache not set in the context

Connection request: [route: {}-http://ws.rhodasol.es:80][total kept alive:
0; route allocated: 0 of 100; total allocated: 0 of 100]

Connection leased: [id: 148][route: {}-http://ws.rhodasol.es:80][total kept
alive: 0; route allocated: 1 of 100; total allocated: 1 of 100]

Connecting to ws.rhodasol.es/188.165.133.226:80

http-outgoing-148: Shutdown connection

http-outgoing-148: Close connection

Connection released: [id: 148][route: {}-http://ws.rhodasol.es:80][total
kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 100]

 

 

In fact, my KeepAliveStrategy class is not called, because none of the
traces I put there are logged.

 

Should I enable keep alive in any way before calling the
setKeepAliveStrategy method?

 

Thanks,

 

Joan.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Final questions about HC4.3.3

[Joan Balagueró]

Hello,

 

Migration from 4.1 to 4.3 is finished. Just some questions about certain
parameters:

 

1.   For performance, is it recommended to set true to ‘setTcpNoDelay’
and false to ‘setStaleConnectionCheckEnabled’? I tried several combinations
in my stress tests and I don’t see any significance difference.

 

2.   What does the “RequestConfig.custom().setAuthenticationEnabled()”
refers to?

 

3.   And HttpClients.custom().disableAuthCaching() and
disableConnectionState()?

 

4.   The HttpClients.custom().disableCookieManagement() is equivalent to
“objHttp.removeRequestInterceptorByClass(RequestAddCookies.class);this.objHt
tp.removeResponseInterceptorByClass(ResponseProcessCookies.class);” in 4.1.?

 

 

I’ve noticed an improvement of between 8-10% in the number of requests that
4.3. can process compared with 4.1. In addition, it seems the garbage
generated by 4.3. is less than 4.1 (the time between young generations is at
least slightly greater with 4.3).

 

So, congratulations. Good work !

 

Joan.

SSL and stale check changes

[Joan Balagueró]

Hello,

 

I’ve just installed HC4.4.1 and some methods related to ssl are deprecated.
Concretely this piece of code:

 

   SSLContext sslContext = SSLContexts.custom().useTLS().build();

   sslContext.init(keyManagers, null, null);

 

   return (new SSLConnectionSocketFactory(sslContext,
SSLConnectionSocketFactory.STRICT_HOSTNAME_VERIFIER));

 

 

But taking a look at the HC4.4. documentation, section 2.7.3., I still see
this way of building a SSLContext.

 

Which methods should I use now? Maybe the SSLContexts.createSystemDefault()?
Does this method create a SSLContext with TLS like the previous one? And how
can I create a SSLConnectionFactory with strict hostname verifier?

 

Furthermore, the following method is also deprecated:

 

  this.rc = RequestConfig.custom().setAuthenticationEnabled(false).

 
setConnectionRequestTimeout(1).

 
setConnectTimeout(this.connectionTimeout).

 
setSocketTimeout(this.responseTimeout).

 
setExpectContinueEnabled(false).

 
setStaleConnectionCheckEnabled(false).

setRedirectsEnabled(false).

build();

 

I know that stale checking has been improved in this version, but how can I
use it?

 

Thanks,

 

Joan. 

RE: Fw:Fw:PoolingHttpClientConnectionManager how to reuse the connection

[Joan Balagueró]

Hello,

Regarding this posts:

 Any help in this issue would be much appreciated. 
 Thanks
 mrhuang
 

 I am not quite sure I understand the issue. You should not even be using 
 #releaseConnection in the first place. It is provided for compatibility with 
 HC 3.x. You should always close CloseableHttpResponse in a try-finally or 
 try-with-resources.
 Oleg

You say that using CloseableHttpResponse is the right way to release a 
connection. Currently, our code to release a connection is the above (works 
fine):

  if (entity != null)
  {
   try { EntityUtils.consume(entity); }
   catch (Exception e) {}
  }
  else
  {
   if (objPost != null) objPost.abort();
   else if (objGet != null) objGet.abort();
  }

Do you think it's preferrable to change this code by one that directly closes 
the CloseableHttpResponse?

Thanks,

Joan.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Help with async client

[Joan Balagueró]

Hello,

 

We have moved from the blocking client (PoolingHttpClientConnectionManager)
to the async one (PoolingNHttpClientConnectionManager), and we are starting
some tests to check performance.

 

We have a test app that sends xml requests to our proxy. If the response is
not found in cache,  the proxy uses the http client to get the responses
from the back end. 

 

Starting 20 threads, with the blocking client we reach about 700 requests
per second.

 

With the async client, the pattern we are seeing is: 20 requests are
processed, then a 5 second pause, then 20 more requests processed, 5 second
pause, etc.

 

Obviously we have something misconfigured in our async pool. We though that
the problem could come from the max htttp connections allowed, but checking
the log:

 

[exchange: 2537] start execution

[exchange: 2537] Request connection for {}->http://10.20.30.246:80

Connection request: [route: {}->http://10.20.30.246:80][total kept alive: 0;
route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]

Connection leased: [id: http-outgoing-209][route:
{}->http://10.20.30.246:80][total kept alive: 0; route allocated: 1 of
2147483647; total allocated: 1 of 2147483647]

[exchange: 2537] Connection allocated: CPoolProxy{http-outgoing-209
[ACTIVE]}

http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][r:]: Set
attribute http.nio.exchange-handler

http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][rw:]: Event
set [w]

http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][rw:]: Set
timeout 5

http-outgoing-209 [ACTIVE]: Connected

http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][rw:]: Set
attribute http.nio.http-exchange-state

Start connection routing

Connection route established

[exchange: 2537] Attempt 1 to execute request

Target auth state: UNCHALLENGED

Proxy auth state: UNCHALLENGED

http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][rw:]: Set
timeout 1000

http-outgoing-209 >> POST /wsserhs/rhodasol HTTP/1.1

http-outgoing-209 >> content-type: application/x-www-form-urlencoded;
charset=ISO-8859-1

http-outgoing-209 >> host: ws.rhodasol.es

http-outgoing-209 >> user-agent: Apache-HttpClient/4.4.1 (Java/1.7.0_75)

http-outgoing-209 >> x-ventusproxy-id:
bnVsbHxudWxsfG51bGx8bnVsbHxudWxsfG51bGx8QlI2MzczfFRFU1RNQ3xGUkF8bnVsbHwyMDA4
MDYwMnxudWxsfG51bGx8MTguMnwxOS4yfDIwLjF8MTguM3wxOS48bnVsbD58MjAuMnwyMi5jb3Vu
dHJ5MzF8MjM

uMHwyNC5FU1B8MjIuYXJlYXwyMy4wfDI0LjExfDIyLnJlZ2lvbnwyMy4wfDI0LnwyMi5jaXR5fDI
zLjB8MjQufDIyLmFjY29tbW9kYXRpb25Db2RlfDIzLjF8MjQufDIyLmFjY29tbW9kYXRpb25UeXB
lfDIzLjF8MjQuMHwyMi5jYXRlZ29yeXwyMy4xfDI0LnwyMi5wc

mljZVR5cGV8MjMuMnwyNC4zfDIyLm9mZmVyfDIzLjJ8MjQufDIyLmNvbmNlcHR8MjMuMnwyNC58M
jIuYm9hcmR8MjMuMnwyNC58MjIuY2FuY2VsUG9saWNpZXN8MjMuMnwyNC4xfGlzby04ODU5LTE=

http-outgoing-209 >> x-forwarded-for: 192.168.1.29

http-outgoing-209 >> Content-Length: 1368

http-outgoing-209 >> Connection: Keep-Alive

http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][rw:]: Event
set [w]

http-outgoing-209 [ACTIVE] Output ready

[exchange: 2537] produce content

[exchange: 2537] Request completed

 

(etc).

 

I can send more information about this log and our pool configuration. But
since the pattern is so clear, maybe someone has experienced something
similar and can tell me what’s wrong.

 

I suppose that the following is not related with this issue, but we have and
idleConnectionHandler that is executed every 5 seconds.

 

Thanks,

 

Joan.

 

 

 

 

 

 

 

RE: Help with async client

[Joan Balagueró]

Hello,

I hope this is the last question ... On our proxy we are reading the response 
from the backend with a FutureCallback. On the complete method, 
we process the response body in this way:

public void completed(HttpResponse objResponse) 
{
 HttpEntity entity  = null;
  
 try
 {
 entity = objResponse.getEntity();
 int contentLength  = (int)entity.getContentLength(); 
 ByteArrayOutputStream baos = new ByteArrayOutputStream(contentLength > 0 ? 
contentLength : this.bufferHttpRespuestas);

 Bis= new BufferedInputStream(entity.getContent(), 
this.bufferHttpRespuestas); 
 byte[] tmp = new byte[this.bufferHttpRespuestas];
 int numBytesRead;
 
 while ((numBytesRead = bis.read(tmp)) >= 0) baos.write(tmp, 0, numBytesRead);
 ( . . .)


The response is read from the inputstream contained in 'objHttpResponse', this 
response is already read so no network calls here (we read locally from the ' 
objResponse' object).
Reading the body in this way, am I storing this body twice, one in the ' 
objResponse' and another in the 'baos' variable?

In the link to the benchmark you sent me, a HttpAsyncResponseConsumer is used, 
so the content is processed in chunks on the  consumeContent method (and I 
understand that the 'responseReceived' method is called once the headers have 
been processed).

So the point is: is one of these methods better taking into account that my 
responses can be really large and I always need to store them in memory for 
further processing?
 
Thanks,

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: jueves, 17 de marzo de 2016 10:50
Para: HttpClient User Discussion
Asunto: Re: Help with async client

On Thu, 2016-03-17 at 10:42 +0100, Joan Balagueró wrote:
> Hi Oleg,
> 
> Thanks. If I have a 400ms connection timeout and a 850ms response timeout, I 
> understand that the select interval must contain the 400ms (the lowest 
> value). True?
> 

Yes.

> In our app these connection/response timeouts can be modified on the fly. 
> That's why we are using a request config, and for each request we perform a 
> copy of the default request config and replace the values for 
> connect/response timeout.
> 
> So, the select interval should also be modified in this case. Is this 
> possible with the 'PoolingNHttpClientConnectionManager' already created? 

No.

> Because  I don't see any method to get the IOReactor config from the nio pool.
> 

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: Help with async client

[Joan Balagueró]

Hi Oleg,

Finally the issue was nothing to do with the async client.

Now we are experimenting with connection and response timeouts, setting just 
1ms for both. In the blocking client, we get connection and response timeouts 
for all requests we send.
But with the async client we are not able to get any exception, all requests 
are processed normally and all responses are obtained from the backend.

Checking the log:
[exchange: 1] start execution
[exchange: 1] Request connection for {}->http://10.20.30.246:80
Connection request: [route: {}->http://10.20.30.246:80][total kept alive: 0; 
route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]
Connection leased: [id: http-outgoing-0][route: 
{}->http://10.20.30.246:80][total kept alive: 0; route allocated: 1 of 
2147483647; total allocated: 1 of 2147483647]
[exchange: 1] Connection allocated: CPoolProxy{http-outgoing-0 [ACTIVE]}
http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][r:]: Set attribute 
http.nio.exchange-handler
http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][rw:]: Event set [w]
http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][rw:]: Set timeout 1 
  <--- CONNECTION TIMEOUT?
http-outgoing-0 [ACTIVE]: Connected
http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][rw:]: Set attribute 
http.nio.http-exchange-state
Start connection routing
Connection route established
[exchange: 1] Attempt 1 to execute request
Target auth state: UNCHALLENGED
Proxy auth state: UNCHALLENGED
http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][rw:]: Set timeout 1 
<-- RESPONSE TIMEOUT?
http-outgoing-0 >> POST /wsserhs/rhodasol HTTP/1.1
http-outgoing-0 >> content-type: application/x-www-form-urlencoded; 
charset=ISO-8859-1
http-outgoing-0 >> host: ws.rhodasol.es
( . . . )

It seems the values are correct, 1ms for both connection and response timeouts. 
Any response from the bakcend takes 100ms at least, because a Thread.sleep(100) 
is set before writing any byte to the servletoutputstream.

So, why are these exceptions not triggered?

Thanks,
Joan.




-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: martes, 15 de marzo de 2016 16:14
Para: HttpClient User Discussion
Asunto: Re: Help with async client

On Tue, 2016-03-15 at 15:54 +0100, Joan Balagueró wrote:
> Hello,
> 
>  
> 
> We have moved from the blocking client 
> (PoolingHttpClientConnectionManager)
> to the async one (PoolingNHttpClientConnectionManager), and we are 
> starting some tests to check performance.
> 
>  
> 
> We have a test app that sends xml requests to our proxy. If the 
> response is not found in cache,  the proxy uses the http client to get 
> the responses from the back end.
> 
>  
> 
> Starting 20 threads, with the blocking client we reach about 700 
> requests per second.
> 
>  
> 
> With the async client, the pattern we are seeing is: 20 requests are 
> processed, then a 5 second pause, then 20 more requests processed, 5 
> second pause, etc.
> 
>  
> 
> Obviously we have something misconfigured in our async pool. We though 
> that the problem could come from the max htttp connections allowed, 
> but checking the log:
> 
>  
> 
> [exchange: 2537] start execution
> 
> [exchange: 2537] Request connection for {}->http://10.20.30.246:80
> 
> Connection request: [route: {}->http://10.20.30.246:80][total kept 
> alive: 0; route allocated: 0 of 2147483647; total allocated: 0 of 
> 2147483647]
> 
> Connection leased: [id: http-outgoing-209][route:
> {}->http://10.20.30.246:80][total kept alive: 0; route allocated: 1 of 
> 2147483647; total allocated: 1 of 2147483647]
> 
> [exchange: 2537] Connection allocated: CPoolProxy{http-outgoing-209 
> [ACTIVE]}
> 
> http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][r:]: Set 
> attribute http.nio.exchange-handler
> 
> http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][rw:]: 
> Event set [w]
> 
> http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][rw:]: 
> Set timeout 5
> 
> http-outgoing-209 [ACTIVE]: Connected
> 
> http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][rw:]: 
> Set attribute http.nio.http-exchange-state
> 
> Start connection routing
> 
> Connection route established
> 
> [exchange: 2537] Attempt 1 to execute request
> 
> Target auth state: UNCHALLENGED
> 
> Proxy auth state: UNCHALLENGED
> 
> http-outgoing-209 10.20.30.70:57638<->10.20.30.246:80[ACTIVE][rw:]: 
> Set timeout 1000
> 
> http-outgoing-209 >> POST /wsserhs/rhodasol HTTP/1.1
> 
> http-outgoing-209 >> content-type: application/x-www-form-urlencoded;
> charset=ISO-885

RE: Help with async client

[Joan Balagueró]

Hi Oleg,

Thanks. If I have a 400ms connection timeout and a 850ms response timeout, I 
understand that the select interval must contain the 400ms (the lowest value). 
True?

In our app these connection/response timeouts can be modified on the fly. 
That's why we are using a request config, and for each request we perform a 
copy of the default request config and replace the values for connect/response 
timeout.

So, the select interval should also be modified in this case. Is this possible 
with the 'PoolingNHttpClientConnectionManager' already created? Because  I 
don't see any method to get the IOReactor config from the nio pool.

Thanks,

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: miércoles, 16 de marzo de 2016 16:22
Para: HttpClient User Discussion
Asunto: Re: Help with async client

On Wed, 2016-03-16 at 15:41 +0100, Joan Balagueró wrote:
> Hi Oleg,
> 
> Finally the issue was nothing to do with the async client.
> 
> Now we are experimenting with connection and response timeouts, setting just 
> 1ms for both. In the blocking client, we get connection and response timeouts 
> for all requests we send.
> But with the async client we are not able to get any exception, all requests 
> are processed normally and all responses are obtained from the backend.
> 

The smallest granularity for I/O timeouts by default is 1 second. One
can reduce it by setting lower value of IOReactorConfig#selectInterval
at the cost of higher CPU utilization.

Oleg

> Checking the log:
> [exchange: 1] start execution
> [exchange: 1] Request connection for {}->http://10.20.30.246:80
> Connection request: [route: {}->http://10.20.30.246:80][total kept alive: 0; 
> route allocated: 0 of 2147483647; total allocated: 0 of 2147483647]
> Connection leased: [id: http-outgoing-0][route: 
> {}->http://10.20.30.246:80][total kept alive: 0; route allocated: 1 of 
> 2147483647; total allocated: 1 of 2147483647]
> [exchange: 1] Connection allocated: CPoolProxy{http-outgoing-0 [ACTIVE]}
> http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][r:]: Set 
> attribute http.nio.exchange-handler
> http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][rw:]: Event set 
> [w]
> http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][rw:]: Set timeout 
> 1   <--- CONNECTION TIMEOUT?
> http-outgoing-0 [ACTIVE]: Connected
> http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][rw:]: Set 
> attribute http.nio.http-exchange-state
> Start connection routing
> Connection route established
> [exchange: 1] Attempt 1 to execute request
> Target auth state: UNCHALLENGED
> Proxy auth state: UNCHALLENGED
> http-outgoing-0 10.20.30.70:53654<->10.20.30.246:80[ACTIVE][rw:]: Set timeout 
> 1 <-- RESPONSE TIMEOUT?
> http-outgoing-0 >> POST /wsserhs/rhodasol HTTP/1.1
> http-outgoing-0 >> content-type: application/x-www-form-urlencoded; 
> charset=ISO-8859-1
> http-outgoing-0 >> host: ws.rhodasol.es
> ( . . . )
> 
> It seems the values are correct, 1ms for both connection and response 
> timeouts. Any response from the bakcend takes 100ms at least, because a 
> Thread.sleep(100) is set before writing any byte to the servletoutputstream.
> 
> So, why are these exceptions not triggered?
> 
> Thanks,
> Joan.
> 
> 
> 
> 
> -Mensaje original-
> De: Oleg Kalnichevski [mailto:ol...@apache.org] 
> Enviado el: martes, 15 de marzo de 2016 16:14
> Para: HttpClient User Discussion
> Asunto: Re: Help with async client
> 
> On Tue, 2016-03-15 at 15:54 +0100, Joan Balagueró wrote:
> > Hello,
> > 
> >  
> > 
> > We have moved from the blocking client 
> > (PoolingHttpClientConnectionManager)
> > to the async one (PoolingNHttpClientConnectionManager), and we are 
> > starting some tests to check performance.
> > 
> >  
> > 
> > We have a test app that sends xml requests to our proxy. If the 
> > response is not found in cache,  the proxy uses the http client to get 
> > the responses from the back end.
> > 
> >  
> > 
> > Starting 20 threads, with the blocking client we reach about 700 
> > requests per second.
> > 
> >  
> > 
> > With the async client, the pattern we are seeing is: 20 requests are 
> > processed, then a 5 second pause, then 20 more requests processed, 5 
> > second pause, etc.
> > 
> >  
> > 
> > Obviously we have something misconfigured in our async pool. We though 
> > that the problem could come from the max htttp connections allowed, 
> > but checking the log:
> > 
> >  
> > 
> > [exchange: 2537] start execution
> > 
> &g

Async client

[Joan Balagueró]

Hello,

We are starting to work with the Async http client. We are creating a 
PoolingNHttpClientConnectionManager (translating our old 
PoolingHttpClientConnectionManager to a PoolingNHttpClientConnectionManager). 
Here we have 2 questions:

1. Hasn't the async client a retry handler and a stale check validation? 

2. When we set the connection and response timeout at IOReactor level and 
RequestConfig level, what takes precedence?


Thanks,

Joan.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: I/O reactor status: STOPPED

[Joan Balagueró]

Within the HttpAsyncResponseConsumer?

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:o...@ok2consulting.com] 
Enviado el: martes, 4 de octubre de 2016 13:59
Para: HttpClient User Discussion; Joan Balagueró
Asunto: Re: I/O reactor status: STOPPED

On October 4, 2016 12:52:42 PM GMT+02:00, "Joan Balagueró" 
<joan.balagu...@grupoventus.com> wrote:
>Hello,
>
>I'm occassionally receiving this error, and the async pool stops
>working:
>
>java.lang.IllegalStateException: Request cannot be executed; I/O 
>reactor
>status: STOPPED
>at org.apache.http.util.Asserts.check(Asserts.java:46)
>at
>org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunn
>ing(C
>loseableHttpAsyncClientBase.java:90)
>at
>org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(Interna
>lHttp
>AsyncClient.java:123)
>at
>org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(Closea
>bleHt
>tpAsyncClient.java:68)
>
>
>I'm using an HttpAsyncResponseConsumer, and I know that one of the 
>reasons for this error is to throw an exception on the "failed" method. 
>That's why this method looks like as follows:
>
>@Override
> public void failed(final Exception e)
> {
>  try { this.setError(e); }
>  catch (Throwable t) {}
> }
>
>Even with this, sometimes the IO reactor stops. Is there anything else 
>I must take into account in order to avoid this error? Any other method 
>I should control? What else can cause the IO reactor to be stopped?
>
>Thanks,
>
>Joan.

Most likely due to a runtime thrown by application code.

Oleg
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

I/O reactor status: STOPPED

[Joan Balagueró]

Hello,

I'm occassionally receiving this error, and the async pool stops working:

java.lang.IllegalStateException: Request cannot be executed; I/O reactor
status: STOPPED
at org.apache.http.util.Asserts.check(Asserts.java:46)
at
org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(C
loseableHttpAsyncClientBase.java:90)
at
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttp
AsyncClient.java:123)
at
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHt
tpAsyncClient.java:68)


I'm using an HttpAsyncResponseConsumer, and I know that one of the reasons
for this error is to throw an exception on the "failed" method. That's why
this method looks like as follows:

@Override
 public void failed(final Exception e)
 {
  try { this.setError(e); }
  catch (Throwable t) {}
 }

Even with this, sometimes the IO reactor stops. Is there anything else I
must take into account in order to avoid this error? Any other method I
should control? What else can cause the IO reactor to be stopped?

Thanks,

Joan.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Socket timeout in the async http client

[Joan Balagueró]

Hello,

 

I’m setting the socket timeout on my requests as follows:
requestBuilder.setConfig(RequestConfig.copy(this.objHttp.getRequestConfig())
.setSocketTimeout(responseTimeoutToSet).build());

 

I have double checked that the variable ‘responseTimeoutToSet’ has the right
value. So if I set up a socket timeout of 1000ms or 2000ms or 3000ms, it
works fine. But if I set up a socket timeout of 1200ms, the exception is
thrown after 2000ms. And if I set up a socket timeout of 2400ms, the
exception is thrown after 3000ms.

 

Is this right behaviour with the async client? The blocking client respects
strictly the socket timeout, but the async seems to skip from 1 to 2 to 3
seconds …

 

Thanks,

 

Joan.

RE: Socket timeout in the async http client

[Joan Balagueró]

Hi Dmitry,

Thanks for the explanantion. It's clear now.

Joan.

-Mensaje original-
De: Dmitry Potapov [mailto:potapo...@gmail.com] 
Enviado el: viernes, 7 de octubre de 2016 15:18
Para: HttpClient User Discussion
Asunto: Re: Socket timeout in the async http client

Hello Joan,

Async client works on reactor model, so timeouts being checked only between 
selects.
You can change interval between selects by using setSelectInterval function in 
IOReactorConfig builder:
For example: IOReactorConfig.custom().setSelectInterval(100L).build();

Please be concern that reducing this value causes reactor threads to consume 
more CPU resources.

--
Dmitry

On Fri, Oct 07, 2016 at 03:03:44PM +0200, Joan Balagueró wrote:
> Hello,
> 
>  
> 
> I’m setting the socket timeout on my requests as follows:
> requestBuilder.setConfig(RequestConfig.copy(this.objHttp.getRequestCon
> fig()) .setSocketTimeout(responseTimeoutToSet).build());
> 
>  
> 
> I have double checked that the variable ‘responseTimeoutToSet’ has the 
> right value. So if I set up a socket timeout of 1000ms or 2000ms or 
> 3000ms, it works fine. But if I set up a socket timeout of 1200ms, the 
> exception is thrown after 2000ms. And if I set up a socket timeout of 
> 2400ms, the exception is thrown after 3000ms.
> 
>  
> 
> Is this right behaviour with the async client? The blocking client 
> respects strictly the socket timeout, but the async seems to skip from 
> 1 to 2 to 3 seconds …
> 
>  
> 
> Thanks,
> 
>  
> 
> Joan.
> 

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Retries with the async client

[Joan Balagueró]

Hello,

I’m trying to retry ‘NoHttpResponse’ exceptions with the async client. The
point is I’m unable to generate this kind of error from my server. So my
question is: does this kind of exceptions that can be retried reach the
callback? In my case the  HttpAsyncResponseConsumer.responseReceived (so
they could be retried in this method),or the exception is thrown in the
CloseableHttpAsyncClient.execute method? 

Thanks,

Joan.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: TimeoutException in async pool

[Joan Balagueró]

Thanks Stefan and yes, that's right. And I already told my client to modify 
these settings. But I still have the doubt about why a Timeout exception in a 
pool that has no upper limit in the number of pooled connections. 


-Mensaje original-
De: Stefan Magnus Landrø [mailto:stefan.lan...@gmail.com] 
Enviado el: lunes, 3 de octubre de 2016 9:00
Para: HttpClient User Discussion
Asunto: Re: TimeoutException in async pool

So - first I'd set pool size to 50 since that's all you need on average. Then 
set acquisition timeout to something that makes sense. 10-20ms?

Sendt fra min iPhone

> Den 3. okt. 2016 kl. 07.32 skrev Joan Balagueró 
> <joan.balagu...@grupoventus.com>:
> 
> Hello,
> 
> I’m using the async PoolingNHttpClientConnectionManager (v 4.1.2).
> 
> Occasionally I’m getting this error:
> 
> java.util.concurrent.TimeoutException
>at
> org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(Abs
> tractN
> IOConnPool.java:364)
>at
> org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:
> 276)
>at
> org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requ
> estCon
> nection(PoolingNHttpClientConnectionManager.java:266)
>at
> org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestC
> onnect
> ion(AbstractClientExchangeHandler.java:363)
>at
> org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start
> (Defau
> ltClientExchangeHandlerImpl.java:125)
>at
> org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(Intern
> alHttp
> AsyncClient.java:141)
>at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(Close
> ableHt
> tpAsyncClient.java:68)
> 
> 
> It seems that all connections in pool are busy. But my setup says:
> 
> this.phccm = new PoolingNHttpClientConnectionManager(…);
> this.phccm.setMaxTotal(Integer.MAX_VALUE);
> this.phccm.setDefaultMaxPerRoute(Integer.MAX_VALUE);
> 
> 
> I don't have more than 1000 requests per second, and the average 
> response time is less than 20ms.
> 
> What could be the reason for this error?
> 
> Thanks,
> Joan.
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

TimeoutException in async pool

[Joan Balagueró]

Hello,

I’m using the async PoolingNHttpClientConnectionManager (v 4.1.2).

Occasionally I’m getting this error:

java.util.concurrent.TimeoutException
at
org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractN
IOConnPool.java:364)
at
org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:
276)
at
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestCon
nection(PoolingNHttpClientConnectionManager.java:266)
at
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnect
ion(AbstractClientExchangeHandler.java:363)
at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(Defau
ltClientExchangeHandlerImpl.java:125)
at
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttp
AsyncClient.java:141)
at
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHt
tpAsyncClient.java:68)


It seems that all connections in pool are busy. But my setup says:

this.phccm = new PoolingNHttpClientConnectionManager(…);
this.phccm.setMaxTotal(Integer.MAX_VALUE);
this.phccm.setDefaultMaxPerRoute(Integer.MAX_VALUE);


I don't have more than 1000 requests per second, and the average response
time is less than 20ms.

What could be the reason for this error?

Thanks,
Joan.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: TimeoutException in async pool

[Joan Balagueró]

Thanks Oleg, is there anything I can do in the pool configuration to mitigate 
this effect?

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: lunes, 3 de octubre de 2016 21:32
Para: HttpClient User Discussion
Asunto: Re: TimeoutException in async pool

On Mon, 2016-10-03 at 10:45 +0200, Joan Balagueró wrote:
> Thanks Stefan and yes, that's right. And I already told my client to modify 
> these settings. But I still have the doubt about why a Timeout exception in a 
> pool that has no upper limit in the number of pooled connections. 
> 
> 

This is because of the lock contention.

Oleg

> -Mensaje original-
> De: Stefan Magnus Landrø [mailto:stefan.lan...@gmail.com] Enviado el: 
> lunes, 3 de octubre de 2016 9:00
> Para: HttpClient User Discussion
> Asunto: Re: TimeoutException in async pool
> 
> So - first I'd set pool size to 50 since that's all you need on average. Then 
> set acquisition timeout to something that makes sense. 10-20ms?
> 
> Sendt fra min iPhone
> 
> > Den 3. okt. 2016 kl. 07.32 skrev Joan Balagueró 
> > <joan.balagu...@grupoventus.com>:
> > 
> > Hello,
> > 
> > I’m using the async PoolingNHttpClientConnectionManager (v 4.1.2).
> > 
> > Occasionally I’m getting this error:
> > 
> > java.util.concurrent.TimeoutException
> >at
> > org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(A
> > bs
> > tractN
> > IOConnPool.java:364)
> >at
> > org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:
> > 276)
> >at
> > org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.re
> > qu
> > estCon
> > nection(PoolingNHttpClientConnectionManager.java:266)
> >at
> > org.apache.http.impl.nio.client.AbstractClientExchangeHandler.reques
> > tC
> > onnect
> > ion(AbstractClientExchangeHandler.java:363)
> >at
> > org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.sta
> > rt
> > (Defau
> > ltClientExchangeHandlerImpl.java:125)
> >at
> > org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(Inte
> > rn
> > alHttp
> > AsyncClient.java:141)
> >at
> > org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(Clo
> > se
> > ableHt
> > tpAsyncClient.java:68)
> > 
> > 
> > It seems that all connections in pool are busy. But my setup says:
> > 
> > this.phccm = new PoolingNHttpClientConnectionManager(…);
> > this.phccm.setMaxTotal(Integer.MAX_VALUE);
> > this.phccm.setDefaultMaxPerRoute(Integer.MAX_VALUE);
> > 
> > 
> > I don't have more than 1000 requests per second, and the average 
> > response time is less than 20ms.
> > 
> > What could be the reason for this error?
> > 
> > Thanks,
> > Joan.
> > 
> > 
> > 
> > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> > 
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: java.util.concurrent.TimeoutException with the async client

[Joan Balagueró]

Hello,

Sorry, I didn't paste the last part of the xml:



Thanks,

Joan.

-Mensaje original-
De: Joan Balagueró [mailto:joan.balagu...@grupoventus.com] 
Enviado el: jueves, 27 de octubre de 2016 12:23
Para: httpclient-users@hc.apache.org
Asunto: java.util.concurrent.TimeoutException with the async client

Hello,

After working in production with the async client (instead of the blocking
one) everything seems ok, except that now I can see this error
intermitently:

java.util.concurrent.TimeoutException
at
org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractN
IOConnPool.java:364)
at
org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest(Abstr
actNIOConnPool.java:344)
at
org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.jav
a:318)
at
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseCon
nection(PoolingNHttpClientConnectionManager.java:303)
at
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.releaseConnect
ion(AbstractClientExchangeHandler.java:239)
at
org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientE
xec.java:387)
at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCom
pleted(DefaultClientExchangeHandlerImpl.java:168)
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAs
yncRequestExecutor.java:436)
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRe
questExecutor.java:326)
at
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNH
ttpClientConnection.java:265)
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODi
spatch.java:81)
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODi
spatch.java:39)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODis
patch.java:114)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:1
62)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIORe
actor.java:337)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOR
eactor.java:315)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor
.java:276)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:10
4)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(Abs
tractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:745)

It seems that the pool is exhausted at this point.

The setup of both pools is similar. MaxConnections to 600,
ConnectionRequestTimeout' is set to 1 (because I don't want to wait for a
pooling connection to be free), an IdleConnectionsHandler, etc.

I'm not sure if some connections are not being closed. With the blocking
client one manually released the connection to the pool, but with the async
one I'm not closing connections explicitly (so I suppose this is done
automatically by the async client).

The point is this error was not present with the blocking client. If I'm not
wrong, with the async client the possibility to exhaust the pool should be
more difficult because every time the server is not sending tcp packets the
thread is released and used for other connections. 

Well, whatever the reason I'm going to enable http traces. Since my client
only permitts updates once a week, I want to be sure that I'm enabling the
http traces correctly (otherwise I will have to wait one more week).

With the blocking client I enabled traces putting these lines in the
log4j.xml file (and it worked). Is the same procedure with the async client?



 






Thanks,

Joan.




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

java.util.concurrent.TimeoutException with the async client

[Joan Balagueró]

Hello,

After working in production with the async client (instead of the blocking
one) everything seems ok, except that now I can see this error
intermitently:

java.util.concurrent.TimeoutException
at
org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractN
IOConnPool.java:364)
at
org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest(Abstr
actNIOConnPool.java:344)
at
org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.jav
a:318)
at
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseCon
nection(PoolingNHttpClientConnectionManager.java:303)
at
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.releaseConnect
ion(AbstractClientExchangeHandler.java:239)
at
org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientE
xec.java:387)
at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCom
pleted(DefaultClientExchangeHandlerImpl.java:168)
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAs
yncRequestExecutor.java:436)
at
org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRe
questExecutor.java:326)
at
org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNH
ttpClientConnection.java:265)
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODi
spatch.java:81)
at
org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODi
spatch.java:39)
at
org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODis
patch.java:114)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:1
62)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIORe
actor.java:337)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOR
eactor.java:315)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor
.java:276)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:10
4)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(Abs
tractMultiworkerIOReactor.java:588)
at java.lang.Thread.run(Thread.java:745)

It seems that the pool is exhausted at this point.

The setup of both pools is similar. MaxConnections to 600,
ConnectionRequestTimeout' is set to 1 (because I don't want to wait for a
pooling connection to be free), an IdleConnectionsHandler, etc.

I'm not sure if some connections are not being closed. With the blocking
client one manually released the connection to the pool, but with the async
one I'm not closing connections explicitly (so I suppose this is done
automatically by the async client).

The point is this error was not present with the blocking client. If I'm not
wrong, with the async client the possibility to exhaust the pool should be
more difficult because every time the server is not sending tcp packets the
thread is released and used for other connections. 

Well, whatever the reason I'm going to enable http traces. Since my client
only permitts updates once a week, I want to be sure that I'm enabling the
http traces correctly (otherwise I will have to wait one more week).

With the blocking client I enabled traces putting these lines in the
log4j.xml file (and it worked). Is the same procedure with the async client?



 






Thanks,

Joan.




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

NoHttpResponseException in the async client

[Joan Balagueró]

Hello,

We have replaced the httpclient by the async client in our application.
Everything works fine, but the ‘NoHttpResponseException’ has disappeared
from our error statistics reports. So, or with the new async pool this error
does not occur for some reason (that I don’t know) or we are not catching
this error correctly (more likely).

We are using an ‘HttpAsyncResponseConsumer’ and overwriting the
‘responseReceived’, ‘consumeContent’ and ‘failed’ methods. We
understand that when a ‘NoHttpResponseException’ occurs,
‘responseReceived’ and ‘consumeContent’ are not called, and  the
‘failed’ method is the only one that is directly called.

Our ‘failed’ method looks like this:

@Override
public void failed(final Exception e)
{
  ProxyServletException pse = null;
  
  if (e.getClass() == java.net.SocketTimeoutException.class) pse = new
ProxyServletException(ErrorConstants.HTTP_RESPONSE_TIMEOUT, e);
  else if (e.getClass() == java.net.ConnectException.class) pse = new
ProxyServletException(ErrorConstants.HTTP_CONNECT_TIMEOUT, e);
  else if (e.getClass() == org.apache.http.NoHttpResponseException.class)
pse = new ProxyServletException(ErrorConstants.HTTP_NO_RESPONSE, e);←
the error is caugth here 
  else if (e.getClass() == java.io.IOException.class) pse = new
ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e);
  else if (e.getClass() ==
com.ventusproxy.proxy.servlet.ProxyServletException.class) pse =
(ProxyServletException)e;
  else if (e.getClass() ==
org.apache.http.conn.ConnectionPoolTimeoutException.class) pse = new
ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e);
  else if (e.getClass() == java.util.concurrent.TimeoutException.class) pse
= new ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e);

  pse = (pse != null ? pse : new
ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e));

  ( . . . )
}


Is this ok? Or I'm missing something?

Thanks,

Joan.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Throwing socketTimeoutException within HttpAsyncResponseConsumer consumeContent method

[Joan Balagueró]

Hello,

In my async client pool, I’ve currently  set up a select interval of 1000ms.
But I need to control response timeouts of 50ms, 100ms, etc.

So I’have tried to implement something like this in the 'consumeContent'
method of my 'HttpAsyncResponseConsumer':

public void consumeContent(final ContentDecoder decoder, final IOControl
ioctrl) throws IOException 
 {
  int numBytesRead;

  if (this.bbuf == null) 
  {
   this.bbuf  = ByteBuffer.allocate(32768);
   this.vbaos = new VentusByteArrayOutputStream();
  }

  // If the difference between now and the starting point of the request
(when I sent it to the server) is greater than 8ms, throw a
SocketTimeoutException.
  long diff = System.currentTimeMillis() - this.startTime;
  if (diff >= 8) throw new java.net.SocketTimeoutException("Socket timeout
:: consumeContent :: [ " + diff + "ms > " + "8ms ]");

  while ( (numBytesRead = decoder.read(this.bbuf)) > 0 ) 
  {
   this.vbaos.write(this.bbuf.array(), 0, numBytesRead);
   this.bbuf.clear();
  }
 }

And it seems to work. The exception is thrown, the consumeContent is no
longer called, the 'responseCompleted' method is not called, and the async
client calls the 'failed' method that logs the exception correctly.

My question is: can I be sure that the response reading was stopped, the
connection was aborted (even without completing the response reading) and
the thread was released?

Thanks,

Joan.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: java.util.concurrent.TimeoutException with the async client

[Joan Balagueró]

Hi Oleg,

Sorry yes, reviewing emails I already asked about TimeoutException. I've been 
reading again the httpcore tutorial because think I had a misunderstanding 
about how the async client works.

In the previous case we had about 1000 requests per second, and the average 
response time was about 20ms. So Stefan proposed a pool of 50 connections with 
a pool timeout of 10-20ms.

In the current case we're having not more than 100 requests per second, but the 
average response time is 1 second.

When a connection is established, if no packets are being received, the 
IOReactor dispatch thread is released to do another task, but the http 
connection continues established. So in the current case, where I have less 
concurrency but much longer response times I should set up a higher max 
connections. Right?

If so, just two questions:

1. Why, exactly in the same load conditions, a pool with 600 connections is 
more than enough for the blocking client and not enough for the async one?

2. Regarding the  'ConnectionRequestTimeout', you told me that the reason was 
probably due to lock contention. Is this lock contention provoked by this 
timeout of only 1ms? Do you always recommend a higher value as Stefan said 
previously?


Thanks in advance,

Joan. 


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: jueves, 27 de octubre de 2016 13:48
Para: HttpClient User Discussion
Asunto: Re: java.util.concurrent.TimeoutException with the async client

On Thu, 2016-10-27 at 12:23 +0200, Joan Balagueró wrote:
> Hello,
> 
> After working in production with the async client (instead of the 
> blocking
> one) everything seems ok, except that now I can see this error
> intermitently:
> 
> java.util.concurrent.TimeoutException
> at
> org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(Abs
> tractN
> IOConnPool.java:364)
> at
> org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest
> (Abstr
> actNIOConnPool.java:344)
> at
> org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPo
> ol.jav
> a:318)
> at
> org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.rele
> aseCon
> nection(PoolingNHttpClientConnectionManager.java:303)
> at
> org.apache.http.impl.nio.client.AbstractClientExchangeHandler.releaseC
> onnect
> ion(AbstractClientExchangeHandler.java:239)
> at
> org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainC
> lientE
> xec.java:387)
> at
> org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.respo
> nseCom
> pleted(DefaultClientExchangeHandlerImpl.java:168)
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(
> HttpAs
> yncRequestExecutor.java:436)
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpA
> syncRe
> questExecutor.java:326)
> at
> org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(Def
> aultNH
> ttpClientConnection.java:265)
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(Intern
> alIODi
> spatch.java:81)
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(Intern
> alIODi
> spatch.java:39)
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(Abstrac
> tIODis
> patch.java:114)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.
> java:1
> 62)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(Abstra
> ctIORe
> actor.java:337)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(Abstr
> actIOR
> eactor.java:315)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOR
> eactor
> .java:276)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.j
> ava:10
> 4)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.r
> un(Abs
> tractMultiworkerIOReactor.java:588)
> at java.lang.Thread.run(Thread.java:745)
> 
> It seems that the pool is exhausted at this point.
> 
> The setup of both pools is similar. MaxConnections to 600, 
> ConnectionRequestTimeout' is set to 1 (because I don't want to wait 
> for a pooling connection to be free), an IdleConnectionsHandler, etc.
> 

This looks wrong to me. I think I already explained that.

> I'm not sure if some connections are not being closed. With the 
> blocking client one manually released the connection to the pool, but 
> with the async one I'm not closing connections explicitly (so I 
> suppose this is done automatically by the async client).
> 
> The

RE: java.util.concurrent.TimeoutException with the async client

[Joan Balagueró]

Hi Oleg,

> 1. Why, exactly in the same load conditions, a pool with 600 connections is 
> more than enough for the blocking client and not enough for the async one?
> 

I cannot answer this question unless I can analyze and re-run the benchmark 
used to load test both clients.
--> ok, I'll retest it.

> 2. Regarding the  'ConnectionRequestTimeout', you told me that the reason was 
> probably due to lock contention. Is this lock contention provoked by this 
> timeout of only 1ms? Do you always recommend a higher value as Stefan said 
> previously?
> 

What is the point of having such a low request timeout? What is it exactly you 
intent to achieve by making requests fail if the pool manager fails to lease a 
connection after 1 ms? 
--> Our software is a proxy. We have some clients with a prerequirement to not 
permitt requests that take more than, for example, 10ms. So if for getting one 
connection from the pool we have to wait 10ms, then the prerequirement cannot 
be accomplished. So that's why, in certain installations, we don't allow to 
wait this time, we simply prefer to fail and return an error to the client like 
"Maximum number of simultaneous connections exceded".


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: Async client with self signed certificate

[Joan Balagueró]

Hi,

Yes, you were right, the keystore didin't have the server's public certificate.

Thanks,

Joan.

-Mensaje original-
De: Hassan Khan [mailto:hassankhan...@gmail.com] 
Enviado el: jueves, 18 de mayo de 2017 21:08
Para: HttpClient User Discussion
Asunto: Re: Async client with self signed certificate

Hi,

This is a issue with the CA certs... SSL handshake is failing...

if java turn on ssl debug... you will see the error in detail...

But if you have added the cacert to the java cacert files.. then java should 
recognize the self signed cert..

This is not a code issue.. it more to do with cert that is the point i am 
trying to make... May be the file Store does not have the self signed 
certificate added to it...

Hope it helps

Thanks
Hassan



On Thu, May 18, 2017 at 2:48 PM, Joan Balagueró < 
joan.balagu...@grupoventus.com> wrote:

> Hello,
>
>
>
> I’ve been using SSL with client authentication with signed 
> certificates in async http client 4.1, with no problem.
>
>
>
> My code is:
>
>
>
> FileInputStream  fKeyStore = new FileInputStream(new 
> File(keyStoreLocation));
>
> KeyStore keyStore = KeyStore.getInstance(keyStoreType);
>
> keyStore.load(fKeyStore, keyStorePassword.toCharArray());
>
>
>
> KeyManagerFactory kmfactory =
> KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm())
> ;
>
> kmfactory.init(keyStore, keyStorePassword.toCharArray());
>
> KeyManager[] keyManagers = kmfactory.getKeyManagers();
>
>
>
> TrustManagerFactory tmf =
> TrustManagerFactory.getInstance(TrustManagerFactory.
> getDefaultAlgorithm());
>
> tmf.init(keyStore);
>
>
>
> SSLContext sslContext = SSLContexts.custom().build();
>
> sslContext.init(keyManagers, tmf.getTrustManagers(), null);
>
>
>
> return (new SSLIOSessionStrategy(sslContext, new String[] { "TLSv1" }, 
> null, SSLIOSessionStrategy.getDefaultHostnameVerifier()));
>
>
>
>
>
> But now I have an installation with ssl and client authentication but 
> with a self-signed certificate. Using the previous code I get the 
> following error (I suppose because it doesn’t find the CA 
> certificate):
>
> Caused by: sun.security.validator.ValidatorException: PKIX path 
> building
> failed: sun.security.provider.certpath.SunCertPathBuilderException: 
> unable to find valid certification path to requested target
>
>
>
> Can anyone help me with this? How should I modify the previous code to 
> have this working? I’ve tried some alternatives but none of them worked.
>
>
>
> Thanks in advance.
>
>
>
> Joan.
>
>
>
>
>
>
>
>
>
>


--
Hassan Khan


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

async http client DNS error

[Joan Balagueró]

Hello,

 

We are having this error sporadically (once every week).

 

Additional information: java.net.UnknownHostException:
live.suppliers.gsisservices.com: System error

at java.net.Inet4AddressImpl.lookupAllHostAddr(Native
Method)

at
java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)

at
java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)

at java.net.InetAddress.getAllByName0(InetAddress.java:1276)

at java.net.InetAddress.getAllByName(InetAddress.java:1192)

at java.net.InetAddress.getAllByName(InetAddress.java:1126)

at
org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsR
esolver.java:45)

at
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAd
dressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:
609)

at
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager$InternalAd
dressResolver.resolveRemoteAddress(PoolingNHttpClientConnectionManager.java:
580)

at
org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractN
IOConnPool.java:427)

at
org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:
276)

at
org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requestCon
nection(PoolingNHttpClientConnectionManager.java:266)

at
org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestConnect
ion(AbstractClientExchangeHandler.java:363)

at
org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start(Defau
ltClientExchangeHandlerImpl.java:125)

at
org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttp
AsyncClient.java:141)

at
org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHt
tpAsyncClient.java:68)

at
com.ventusproxy.proxy.services.http.ServerHttpRequest.sendRequest(ServerHttp
Request.java:282)

 

At the hosts file of this server we have mapped this domain
(live.suppliers.gsisservices.com) to its ip address. Anyways sometimes this
resolution fails.

I see this line in the trace: at
org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsR
esolver.java:45)

Is it possible that the async http client is only using the dns by default
to resolve hosts names?

 

The code to send the request:

this.asyncHttp.execute(new BasicAsyncRequestProducer(new
HttpHost(uri.getHost(), uri.getPort(), uri.getScheme()), httpRequest), harc,
null);

 

Thanks,

 

Joan.

 

Async client with self signed certificate

[Joan Balagueró]

Hello,

 

I’ve been using SSL with client authentication with signed certificates in
async http client 4.1, with no problem.

 

My code is:

 

FileInputStream  fKeyStore = new FileInputStream(new
File(keyStoreLocation));

KeyStore keyStore = KeyStore.getInstance(keyStoreType);

keyStore.load(fKeyStore, keyStorePassword.toCharArray());

 

KeyManagerFactory kmfactory =
KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());

kmfactory.init(keyStore, keyStorePassword.toCharArray());

KeyManager[] keyManagers = kmfactory.getKeyManagers();

 

TrustManagerFactory tmf =
TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());

tmf.init(keyStore);

 

SSLContext sslContext = SSLContexts.custom().build();

sslContext.init(keyManagers, tmf.getTrustManagers(), null);

 

return (new SSLIOSessionStrategy(sslContext, new String[] { "TLSv1" }, null,
SSLIOSessionStrategy.getDefaultHostnameVerifier()));

 

 

But now I have an installation with ssl and client authentication but with a
self-signed certificate. Using the previous code I get the following error
(I suppose because it doesn’t find the CA certificate):

Caused by: sun.security.validator.ValidatorException: PKIX path building
failed: sun.security.provider.certpath.SunCertPathBuilderException: unable
to find valid certification path to requested target

 

Can anyone help me with this? How should I modify the previous code to have
this working? I’ve tried some alternatives but none of them worked.

 

Thanks in advance.

 

Joan.

 

 

 

 

http 5.0

[Joan Balagueró]

Hello,

Is there any scheduled  release date for HttpClient 5.0 (GA)?

Thanks,

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: martes, 10 de julio de 2018 13:34
Para: HttpClient User Discussion
Asunto: Re: Global lock in http pool

On Tue, 2018-07-10 at 13:25 +0200, Joan Balagueró wrote:
> Thanks Oleg,
> I'm using the Async Client (4.1.2), what must I do to have it 
> available? Just move to HttpCore 5.0 / HttpClient 5.0?
> 

You have two options: upgrading to HttpClient 5.0 or building a custom 
connection manager.

Oleg

> 
> -Mensaje original-
> De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: martes, 10 
> de julio de 2018 13:22
> Para: HttpClient User Discussion
> Asunto: Re: Global lock in http pool
> 
> On Tue, 2018-07-10 at 12:01 +0200, Joan Balagueró wrote:
> > Hello,
> > 
> >  
> > 
> > We are using the async http client 4.1.2 in our app (an api 
> > Gateway).
> > Several installations are managing more than 15.000 requests 
> > simultaneous opened, and now we are in the process to optimize some 
> > code in the proxy.
> > 
> >  
> > 
> > Some time ago we could read a thread talking about the global lock 
> > that http pool has, in order to control the max connections 
> > parameter.
> > To avoid a possible lock contention in high concurrent scenarios, 
> > Oleg planned to implement a new method to control the max 
> > connections in a more “relaxed”
> > way, that would not use the lock at the cost to not exactly control 
> > the number of connections opened at any time.
> > 
> >  
> > 
> > Our question is if there is any schedule to have this method 
> > implemented, or if it’s planned to release it in any of the next 
> > http client versions.
> > 
> 
> It has already been implemented in HttpCore 5.0 and HttpClient 5.0
> 
> Oleg
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Global lock in http pool

[Joan Balagueró]

Hello,

 

We are using the async http client 4.1.2 in our app (an api Gateway).
Several installations are managing more than 15.000 requests simultaneous
opened, and now we are in the process to optimize some code in the proxy.

 

Some time ago we could read a thread talking about the global lock that http
pool has, in order to control the max connections parameter. To avoid a
possible lock contention in high concurrent scenarios, Oleg planned to
implement a new method to control the max connections in a more “relaxed”
way, that would not use the lock at the cost to not exactly control the
number of connections opened at any time.

 

Our question is if there is any schedule to have this method implemented, or
if it’s planned to release it in any of the next http client versions.

 

Thanks,

 

Joan.

RE: Global lock in http pool

[Joan Balagueró]

Thanks Oleg, 
I'm using the Async Client (4.1.2), what must I do to have it available? Just 
move to HttpCore 5.0 / HttpClient 5.0?


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: martes, 10 de julio de 2018 13:22
Para: HttpClient User Discussion
Asunto: Re: Global lock in http pool

On Tue, 2018-07-10 at 12:01 +0200, Joan Balagueró wrote:
> Hello,
> 
>  
> 
> We are using the async http client 4.1.2 in our app (an api Gateway).
> Several installations are managing more than 15.000 requests 
> simultaneous opened, and now we are in the process to optimize some 
> code in the proxy.
> 
>  
> 
> Some time ago we could read a thread talking about the global lock 
> that http pool has, in order to control the max connections parameter. 
> To avoid a possible lock contention in high concurrent scenarios, Oleg 
> planned to implement a new method to control the max connections in a 
> more “relaxed”
> way, that would not use the lock at the cost to not exactly control 
> the number of connections opened at any time.
> 
>  
> 
> Our question is if there is any schedule to have this method 
> implemented, or if it’s planned to release it in any of the next http 
> client versions.
> 

It has already been implemented in HttpCore 5.0 and HttpClient 5.0

Oleg

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

That would be great.
Thanks Oleg.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: domingo, 21 de octubre de 2018 13:11
Para: HttpClient User Discussion
Asunto: Re: Migration from Async 4.1.3 to HttpClient 5

On Sat, 2018-10-20 at 20:23 +0200, Joan Balagueró wrote:
> Ok, I understand whtat you mean. Since this is a non-blocking model 
> when we send the request the thread is released, so I simply have to 
> count the time passed from I send the response until the 
> 'responseReceived' method is invoked, and if this time has been 
> exceeded just throw an exception (and avoid read the response body 
> content).
> 
> Thanks,
> Joan.
> 

Hi Joan

That will likely require an observer thread and creating a thread per request 
can be expensive. 

I will see if there is a way provide a socket timeout per request setting for 
HTTP/1.1 transport only without exposing the same setting to HTTP/2 transport.

Oleg

-Mensaje original-
> De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: sábado, 20 
> de octubre de 2018 18:17
> Para: HttpClient User Discussion
> Asunto: Re: Migration from Async 4.1.3 to HttpClient 5
> 
> On Fri, 2018-10-19 at 21:07 +0200, Joan Balagueró wrote:
> > Hello Oleg,
> > 
> > I think it's a bit more complicated ... Let me explain it:
> > 
> > - we have a pool with a response timeout of 15s
> > - this pool is shared by 2 webservices, ws1 and ws2. Ws1 uses the 
> > pool's response timeout, but ws2 uses its own response timeout of 
> > 10s.
> > - the webservice ws2 has 2 methods, m1 and m2, m1 uses the ws2 
> > timeout but m2 uses its own response timeout of 12s.
> > - and finally the method m2 is used by 2 clients, c1 and c2. c1 is a 
> > very good client so his response timeout is 20s, and c2 is very bad 
> > so he only has a 3s response timeout.
> > 
> > When we set the response timeout at request level, we do this:
> > 
> > /**
> >   * This method sets the 'responseTimeout' to this http method.
> >   * 
> >   * @param  RequestBuilder  the
> > 'RequestBuilder'
> > object
> >   * @param  responseTimeout the response
> > timeout
> > to apply
> >   * @return none
> >  */
> >  private void setTimeout(RequestBuilder rb)  {
> >   // 1. If the client has a timeout (clientResponseTimeout != -1), 
> > then set this value
> >   // 2. If the client has no timeout but the method has, then set 
> > this value.
> >   // 3. If the method has no timeout but the api has, then set this 
> > value.
> >   // 4. Otherwise set the pool's response timeout.
> >   int clientResponseTimeout =
> > this.objClient.getResponseTimeout(this.objCall.getId());
> >   int responseTimeout =  (clientResponseTimeout != -1 ?
> > clientResponseTimeout : (this.objCall.getResponseTimeout() != -1 ?
> > this.objCall.getResponseTimeout() :
> > (this.objInterface.getResponseTimeout() != -1 ?
> > this.objInterface.getResponseTimeout() :
> > this.objHttp.getResponseTimeout())) );
> >  
> > rb.setConfig(RequestConfig.copy(this.objHttp.getRequestConfig()).se
> > tS
> > ocketTimeout(responseTimeout).build());
> >  }
> > 
> > And that's my problem now. Do you think this can be solved in any 
> > way using http5?
> > 
> 
> I must admit I do not understand the reason for doing all that in the 
> first place. I also do not understand what exactly you mean by 
> response timeout. Max waiting time until a response head is received?
> 
> Oleg
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

More about migration from async4.1.3 to http5

[Joan Balagueró]

Hello,

 

Just three more questions:

 

1.   Is this the way to set a byte array to the RequestBuilder’s entity
(using a org.apache.http.nio.entity.NByteArrayEntity)?

 

org.apache.hc.client5.http.classic.methods.RequestBuilder.RequestBuilder rb
= RequestBuilder.post(uri);

org.apache.http.nio.entity.NByteArrayEntity nbae = new
NByteArrayEntity(requestBody);

nbae.setContentType(contentType);

rb.setEntity(nbae);

return rb.build();

 

 

2.   With the Async client, in order to send an async request to be
consumed for an AsyncConsumer:

 

   HttpRequest httpRequest= this.createPostRequest(uri,
responseTimeout);

HttpAsyncResponseConsumer harc = new ServerHttpResponse(…);

this.objHttp.getAsyncHttpClient().execute(new
BasicAsyncRequestProducer(this.createHttpHost(uri), httpRequest), harc,
null);

 

What request producer must I use now? I can’t find anyone that accomplish
with (
 org.
 apache.
 hc.
 core5.
 http.HttpHost,
 org.
 apache.
 hc.
 core5.
 http.Request)

 

 

3.   Regarding to the HttpAsyncResponseConsumer, what is the equivalent
interface in Http5 with the methods ‘responseReceived’, ‘consumeContent’ and
‘responseCompleted’?

 

 

I have been looking at the examples, but I couldn’t find anything similar.

 

Thanks,

 

Joan.

RE: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Hi Oleg,

Thanks a lot for all your answers. I'll work on this a bit more.

There is just one problematic thing for us. Our app is an api gateway, and one 
important piece is the response timeouts between our app and the client 
application servers. If fact, it's common the user changes these response 
timeouts from the admin console.

If now they cannot change them on the fly, that means stopping the http pool, 
recreating the ioreactor with the new timeout and starting the pool again, even 
if this takes just several seconds is something I cannot do with installations 
that are processing more than 15.000 requests per second.

The main purpose to move to HttpClient5 is the http pool with Lax Max 
connections (PoolConcurrencyPolicy.LAX), because we were detecting contention 
on the current pool due to the global lock. We don't need HTTP/2  so, from my 
ignorance, is there any way to use a Lax Pool with http1.1 in order to get this 
socket timeout at request level?

Thanks,

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: viernes, 19 de octubre de 2018 17:21
Para: HttpClient User Discussion
Asunto: Re: Migration from Async 4.1.3 to HttpClient 5

On Fri, 2018-10-19 at 15:08 +0200, Joan Balagueró wrote:
> Hello,
> 
>  
> 
> We are in the process of migrating to HttpClient5 from 
> AsyncClient4.1.3, and we have some quiestions:
> 
>  
> 
> 1.   this.rc =
> RequestConfig.custom().setAuthenticationEnabled(false).
> 
>  
> setConnectionRequestTimeout(Timeout.ofMillis(this.poolTimeout)).
> 
>  
> setConnectionRequestTimeout(this.poolTimeout, TimeUnit.MILLISECONDS).
> 
>  
> setConnectionTimeout(Timeout.ofMillis(this.connectionTimeout)).
> 
>  
> setConnectTimeout(this.connectionTimeout, TimeUnit.MILLISECONDS).
> 

#setConnectTimeout was left out by mistake. It got removed after 5.0-
beta1

https://github.com/apache/httpcomponents-client/commit/60571ae8fa89918518fed57bc5a9785362f9a39a

   
> 
>  
> 
> 2.   The
> PoolingAsyncClientConnectionManagerBuilder.setConnectionTimeToLive()
> is the
> keep alive to set to connections in this pool?
> 
> (similar to what we do in
> HttpAsyncClients.custom().setKeepAliveStrategy()?)
> 

PoolingAsyncClientConnectionManagerBuilder#setConnectionTimeToLive sets the 
_total_ time to live for any connection. No connection will be kept alive past 
its TTL (time to live). Keep-alive time is always incremental.

> 
> 3.   In previous AsyncClient, we set the setSocketTimeout at
> RequestConfig level, so on each request I could copy this base ‘rc’
> and
> modify the response timeout.
> 
> But this method does no longer exist in RequestConfig. Where can I set 
> the response timeout now for each request?
> 

With HTTP/2 a single request can no longer alter the socket timeout for
the connection shared with other message streams. One needs to set the
socket timeout while building HttpAsyncClient instances with
IOReactorConfig

https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/async/HttpAsyncClientBuilder.java#L317

>  
> 
> 4.   I imagine the PoolConcurrencyPolicy cannot be modified after
> creating the pool …
> 

No, it cannot.

>  
> 
> 5.   Before I had this method to set buffer sizes (and modify
> them on
> the fly), getting the ConnectionConfig from the pool:
> 
>  
> 
> public void setBufferInInterface(int bufferInInterface) { 
> 
>  
> this.phccm.setDefaultConnectionConfig(ConnectionConfig.custom().setBu
> fferSiz
> e(bufferInInterface).setFragmentSizeHint(bufferInInterface).build());
>  
> 
> }
> 
>  
> 
> How can I do this now?
> 

Why would you want to modify those settings at runetime?

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Hello,

 

We are in the process of migrating to HttpClient5 from AsyncClient4.1.3, and
we have some quiestions:

 

1.   this.rc = RequestConfig.custom().setAuthenticationEnabled(false).

 
setConnectionRequestTimeout(Timeout.ofMillis(this.poolTimeout)).

 
setConnectionRequestTimeout(this.poolTimeout, TimeUnit.MILLISECONDS).

 
setConnectionTimeout(Timeout.ofMillis(this.connectionTimeout)).

 
setConnectTimeout(this.connectionTimeout, TimeUnit.MILLISECONDS).

 

I wrote here the 4 possible methods ‘setConnectionx’.
The two first have the same name and different parameters, two ways to do
the same.

But the next two methods seem to be the same, but the method names are
different. Shouldn’t the fourth method be ‘setConnectionTimeout’ as well?



 

2.   The
PoolingAsyncClientConnectionManagerBuilder.setConnectionTimeToLive() is the
keep alive to set to connections in this pool?

(similar to what we do in HttpAsyncClients.custom().setKeepAliveStrategy()?)

 

3.   In previous AsyncClient, we set the setSocketTimeout at
RequestConfig level, so on each request I could copy this base ‘rc’ and
modify the response timeout.

But this method does no longer exist in RequestConfig. Where can I set the
response timeout now for each request?

 

4.   I imagine the PoolConcurrencyPolicy cannot be modified after
creating the pool …

 

5.   Before I had this method to set buffer sizes (and modify them on
the fly), getting the ConnectionConfig from the pool:

 

public void setBufferInInterface(int bufferInInterface) { 

 
this.phccm.setDefaultConnectionConfig(ConnectionConfig.custom().setBufferSiz
e(bufferInInterface).setFragmentSizeHint(bufferInInterface).build()); 

}

 

How can I do this now?

 

 

Thanks,

 

Joan.

RE: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Hello Oleg,

I think it's a bit more complicated ... Let me explain it:

- we have a pool with a response timeout of 15s
- this pool is shared by 2 webservices, ws1 and ws2. Ws1 uses the pool's 
response timeout, but ws2 uses its own response timeout of 10s.
- the webservice ws2 has 2 methods, m1 and m2, m1 uses the ws2 timeout but m2 
uses its own response timeout of 12s.
- and finally the method m2 is used by 2 clients, c1 and c2. c1 is a very good 
client so his response timeout is 20s, and c2 is very bad so he only has a 3s 
response timeout.

When we set the response timeout at request level, we do this:

/**
  * This method sets the 'responseTimeout' to this http method.
  * 
  * @param  RequestBuilder  the 'RequestBuilder' object
  * @param  responseTimeout the response timeout to apply
  * @return none
 */
 private void setTimeout(RequestBuilder rb)
 {
  // 1. If the client has a timeout (clientResponseTimeout != -1), then set 
this value
  // 2. If the client has no timeout but the method has, then set this value.
  // 3. If the method has no timeout but the api has, then set this value.
  // 4. Otherwise set the pool's response timeout.
  int clientResponseTimeout = 
this.objClient.getResponseTimeout(this.objCall.getId());
  int responseTimeout =  (clientResponseTimeout != -1 ? clientResponseTimeout : 
(this.objCall.getResponseTimeout() != -1 ? this.objCall.getResponseTimeout() : 
(this.objInterface.getResponseTimeout() != -1 ? 
this.objInterface.getResponseTimeout() : this.objHttp.getResponseTimeout())) );
  
rb.setConfig(RequestConfig.copy(this.objHttp.getRequestConfig()).setSocketTimeout(responseTimeout).build());
 }

And that's my problem now. Do you think this can be solved in any way using 
http5?

Thanks,

Joan.



-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: viernes, 19 de octubre de 2018 18:22
Para: HttpClient User Discussion
Asunto: Re: Migration from Async 4.1.3 to HttpClient 5

On Fri, 2018-10-19 at 18:01 +0200, Joan Balagueró wrote:
> Hi Oleg,
> 
> Thanks a lot for all your answers. I'll work on this a bit more.
> 
> There is just one problematic thing for us. Our app is an api gateway, 
> and one important piece is the response timeouts between our app and 
> the client application servers. If fact, it's common the user changes 
> these response timeouts from the admin console.
> 
> If now they cannot change them on the fly, that means stopping the 
> http pool, recreating the ioreactor with the new timeout and starting 
> the pool again, even if this takes just several seconds is something I 
> cannot do with installations that are processing more than 15.000 
> requests per second.
> 
> The main purpose to move to HttpClient5 is the http pool with Lax Max 
> connections (PoolConcurrencyPolicy.LAX), because we were detecting 
> contention on the current pool due to the global lock. We don't need
> HTTP/2  so, from my ignorance, is there any way to use a Lax Pool with 
> http1.1 in order to get this socket timeout at request level?
> 

No, there is not. There will be no request level socket timeout override in HC 
5.0 because even with HTTP/1.1 it makes no sense when messages are being 
pipelined. 

HC 5.0 could however provide APIs to enumerate all open connection and let 
their socket timeout get re-set. Feel free to raise a change request for this 
feature or better yet contribute it. 

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Hi Oleg,

Thanks! I'll try it asap

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: martes, 23 de octubre de 2018 9:28
Para: HttpClient User Discussion
Asunto: Re: Migration from Async 4.1.3 to HttpClient 5

On Mon, 2018-10-22 at 20:40 +0200, Joan Balagueró wrote:
> That would be great.
> Thanks Oleg.


Hi Joan

I added #responseTimeout to RequestConfig that will work the same way as socket 
timeout in 4.1 for HTTP/1.1 transport 

https://github.com/apache/httpcomponents-client/commit/75ca519314b783ba0d314e84f3a7e3488a7c968c

Oleg

> 
> -Mensaje original-
> De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: domingo, 
> 21 de octubre de 2018 13:11
> Para: HttpClient User Discussion
> Asunto: Re: Migration from Async 4.1.3 to HttpClient 5
> 
> On Sat, 2018-10-20 at 20:23 +0200, Joan Balagueró wrote:
> > Ok, I understand whtat you mean. Since this is a non-blocking model 
> > when we send the request the thread is released, so I simply have to 
> > count the time passed from I send the response until the 
> > 'responseReceived' method is invoked, and if this time has been 
> > exceeded just throw an exception (and avoid read the response body 
> > content).
> > 
> > Thanks,
> > Joan.
> > 
> 
> Hi Joan
> 
> That will likely require an observer thread and creating a thread per 
> request can be expensive.
> 
> I will see if there is a way provide a socket timeout per request 
> setting for HTTP/1.1 transport only without exposing the same setting 
> to HTTP/2 transport.
> 
> Oleg
> 
> -Mensaje original-
> > De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: sábado,
> > 20
> > de octubre de 2018 18:17
> > Para: HttpClient User Discussion
> > Asunto: Re: Migration from Async 4.1.3 to HttpClient 5
> > 
> > On Fri, 2018-10-19 at 21:07 +0200, Joan Balagueró wrote:
> > > Hello Oleg,
> > > 
> > > I think it's a bit more complicated ... Let me explain it:
> > > 
> > > - we have a pool with a response timeout of 15s
> > > - this pool is shared by 2 webservices, ws1 and ws2. Ws1 uses the 
> > > pool's response timeout, but ws2 uses its own response timeout of 
> > > 10s.
> > > - the webservice ws2 has 2 methods, m1 and m2, m1 uses the ws2 
> > > timeout but m2 uses its own response timeout of 12s.
> > > - and finally the method m2 is used by 2 clients, c1 and c2. c1 is 
> > > a very good client so his response timeout is 20s, and c2 is very 
> > > bad so he only has a 3s response timeout.
> > > 
> > > When we set the response timeout at request level, we do this:
> > > 
> > > /**
> > >   * This method sets the 'responseTimeout' to this http method.
> > >   * 
> > >   * @paramRequestBuilder  the
> > > 'RequestBuilder'
> > > object
> > >   * @paramresponseTimeout the
> > > response
> > > timeout
> > > to apply
> > >   * @return   none
> > >  */
> > >  private void setTimeout(RequestBuilder rb)  {
> > >   // 1. If the client has a timeout (clientResponseTimeout != -1), 
> > > then set this value
> > >   // 2. If the client has no timeout but the method has, then set 
> > > this value.
> > >   // 3. If the method has no timeout but the api has, then set 
> > > this value.
> > >   // 4. Otherwise set the pool's response timeout.
> > >   int clientResponseTimeout =
> > > this.objClient.getResponseTimeout(this.objCall.getId());
> > >   int responseTimeout =  (clientResponseTimeout != -1 ?
> > > clientResponseTimeout : (this.objCall.getResponseTimeout() != -1 ?
> > > this.objCall.getResponseTimeout() :
> > > (this.objInterface.getResponseTimeout() != -1 ?
> > > this.objInterface.getResponseTimeout() :
> > > this.objHttp.getResponseTimeout())) );
> > >  
> > > rb.setConfig(RequestConfig.copy(this.objHttp.getRequestConfig()).
> > > se
> > > tS
> > > ocketTimeout(responseTimeout).build());
> > >  }
> > > 
> > > And that's my problem now. Do you think this can be solved in any 
> > > way using http5?
> > > 
> > 
> > I must admit I do not understand the reason for doing all that in 
> > the first place. I also do not understand what exactly you mean by 
> > response timeout. Max waiting time until a response head is 
> > received?
> > 
> > Oleg
> >

RE: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Ok, I understand whtat you mean. Since this is a non-blocking model when we 
send the request the thread is released, so I simply have to count the time 
passed from I send the response until the 'responseReceived' method is invoked, 
and if this time has been exceeded just throw an exception (and avoid read the 
response body content).

Thanks,
Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: sábado, 20 de octubre de 2018 18:17
Para: HttpClient User Discussion
Asunto: Re: Migration from Async 4.1.3 to HttpClient 5

On Fri, 2018-10-19 at 21:07 +0200, Joan Balagueró wrote:
> Hello Oleg,
> 
> I think it's a bit more complicated ... Let me explain it:
> 
> - we have a pool with a response timeout of 15s
> - this pool is shared by 2 webservices, ws1 and ws2. Ws1 uses the 
> pool's response timeout, but ws2 uses its own response timeout of 10s.
> - the webservice ws2 has 2 methods, m1 and m2, m1 uses the ws2 timeout 
> but m2 uses its own response timeout of 12s.
> - and finally the method m2 is used by 2 clients, c1 and c2. c1 is a 
> very good client so his response timeout is 20s, and c2 is very bad so 
> he only has a 3s response timeout.
> 
> When we set the response timeout at request level, we do this:
> 
> /**
>   * This method sets the 'responseTimeout' to this http method.
>   * 
>   * @paramRequestBuilder  the 'RequestBuilder'
> object
>   * @paramresponseTimeout the response timeout
> to apply
>   * @return   none
>  */
>  private void setTimeout(RequestBuilder rb)  {
>   // 1. If the client has a timeout (clientResponseTimeout != -1), 
> then set this value
>   // 2. If the client has no timeout but the method has, then set this 
> value.
>   // 3. If the method has no timeout but the api has, then set this 
> value.
>   // 4. Otherwise set the pool's response timeout.
>   int clientResponseTimeout =
> this.objClient.getResponseTimeout(this.objCall.getId());
>   int responseTimeout =  (clientResponseTimeout != -1 ?
> clientResponseTimeout : (this.objCall.getResponseTimeout() != -1 ?
> this.objCall.getResponseTimeout() :
> (this.objInterface.getResponseTimeout() != -1 ?
> this.objInterface.getResponseTimeout() :
> this.objHttp.getResponseTimeout())) );
>  
> rb.setConfig(RequestConfig.copy(this.objHttp.getRequestConfig()).setS
> ocketTimeout(responseTimeout).build());
>  }
> 
> And that's my problem now. Do you think this can be solved in any way 
> using http5?
> 

I must admit I do not understand the reason for doing all that in the first 
place. I also do not understand what exactly you mean by response timeout. Max 
waiting time until a response head is received?  

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Hi Oleg,

I must admit I do not understand the reason for doing all that in the first 
place.
--> It doesn't matter, it's important for B2B integrations in tourism sector, 
but that's another story.

 I also do not understand what exactly you mean by response timeout. Max 
waiting time until a response head is received?  
--> Yes, if we have a response timeout of 15s this is the maximum time our app 
will wait for getting a response from the application servers once the request 
has been sent. And yes, this is when response headers start to arrive (and read 
on 'responseReceived' method).

Thanks,

Joan. 


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: sábado, 20 de octubre de 2018 18:17
Para: HttpClient User Discussion
Asunto: Re: Migration from Async 4.1.3 to HttpClient 5

On Fri, 2018-10-19 at 21:07 +0200, Joan Balagueró wrote:
> Hello Oleg,
> 
> I think it's a bit more complicated ... Let me explain it:
> 
> - we have a pool with a response timeout of 15s
> - this pool is shared by 2 webservices, ws1 and ws2. Ws1 uses the 
> pool's response timeout, but ws2 uses its own response timeout of 10s.
> - the webservice ws2 has 2 methods, m1 and m2, m1 uses the ws2 timeout 
> but m2 uses its own response timeout of 12s.
> - and finally the method m2 is used by 2 clients, c1 and c2. c1 is a 
> very good client so his response timeout is 20s, and c2 is very bad so 
> he only has a 3s response timeout.
> 
> When we set the response timeout at request level, we do this:
> 
> /**
>   * This method sets the 'responseTimeout' to this http method.
>   * 
>   * @paramRequestBuilder  the 'RequestBuilder'
> object
>   * @paramresponseTimeout the response timeout
> to apply
>   * @return   none
>  */
>  private void setTimeout(RequestBuilder rb)  {
>   // 1. If the client has a timeout (clientResponseTimeout != -1), 
> then set this value
>   // 2. If the client has no timeout but the method has, then set this 
> value.
>   // 3. If the method has no timeout but the api has, then set this 
> value.
>   // 4. Otherwise set the pool's response timeout.
>   int clientResponseTimeout =
> this.objClient.getResponseTimeout(this.objCall.getId());
>   int responseTimeout =  (clientResponseTimeout != -1 ?
> clientResponseTimeout : (this.objCall.getResponseTimeout() != -1 ?
> this.objCall.getResponseTimeout() :
> (this.objInterface.getResponseTimeout() != -1 ?
> this.objInterface.getResponseTimeout() :
> this.objHttp.getResponseTimeout())) );
>  
> rb.setConfig(RequestConfig.copy(this.objHttp.getRequestConfig()).setS
> ocketTimeout(responseTimeout).build());
>  }
> 
> And that's my problem now. Do you think this can be solved in any way 
> using http5?
> 


Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RV: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Hello Oleg,

We are finishing the migration and have the last questions:

1. If a connection is kept-alive for 30s at second 0, and after 10s is reused, 
this connection will die at second 30 or will survive until second 40?

2. Regarding the RetryHandler, below the method inherited from http 4.5 and 
modified to work with http5:

public boolean retryRequest(HttpRequest request, IOException exception, int 
executionCount, HttpContext context)  
{
  // Don't retry if max retries are reached.
  if (executionCount > this.maxExecutionCount) return false;

  // Don't retry if any of these exceptions occur.
  if (exception instanceof InterruptedIOException || exception instanceof 
UnknownHostException || exception instanceof ConnectException || exception 
instanceof SSLException) return false;

  // Retry of if this request is considered 'idempotent'.
  return (!(request instanceof HttpEntityEnclosingRequest));  
}

I understand the first two conditions are still ok (not sure if we have to add 
new exceptions on that list) but regarding the last condition,what would the 
equivalent condition be in Http5?


3. We have increased the response time of our backend (ip ended with '182') in 
order to exhaust the strict/lax pool. When this happens the pool starts to 
throw a DeadlineTimeoutException. At this moment the number of sockets in 
TIME_WAIT increases a lot until making the server unresponsive (probably 
exhausting the local ports):

 [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
99
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
101
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
98
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
25876
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
61507
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
97615

Is this the right behaviour? If Http5 cannot create new connections, so no new 
sockets are opened, why does the number of sockets in TIME_WAIT raise at those 
values?


Thanks,

Joan.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: RV: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Hi Oleg,

After applying the route in concrete now TEST5 works, but TEST3 still works in 
the same way as the previous test. I have tried TEST3 with the Async 4.1.3 and 
the result is the same, when you decrease the max connections value  the pool 
still reuses connections and does not honor the maxconn parameter.

Adding routes is not feasible in our use case, so at the moment we will provoke 
a pool restart in case a user changes the max connections in a lax pool.

Thanks for your time,

Joan.


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: sábado, 10 de noviembre de 2018 19:30
Para: HttpClient User Discussion
Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5

On Sat, 2018-11-10 at 18:02 +0100, Joan Balagueró - ventusproxy wrote:
> Hello Oleg,
> 
> Sorry, but I think I'm going to need a bit more help to finish 
> understand this ... This was my test:
> 
> a. A load of around 2000 req/s
> b. Just 1 route = http://54.38.179.182:8080 c. Every time we change 
> the max_connections value this method is
> executed:
>   public void setMaxConnections(int maxConnections)  {
> this.phccm.setMaxTotal(maxConnections);
> this.phccm.setDefaultMaxPerRoute(maxConnections);
>}


You might want to add this.phccm.setMaxPerRoute(new HttpRoute(new 
HttpHost("54.38.179.182", 8080)), maxConnections);


this.phccm.setDefaultMaxPerRoute(maxConnections)


> d. Printing stats every 1s:
>   public void printStats() {
>System.out.println("Stats total: " +
> this.phccm.getTotalStats().getLeased() + " / " + 
> this.phccm.getTotalStats().getMax());
>System.out.println("Stats route: " + this.phccm.getStats(new 
> HttpRoute(new HttpHost("54.38.179.182", 8080,
> "http"))).getLeased() + " / " + this.phccm.getStats(new HttpRoute(new 
> HttpHost("54.38.179.182", 8080, "http"))).getMax());
>   }
> 
> 
> TEST 1.   Strict pool, max connections = 1, keep-alive = 1s, pool
> timeout = 1m
>   - almost all requests end up with max connections error
>   - Stats total = Stats route = 1 / 1
>   --> So test ok.
> 
> TEST 2.   Strict pool, max connections = 5000 (value changed
> without restarting pool), keep-alive = 1s, pool timeout = 1m
>   - all requests processed ok
>   - Stats total = Stats route ~ 1030 / 5000
>   --> So test ok.
> 
> TEST 3.   Strict pool, max connections = 1 (value changed without
> restarting pool), keep-alive = 1s, pool timeout = 1m
>   - some requests processed ok, some returning max connections error
>   - Stats total = Stats route ~ n / 1, with 'n' lowering slowly from 
> 1.030 to 
>   --> It seems that even with a maxConn = 1 the pool is reusing pooled 
> connections.
> 
> TEST 4:   Lax pool, max connections = 1, POOL RESTARTED before
> sending traffic, keep-alive = 1s, pool timeout = 1m
>   - almost all requests end up with max connections error
>   - Stats total = Stats route = 1 / 1 (sometimes 2 / 1, ok because it's 
> lax).
>   --> So test ok. 
> 
> TEST 5.   Lax pool, max connections = 5000 (value changed without
> restarting pool), keep-alive = 1s, pool timeout = 1m
>   - almost all requests end up with max connections error
>   - Stats total = Stats route = 1 / 1 (sometimes 2 / 1).
> 
> 
> So my doubts are:
> 
> 1. Is TEST 3 ok? Even having pooled connections to reuse, shouldn't 
> the max_conn value have preference?
> 
> 2. Is TEST 5 ok? It seems the 'DefaultMaxPerRoute' cannot be applied 
> on the fly in a lax pool. It should have a value of 5000 but it's 
> preserving the previous value of 1 (test 4).
> 
> 
> 
> Thanks ,
> 
> Joan.
> 
> 
> 
> -Mensaje original-
> De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: viernes, 9 
> de noviembre de 2018 15:31
> Para: HttpClient User Discussion
> Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
> 
> On Fri, 2018-11-09 at 15:19 +0100, Joan Balagueró wrote:
> > Ok, so if I have a defaultMaxPerRoute = 1, and all requests I'm 
> > sending are using plain http to the same ip (without proxy) and only 
> > using 4 different ports  (8080, 8081, 8082, 8083), than this means I 
> > have 1 max connection for ip:8080, 1 for ip:8081, 1 for ip:80802 and
> > 1 for ip:80803?
> > Joan.
> > 
> 

defaultMaxPerRoute applies only once when the per route pool is created. I 
guess this is what skews the results of TEST 3 and TEST 5.

Oleg 



> Correct.
> 
> Oleg
> 
> > -Mensaje original-
> > De: Oleg Kalnichevski [mail

2 HttpClient5 questions

[Joan Balagueró]

Hello,

 

I hope the last questions:

 

1.   The validateAfterInactivity with a value of zero means no
validation?

 

2.   Regarding the consume method, this is our implementation now:

 

@Override

public int consume(ByteBuffer src) throws IOException

{

  while (src.hasRemaining()) {

 byte[] dst = new byte[src.remaining()];

 src.get(dst, 0, src.remaining());

 this.vbaos.write(dst);

  }

 

  return ;

}

 

Triggered to pass incoming data to the data consumer. The consumer must
consume the entire content of the data buffer.

è Is this the right (and optimal) way to consume the data buffer?

 

The consumer must stop incrementing its capacity on the capacity channel and
must return zero capacity from this method if it is unable to accept more
data. Once the data consumer has handled accumulated data or allocated more
intermediate storage it can update its capacity information on the capacity
channel.

è Here I must admit I don’t understand this sentence at all. After googling
a lot and reviewing your samples, I couldn’t find an example of consume
method working with byte buffers (just with CharBuffer in the
‘AsyncClientHttpExchangeStreaming.java’, but in this case no value is
returned in ‘data’ function). 

 protected void data(final CharBuffer data, final boolean
endOfStream) throws IOException {

while (data.hasRemaining()) {

 System.out.print(data.get());

}

if (endOfStream) {

 System.out.println();

}

}

 

What is the suggested value to return in this function?

 

Thanks,

 

Joan.

RE: RV: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Thanks Oleg. One more thing about the max connections with lax/strict pool. Our 
code to modify the number of max connections is:

public void setMaxConnections(int maxConnections)
 {
  this.phccm.setMaxTotal(maxConnections);
  this.phccm.setDefaultMaxPerRoute(maxConnections);
}

If I modify (on the fly) the max connections in a strict pool it works. For 
example I set a very low value and I start to receive DeadlineTimeoutException, 
when I set a higher value the error disappears. If I print the 
pool.getMaxTotal() I get the right value.

But this does not work with a lax pool. I set up a lax pool with max 
connections = 1, and no DeadlineTimeoutException is thrown (with the same 
load). When I print the maxTotal and defaultMaxPerRoute I get 0 and 1 (instead 
of 1 and 1).

Is this a bug or am I missing something?

Thanks,

Joan.


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: jueves, 8 de noviembre de 2018 11:04
Para: HttpClient User Discussion
Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5

On Wed, 2018-11-07 at 19:30 +0100, Joan Balagueró wrote:
> Hello Oleg,
> 
> We are finishing the migration and have the last questions:
> 
> 1. If a connection is kept-alive for 30s at second 0, and after 10s is 
> reused, this connection will die at second 30 or will survive until 
> second 40?

Keep-alive value is always relative to the last connection release. If you want 
to limit the absolute connection life time please use set a finite TTL (total 
time to live) value. 

> 
> 2. Regarding the RetryHandler, below the method inherited from http
> 4.5 and modified to work with http5:
> 

I would recommend using DefaultHttpRequestRetryHandler shipped with the library 
unless you have some application specific requirements.

https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160

> public boolean retryRequest(HttpRequest request, IOException 
> exception, int executionCount, HttpContext context) {
>   // Don't retry if max retries are reached.
>   if (executionCount > this.maxExecutionCount) return false;
> 
>   // Don't retry if any of these exceptions occur.
>   if (exception instanceof InterruptedIOException || exception 
> instanceof UnknownHostException || exception instanceof 
> ConnectException || exception instanceof SSLException) return false;
> 
>   // Retry of if this request is considered 'idempotent'.
>   return (!(request instanceof HttpEntityEnclosingRequest)); }
> 
> I understand the first two conditions are still ok (not sure if we 
> have to add new exceptions on that list) but regarding the last 
> condition,what would the equivalent condition be in Http5?
> 

I would suggest the following:

https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160


> 
> 3. We have increased the response time of our backend (ip ended with
> '182') in order to exhaust the strict/lax pool. When this happens the 
> pool starts to throw a DeadlineTimeoutException. At this moment the 
> number of sockets in TIME_WAIT increases a lot until making the server 
> unresponsive (probably exhausting the local ports):
> 
>  [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | 
> wc -l
> 99
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | 
> wc -l
> 101
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | 
> wc -l
> 98
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | 
> wc -l
> 25876
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | 
> wc -l
> 61507
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | 
> wc -l
> 97615
> 
> Is this the right behaviour? If Http5 cannot create new connections, 
> so no new sockets are opened, why does the number of sockets in 
> TIME_WAIT raise at those values?
> 

I believe it is. There is pretty good explanation of what the TIME_WAIT state 
represents in our old wiki: 

https://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org




-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: RV: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró]

Ok, so if I have a defaultMaxPerRoute = 1, and all requests I'm sending are 
using plain http to the same ip (without proxy) and only using 4 different 
ports  (8080, 8081, 8082, 8083), than this means I have 1 max connection for 
ip:8080, 1 for ip:8081, 1 for ip:80802 and 1 for ip:80803?
Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: viernes, 9 de noviembre de 2018 15:01
Para: HttpClient User Discussion
Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5

On Fri, 2018-11-09 at 13:39 +0100, Joan Balagueró wrote:
> Thanks Oleg. One more thing about the max connections with lax/strict 
> pool. Our code to modify the number of max connections is:
> 
> public void setMaxConnections(int maxConnections)  {
>   this.phccm.setMaxTotal(maxConnections);
>   this.phccm.setDefaultMaxPerRoute(maxConnections);
> }
> 
> If I modify (on the fly) the max connections in a strict pool it 
> works. For example I set a very low value and I start to receive 
> DeadlineTimeoutException, when I set a higher value the error 
> disappears. If I print the pool.getMaxTotal() I get the right value.
> 
> But this does not work with a lax pool. I set up a lax pool with max 
> connections = 1, and no DeadlineTimeoutException is thrown (with the 
> same load). When I print the maxTotal and defaultMaxPerRoute I get 0 
> and 1 (instead of 1 and 1).
> 
> Is this a bug or am I missing something?
> 

Max total is not enforced by the lax pool, only max per route.

Oleg


> Thanks,
> 
> Joan.
> 
> 
> -Mensaje original-
> De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: jueves, 8 
> de noviembre de 2018 11:04
> Para: HttpClient User Discussion
> Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
> 
> On Wed, 2018-11-07 at 19:30 +0100, Joan Balagueró wrote:
> > Hello Oleg,
> > 
> > We are finishing the migration and have the last questions:
> > 
> > 1. If a connection is kept-alive for 30s at second 0, and after 10s 
> > is reused, this connection will die at second 30 or will survive 
> > until second 40?
> 
> Keep-alive value is always relative to the last connection release.
> If you want to limit the absolute connection life time please use set 
> a finite TTL (total time to live) value.
> 
> > 
> > 2. Regarding the RetryHandler, below the method inherited from http
> > 4.5 and modified to work with http5:
> > 
> 
> I would recommend using DefaultHttpRequestRetryHandler shipped with 
> the library unless you have some application specific requirements.
> 
> 
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
> 
> > public boolean retryRequest(HttpRequest request, IOException 
> > exception, int executionCount, HttpContext context) {
> >   // Don't retry if max retries are reached.
> >   if (executionCount > this.maxExecutionCount) return false;
> > 
> >   // Don't retry if any of these exceptions occur.
> >   if (exception instanceof InterruptedIOException || exception 
> > instanceof UnknownHostException || exception instanceof 
> > ConnectException || exception instanceof SSLException) return false;
> > 
> >   // Retry of if this request is considered 'idempotent'.
> >   return (!(request instanceof HttpEntityEnclosingRequest)); }
> > 
> > I understand the first two conditions are still ok (not sure if we 
> > have to add new exceptions on that list) but regarding the last 
> > condition,what would the equivalent condition be in Http5?
> > 
> 
> I would suggest the following:
> 
> 
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
> 
> 
> > 
> > 3. We have increased the response time of our backend (ip ended with
> > '182') in order to exhaust the strict/lax pool. When this happens 
> > the pool starts to throw a DeadlineTimeoutException. At this moment 
> > the number of sockets in TIME_WAIT increases a lot until making the 
> > server unresponsive (probably exhausting the local ports):
> > 
> >  [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > | 
> > wc -l
> > 99
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > | 
> > wc -l
> > 101
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > | 
> > wc -l
> > 98
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > | 
> > wc -l
&g

Right way to use EntityUtils.toString()

[Joan Balagueró]

Hello,

 

This is the first time I need to use EntityUtils.toString(). My code is:

 

  try 

  {

   ( . . . )

   response = cluster.getHttpClient().performRequest(request);

   String body  = EntityUtils.toString(response.getEntity(),
GenericConstants.ENCODING_UTF8);

  

   ( . . . )

  }

  catch (Exception e) 

  {

   ( . . . )

  }

 

Is this the right way to use it? My questions:

 

1.   Do I need to check previously that response.getEntity is null
before calling EntityUtils.toString()?

2.   Do I need to close the connection in any way, or
EntityUtils.toString() is responsible to get the connection back to the pool
(even in case of exception)?

 

Thanks,

Joan.

RE: Thread named 'httpclient-main-1' not stopped when shutting down pool

[Joan Balagueró]

Hi Oleg,

I'm not able to log anything in the http log file.

I'm using log4j2, and these are my appender and logger:
 

%m%n



 


The ' httpClient5.log' is correctly created, but nothing is logged when I start 
and stop my HttpClient instance. Is the logger name 'org.apache.hc' correct?

Thanks,

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: sábado, 23 de 
febrero de 2019 17:36
Para: HttpClient User Discussion
Asunto: Re: Thread named 'httpclient-main-1' not stopped when shutting down pool

On Fri, 2019-02-22 at 13:14 +0100, Joan Balagueró wrote:
> Hello,
> 
>  
> 
> When I shutdown my ‘CloseableHttpAsyncClient’instance (that has set a
> ‘PoolingAsyncClientConnectionManager’) a thread called ‘httpclient- 
> main-1’
> is not stopped and the following message is logged in tomcat log:
> 
>  
> 
> 22-Feb-2019 13:00:59.326 ADVERTENCIA [localhost-startStop-1] 
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThrea
> ds The
> web application [ROOT] appears to have started a thread named 
> [httpclient-main-1] but has failed to stop it. This is very likely to 
> create a memory leak. Stack trace of thread:
> 
> sun.misc.Unsafe.park(Native Method)
> 
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
> .await(
> AbstractQueuedSynchronizer.java:2039)
> 
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.jav
> a:442)
> 
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.ja
> va:1074
> )
> 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
> java:11
> 34)
> 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
> .java:6
> 24)
> 
> java.lang.Thread.run(Thread.java:748)
> 
>  
> 
> Our code to shutdown:
> 
> public void stop() throws IOException
> 
> {
> 
>   this.phccm.close(CloseMode.GRACEFUL); // close the 
> ‘PoolingAsyncClientConnectionManager’’ gracefully
> 
>   this.objHttp.close(); // close the
> ‘CloseableHttpAsyncClient’instance
> 
> }
> 
>  
> 
> Is this something I’m forgetting to close or it’s a matter of the 
> Httpclient?
> 

Please post a test app reproducing the problem outside the servlet container.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org






-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Thread named 'httpclient-main-1' not stopped when shutting down pool

[Joan Balagueró]

Hello,

 

When I shutdown my ‘CloseableHttpAsyncClient’instance (that has set a
‘PoolingAsyncClientConnectionManager’) a thread called ‘httpclient-main-1’
is not stopped and the following message is logged in tomcat log:

 

22-Feb-2019 13:00:59.326 ADVERTENCIA [localhost-startStop-1]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The
web application [ROOT] appears to have started a thread named
[httpclient-main-1] but has failed to stop it. This is very likely to create
a memory leak. Stack trace of thread:

sun.misc.Unsafe.park(Native Method)

java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)

java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(
AbstractQueuedSynchronizer.java:2039)

java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074
)

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
34)

java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
24)

java.lang.Thread.run(Thread.java:748)

 

Our code to shutdown:

public void stop() throws IOException

{ 

  this.phccm.close(CloseMode.GRACEFUL); // close the
‘PoolingAsyncClientConnectionManager’’ gracefully

  this.objHttp.close(); // close the
‘CloseableHttpAsyncClient’instance 

}

 

Is this something I’m forgetting to close or it’s a matter of the
Httpclient?

 

Thanks,

 

Joan.

 

 

H4 - B3

[Joan Balagueró Valls]

Hello Oleg,

 

Do you know any idea about when you plan to release H4-beta3?

 

Many thanks,

 

Joan.

RE: QUESTION ABOUT COOKIES

[Joan Balagueró Valls]

Hello Oleg,

Thanks, after some tests this is exactly what it happens.

When I send this cookie to my servlet, I receive all its data correctly
except the max-age attribute, which is always -1.

I'm trying to set MAX_AGE with your API, and I'm getting crazy...

I've tried stdCookie.setExpiryDate(new
java.util.Date(System.currentTimeMillis() + 6));  // Expires after 60
seconds

But my servlet gets -1 (when I get cookies from HttpServletRequest with
request.getCookies()).

I supposed that I had to set the same value for MAX_AGE attribute. Then I
tried:

stdCookie.setExpiryDate(new java.util.Date(System.currentTimeMillis() +
6));
stdCookie.setAttribute(ClientCookie.MAX_AGE_ATTR, 60);  // 60 seconds

But my servlet still receives -1 in MAX_AGE.

Finally, I'm trying to set the ClientCookie.EXPIRES_ATTR with the value of
java.util.Date(System.currentTimeMillis() + 6)), but the value passed to
setAttribute is expected to be a String, and I have a java.util.Date. How
can I make this conversion?

Thanks in advance,

Joan. 




-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: martes, 07 de julio de 2009 16:18
Para: HttpClient User Discussion
Asunto: Re: QUESTION ABOUT COOKIES

On Mon, Jul 06, 2009 at 08:14:48PM +0200, Joan Balaguer? Valls wrote:
 Hello Oleg,
 
  
 
 I?m trying to send cookies to a servlet with a simple app. Following the
 tutorial:
 
  
 
 HttpContext localContext = new BasicHttpContext();
 
 CookieStore cookieStore  = new BasicCookieStore();
 
  
 BasicClientCookie stdCookie = new BasicClientCookie(name, value);
 stdCookie.setVersion(1);
 stdCookie.setDomain(.mycompany.com);
 stdCookie.setPath(/);
 stdCookie.setSecure(true);
  
 // Set attributes EXACTLY as sent by the server 
 stdCookie.setAttribute(ClientCookie.VERSION_ATTR, 1);
 stdCookie.setAttribute(ClientCookie.DOMAIN_ATTR, .mycompany.com);
 
  
 
  cookieStore.addCookie(stdCookie);
 
  localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
 
  
 
  HttpEntity entity = objHttp.execute(objPost, localContext).getEntity();
 
  
 
  
 
 This does not work (at least for me). To work, you need to add:
 
  
 
 stdCookie.setAttribute(ClientCookie.PATH, /);
  
 If I forget any of the ?set? statement, or any of the ?setAttribute?
 statement, it does not work.
  
  
 The question is: Why have we to set twice the components of the
 ?BasicClientCookie?, the first using ?stdCookie.set? and the second using
 ?stdCookie.setAttribute??
  
 And should I set ?ClientCookie.SECURE_ATTR? and
?ClientCookie.MAX_AGE_ATTR?
 ?
 


Joan

This is because some cookies set domain / path / port attributes explicitly,
while some do not, in which case values of those attributes are derived from
the properties of the origin server.

Consider the following example:

Set-Cookie: 
  stuff=very important; path=/; domain=myhost.mydomain.com; version=1
Set-Cookie: 
  stuff=very important; version=1

These two cookies are obviously different but they essentially represent the
same piece of state information if sent in response to a request for
http://myhost.mydomain.com/index.html;

 
  
 And the second part: when the servlet receives this cookie, it is resent
to
 another servlet using the same sequence of operations. But debugging, one
 can see that the cookie is not added to the cookieStore (the sentence
 ?cookieStore.addCookie(stdCookie);? does not add anything to
?cookieStore?).
 

A cookie does not get added to the cookie store only if it has expired.

Hope this helps

Oleg

 
 Can you help me?
  
  
 Thanks in advance,
  
 Joan.
 
  
 

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: QUESTION ABOUT COOKIES

[Joan Balagueró Valls]

Hello,

Here the attach with the trace. It's a post request from my local app to my
servlet.

In all tests I've always used:
bcc.setVersion(1);
bcc.setAttribute(ClientCookie.VERSION_ATTR, 1);


Thanks,

Joan.

-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: martes, 07 de julio de 2009 17:18
Para: HttpClient User Discussion
Asunto: Re: QUESTION ABOUT COOKIES

On Tue, Jul 07, 2009 at 05:04:59PM +0200, Joan Balaguer? Valls wrote:
 Hello Oleg,
 
 Thanks, after some tests this is exactly what it happens.
 
 When I send this cookie to my servlet, I receive all its data correctly
 except the max-age attribute, which is always -1.
 
 I'm trying to set MAX_AGE with your API, and I'm getting crazy...
 
 I've tried stdCookie.setExpiryDate(new
 java.util.Date(System.currentTimeMillis() + 6));  // Expires after 60
 seconds
 
 But my servlet gets -1 (when I get cookies from HttpServletRequest with
 request.getCookies()).
 
 I supposed that I had to set the same value for MAX_AGE attribute. Then I
 tried:
 
 stdCookie.setExpiryDate(new java.util.Date(System.currentTimeMillis() +
 6));
 stdCookie.setAttribute(ClientCookie.MAX_AGE_ATTR, 60);  // 60 seconds
 
 But my servlet still receives -1 in MAX_AGE.
 
 Finally, I'm trying to set the ClientCookie.EXPIRES_ATTR with the value
of
 java.util.Date(System.currentTimeMillis() + 6)), but the value passed
to
 setAttribute is expected to be a String, and I have a java.util.Date. How
 can I make this conversion?
 
 Thanks in advance,
 

Post a wire log of the session:

http://hc.apache.org/httpcomponents-client/logging.html

Also, try setting cookie version to version 1 to force the use of a RFC
compliant cookie spec and see if that makes any difference.

Oleg

 Joan. 
 
 
 
 
 -Mensaje original-
 De: Oleg Kalnichevski [mailto:ol...@apache.org] 
 Enviado el: martes, 07 de julio de 2009 16:18
 Para: HttpClient User Discussion
 Asunto: Re: QUESTION ABOUT COOKIES
 
 On Mon, Jul 06, 2009 at 08:14:48PM +0200, Joan Balaguer? Valls wrote:
  Hello Oleg,
  
   
  
  I?m trying to send cookies to a servlet with a simple app. Following the
  tutorial:
  
   
  
  HttpContext localContext = new BasicHttpContext();
  
  CookieStore cookieStore  = new BasicCookieStore();
  
   
  BasicClientCookie stdCookie = new BasicClientCookie(name, value);
  stdCookie.setVersion(1);
  stdCookie.setDomain(.mycompany.com);
  stdCookie.setPath(/);
  stdCookie.setSecure(true);
   
  // Set attributes EXACTLY as sent by the server 
  stdCookie.setAttribute(ClientCookie.VERSION_ATTR, 1);
  stdCookie.setAttribute(ClientCookie.DOMAIN_ATTR, .mycompany.com);
  
   
  
   cookieStore.addCookie(stdCookie);
  
   localContext.setAttribute(ClientContext.COOKIE_STORE, cookieStore);
  
   
  
   HttpEntity entity = objHttp.execute(objPost, localContext).getEntity();
  
   
  
   
  
  This does not work (at least for me). To work, you need to add:
  
   
  
  stdCookie.setAttribute(ClientCookie.PATH, /);
   
  If I forget any of the ?set? statement, or any of the ?setAttribute?
  statement, it does not work.
   
   
  The question is: Why have we to set twice the components of the
  ?BasicClientCookie?, the first using ?stdCookie.set? and the second
using
  ?stdCookie.setAttribute??
   
  And should I set ?ClientCookie.SECURE_ATTR? and
 ?ClientCookie.MAX_AGE_ATTR?
  ?
  
 
 
 Joan
 
 This is because some cookies set domain / path / port attributes
explicitly,
 while some do not, in which case values of those attributes are derived
from
 the properties of the origin server.
 
 Consider the following example:
 
 Set-Cookie: 
   stuff=very important; path=/; domain=myhost.mydomain.com;
version=1
 Set-Cookie: 
   stuff=very important; version=1
 
 These two cookies are obviously different but they essentially represent
the
 same piece of state information if sent in response to a request for
 http://myhost.mydomain.com/index.html;
 
  
   
  And the second part: when the servlet receives this cookie, it is resent
 to
  another servlet using the same sequence of operations. But debugging,
one
  can see that the cookie is not added to the cookieStore (the sentence
  ?cookieStore.addCookie(stdCookie);? does not add anything to
 ?cookieStore?).
  
 
 A cookie does not get added to the cookie store only if it has expired.
 
 Hope this helps
 
 Oleg
 
  
  Can you help me?
   
   
  Thanks in advance,
   
  Joan.
  
   
  
 
 -
 To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
 For additional commands, e-mail: httpclient-users-h...@hc.apache.org
 
 
 
 -
 To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
 For additional commands, e-mail: httpclient-users-h...@hc.apache.org
 

-
To unsubscribe, e-mail: 

QUESTION ABOUT COOKIES

[Joan Balagueró Valls]

Again the attach zipped.

 

Joan.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

TRYING AGAIN THE ATTACH FOR COOKIES QUESTION

[Joan Balagueró Valls]

Hello,

 

Here the attach (I hope)

 

 

Joan.

 

 

 


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Http trace for cookies

[Joan Balagueró Valls]

Hello Oleg,

 

I’m trying to send you the trace, but your mail server does not accept it.

 

I paste here a couple of lines of this trace:

 

This is the interesting part of the trace. The expiry date is correct in the
line 7 (60 seconds from now). But it seems this expiry date does not appear
in Cookie header.

 

Joan.

 

 

CookieSpec selected: best-match

Cookie [version: 1][name: testName][value: testValue][domain:
wstest.rhodasol.es][path: /wsserhs/rhodasol][expiry: Wed Jul 08 16:22:39
CEST 2009] match [wstest.rhodasol.es:81/wsserhs/rhodasol]

 

Sending request: POST /wsserhs/rhodasol HTTP/1.1

 POST /wsserhs/rhodasol HTTP/1.1

 Accept-Encoding: gzip

 Content-Length: 444

 Content-Type: text/xml

 Host: wstest.rhodasol.es:81

 Connection: Keep-Alive

 Cookie: $Version=1; testName=testValue; $Path=/wsserhs/rhodasol;
$Domain=wstest.rhodasol.es

 Cookie2: $Version=1

Receiving response: HTTP/1.1 200 OK

 HTTP/1.1 200 OK

 

RE: Http trace for cookies

[Joan Balagueró Valls]

Hello Oleg,

Then, how is it possible that I always receive -1 in maxAge attribute? Am
I forgetting something, or doing something wrong?

If I try to set the MAX_AGE_ATTR and the EXPIRES_ATTR, I get -1 again.
How is the correct way to propagate the cookie expiry date?

Thanks in advance,

Joan.


Date expires = new Date(System.currentTimeMillis() + 6);
   
BasicClientCookie bcc = new BasicClientCookie(testName, testValue);
bcc.setVersion(1);
bcc.setDomain(wstest.rhodasol.es);
bcc.setPath(/wsserhs/rhodasol);
bcc.setSecure(false);
bcc.setExpiryDate(expires);
 
bcc.setAttribute(ClientCookie.VERSION_ATTR, 1);
bcc.setAttribute(ClientCookie.DOMAIN_ATTR, wstest.rhodasol.es);
bcc.setAttribute(ClientCookie.PATH_ATTR, /wsserhs/rhodasol);
bcc.setAttribute(ClientCookie.MAX_AGE_ATTR, 60);
bcc.setAttribute(ClientCookie.EXPIRES_ATTR, DateUtils.formatDate(expires));



-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: miércoles, 08 de julio de 2009 23:54
Para: HttpClient User Discussion
Asunto: Re: Http trace for cookies

Joan Balagueró Valls wrote:
 Hello Oleg,
 
  
 
 I’m trying to send you the trace, but your mail server does not accept it.
 
  
 
 I paste here a couple of lines of this trace:
 
  
 
 This is the interesting part of the trace. The expiry date is correct in
the
 line 7 (60 seconds from now). But it seems this expiry date does not
appear
 in Cookie header.
 
  
 
 Joan.
 
  
 
  
 
 CookieSpec selected: best-match
 
 Cookie [version: 1][name: testName][value: testValue][domain:
 wstest.rhodasol.es][path: /wsserhs/rhodasol][expiry: Wed Jul 08 16:22:39
 CEST 2009] match [wstest.rhodasol.es:81/wsserhs/rhodasol]
 
  
 
 Sending request: POST /wsserhs/rhodasol HTTP/1.1
 
  POST /wsserhs/rhodasol HTTP/1.1
 
  Accept-Encoding: gzip
 
  Content-Length: 444
 
  Content-Type: text/xml
 
  Host: wstest.rhodasol.es:81
 
  Connection: Keep-Alive
 
  Cookie: $Version=1; testName=testValue; $Path=/wsserhs/rhodasol;
 $Domain=wstest.rhodasol.es
 
  Cookie2: $Version=1
 
 Receiving response: HTTP/1.1 200 OK
 
  HTTP/1.1 200 OK
 
  

I am sorry, Joan, but I see nothing wrong with the cookie. The cookie 
looks perfectly valid to me.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

EXPECT-CONTINUE HANDSHAKE

[Joan Balagueró Valls]

Hello Oleg,

Can I set the expect-continue handshake at httpclient level?

  HttpParams objHttpParams = new BasicHttpParams();
  HttpProtocolParams.setVersion(objHttpParams, HttpVersion.HTTP_1_1);
  ClientConnectionManager cm = new
ThreadSafeClientConnManager(this.objHttpParams,
HttpUtils.createDefaultSchemeRegistry());

  this.objHttp = new DefaultHttpClient(cm, this.objHttpParams);

-- this.objHttpParams.setParameter(CoreProtocolPNames.USE_EXPECT_CONTINUE,

 Boolean.FALSE);

Or should I set at request level?

Thanks,

Joan.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RE: RV: NO_HTTP_RESPONSE_EXCEPTION

[Joan Balagueró Valls]

Hello Oleg,

I attach the http log. I've been taking a look to this trace, and I've
observed the following. I don't know if it's important or not.

In line 53315, we have a request that is correctly processed. When
HttpClient tries to get a connection from pool, the information is:

Total connections kept alive: 2
Total issued connections: 0
Total allocated connection: 2 out of 100
Getting free connection [HttpRoute[{}-http://192.168.38.87:80]][null]


The following request fails with noHttpResponse (starts at line 53362). In
this request, when HttpClient tries to get a connection from pool, the
information is:

Total connections kept alive: 11
Total issued connections: 0
Total allocated connection: 11 out of 100
Getting free connection [HttpRoute[{}-http://192.168.46.152:11003]][null]


Is it normal that this request shows a pool with 11 allocated connections
when the previous request only shows 2?

Here, httpNoResponseException starts until the pool is empty. You can see 11
consecutive erroneous requests with Connection closed and Released
connection is not reusable UNTIL total allocated connection is 0.

When total allocated connection is 0, HttpClient has to create a new
connection (line 53713), and then everything works again.

Well, exactly the same situation is reproduced starting at line 66610.
In this line, the request goes OK. The information is:

Total connections kept alive: 2
Total issued connections: 0
Total allocated connection: 2 out of 100
Getting free connection [HttpRoute[{}-http://10.12.112.180:12003]][null]


The following request again fails, and the information is:
Total connections kept alive: 10
Total issued connections: 0
Total allocated connection: 10 out of 100
Getting free connection [HttpRoute[{}-http://192.168.46.152:11003]][null]

Again, we can see 10 consecutive erroneous requests, until allocated
connections comes 0 again. In this point, everything starts to work again
when Httpclient has to create a new connection.


I try to attach the log file as a zip file. Last time it was impossible to
send it to you. If you don't receive it, tell how I can send it.


Thanks for your time, Oleg.

Joan.


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: miércoles, 22 de julio de 2009 14:41
Para: HttpClient User Discussion
Asunto: Re: RV: NO_HTTP_RESPONSE_EXCEPTION

On Wed, Jul 22, 2009 at 02:17:17PM +0200, Joan Balaguer? Valls wrote:
 Hello Oleg,
 
 Five months ago, I sent you the emails below because I was experiencing a
 lot of NoHttpResponseException errors when sending requests to a couple of
 servers.
 
 Yesterday night, I installed my app with H4 in one of my clients. This
 morning I've had to move back to H3 because of these errors.
 
 The app consists of a proxy that is sending requests to 3 different
 webservices. The app has a different connection pool for each webservice,
 and the target servers are also different for each one. 
 
 App   - requests to Webservice1 -- uses pool1 -- send to serversA,B 
   - requests to Webservice2 -- uses pool2 -- send to serversC,D 
   - requests to Webservice3 -- uses pool3 -- send to serversE,F
 
 


Joan,

Please post a context / wire log of the HTTP session that exhibits the
problem

http://hc.apache.org/httpcomponents-client/logging.html

Please also consider using the latest HttpClient 4.0 RC build:

http://people.apache.org/~olegk/httpclient-4.0-rc2/

Mind you it is very unlikely this has anything to do with HttpClient, so
expect no miracles.

Oleg



 The results for webservice1 are:
 
 Tuesday 22/07:HttpClient 3.1. Total Requests sent :
175.003
 From 00:00 to 23:59   OK Requests :
 174.936
   Error Requests: :
 67
 Wednes. 23/07:HttpClient 4.0  Total Requests sent :
54.851
 From 00:00 to 13:30   OK Requests :
 51.183
   Error Requests: :
 3.668
 
 From these 3.668 errors, 3.647 corresponding to NoHttpResponseException.
 
 
 
 The error percentages are similar in ws2 and ws3. The target servers for
the
 ws1 and ws2 are in the same LAN than my app, while target servers for w3
are
 on the Internet.
 
 After moving back to H3, again 0 errors in the last 2 hours.
 
 Please, just tell me what you need to get more information about this
 problem (traces, configurations, ...). Anything you need.
 
 
 Thanks in advance,
 
 Joan.
 
 -Mensaje original-
 De: Oleg Kalnichevski [mailto:ol...@apache.org] 
 Enviado el: s?bado, 07 de febrero de 2009 14:08
 Para: HttpClient User Discussion
 Asunto: Re: NO_HTTP_RESPONSE_EXCEPTION
 
 sebb wrote:
  On 06/02/2009, Oleg Kalnichevski ol...@apache.org wrote:
  Joan Balaguer? wrote:
 
  Thanks Oleg. It seems strange to me because HttpClient3 (sending
 requests
  to
  the same server) never fails. I'll take a look at the code again and
 I'll
  comment you 

RE: RV: NO_HTTP_RESPONSE_EXCEPTION

[Joan Balagueró Valls]

Hello Oleg,

Ok, then, in your opinion, it seems that some connections were closed on the
server side, but for H4 they are OK, and when H4 tries to use them, then a
NoHttpResponseException is thrown.

In fact, I've implemented an IdleConnectionsHandler that closes expired
connections and idle connections (with a timeout). Now, this process is
executed every 5 minutes, and closes expired connections and connections
idle for more than 60 seconds.

I suppose that this is too much wait time. What values do you recommend?

Another question: the same app works perfectly with Http3. What does it
mean? Is H4 much more sensible to server closed connections than H3?
I've never used the stale check (H3 or H4).

Thanks,

Joan.


-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: miércoles, 22 de julio de 2009 18:42
Para: HttpClient User Discussion
Asunto: Re: RV: NO_HTTP_RESPONSE_EXCEPTION

On Wed, Jul 22, 2009 at 06:06:25PM +0200, Joan Balaguer? Valls wrote:
 Hello Oleg,
 
 I attach the http log.

Hi Joan,

Unfortunately the log did not get through


 I've been taking a look to this trace, and I've
 observed the following. I don't know if it's important or not.
 
 In line 53315, we have a request that is correctly processed. When
 HttpClient tries to get a connection from pool, the information is:
 
 Total connections kept alive: 2
 Total issued connections: 0
 Total allocated connection: 2 out of 100
 Getting free connection [HttpRoute[{}-http://192.168.38.87:80]][null]
 
 
 The following request fails with noHttpResponse (starts at line 53362). In
 this request, when HttpClient tries to get a connection from pool, the
 information is:
 
 Total connections kept alive: 11
 Total issued connections: 0
 Total allocated connection: 11 out of 100
 Getting free connection [HttpRoute[{}-http://192.168.46.152:11003]][null]
 
 
 Is it normal that this request shows a pool with 11 allocated connections
 when the previous request only shows 2?


It is normal. Those two requests have _completely_ different routes. 

 
 Here, httpNoResponseException starts until the pool is empty. You can see
11
 consecutive erroneous requests with Connection closed and Released
 connection is not reusable UNTIL total allocated connection is 0.
 
 When total allocated connection is 0, HttpClient has to create a new
 connection (line 53713), and then everything works again.
 

It is all very simple. Those 11 connections have most likely been idle for
two
long and therefore got dropped on the server side. You should either use the
stale connection check or implement some sort of a stale connection eviction
policy to make sure HttpClient does not try to re-use them. For details see

http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e638

Hope this helps

Oleg




 Well, exactly the same situation is reproduced starting at line 66610.
 In this line, the request goes OK. The information is:
 
 Total connections kept alive: 2
 Total issued connections: 0
 Total allocated connection: 2 out of 100
 Getting free connection [HttpRoute[{}-http://10.12.112.180:12003]][null]
 
 
 The following request again fails, and the information is:
 Total connections kept alive: 10
 Total issued connections: 0
 Total allocated connection: 10 out of 100
 Getting free connection [HttpRoute[{}-http://192.168.46.152:11003]][null]
 
 Again, we can see 10 consecutive erroneous requests, until allocated
 connections comes 0 again. In this point, everything starts to work again
 when Httpclient has to create a new connection.
 
 
 I try to attach the log file as a zip file. Last time it was impossible to
 send it to you. If you don't receive it, tell how I can send it.
 
 
 Thanks for your time, Oleg.
 
 Joan.
 
 
 -Mensaje original-
 De: Oleg Kalnichevski [mailto:ol...@apache.org] 
 Enviado el: mi?rcoles, 22 de julio de 2009 14:41
 Para: HttpClient User Discussion
 Asunto: Re: RV: NO_HTTP_RESPONSE_EXCEPTION
 
 On Wed, Jul 22, 2009 at 02:17:17PM +0200, Joan Balaguer? Valls wrote:
  Hello Oleg,
  
  Five months ago, I sent you the emails below because I was experiencing
a
  lot of NoHttpResponseException errors when sending requests to a couple
of
  servers.
  
  Yesterday night, I installed my app with H4 in one of my clients. This
  morning I've had to move back to H3 because of these errors.
  
  The app consists of a proxy that is sending requests to 3 different
  webservices. The app has a different connection pool for each
webservice,
  and the target servers are also different for each one. 
  
  App - requests to Webservice1 -- uses pool1 -- send to serversA,B 
  - requests to Webservice2 -- uses pool2 -- send to serversC,D 
  - requests to Webservice3 -- uses pool3 -- send to serversE,F
  
  
 
 
 Joan,
 
 Please post a context / wire log of the HTTP session that exhibits the
 problem
 
 http://hc.apache.org/httpcomponents-client/logging.html
 
 Please also consider using the latest HttpClient 4.0 

Re: HttpClient 4.0

[Joan Balagueró Valls]

--Mensaje original--
De: visualize
Para:httpclient-users@hc.apache.org
Responder a:HttpClient User Discussion
Asunto: Re: HttpClient 4.0
Enviado: 28 Sep, 2009 11:06



olegk wrote:
 
 visualize wrote:
 Hello,
 
 I'm trying to port my application to use HttpClient 4.0 instead of
 version
 3.0 I make a substantial number of HTTP-gets and used to use
 aMultiThreadedHttpConnectionManager to setup a pool and retrieve clients. 
 
 Let's say I want to get some stuff from
 http://my.server.com:8080/my-rest-service/data; and
 http://my.server.com:8080/my-rest-service/data2;.
 
 In 3.0 you are able to set host and port via
 getHostConfiguration().setHost(host, port) which was kind of 
 convenient since I then could just create the pool, and when getting
 clients
 just set the host and port to http://my.server.com; and 8080. In each
 request, I just had to set the URL to /my-rest-service/data or
 /my-rest-service/data2 and didn't have to care about the host and port.
 
 Is there a way to accomplish the same with a client of version 4.0? Like
 set
 a default host/route (or whatever you may call it), so when doing new
 HttpGet(/my-rest-service/data) it falls back to the default one?
 
 / K
 
 
 
 
 Use http.default-host parameter.
 
 -
 DefaultHttpClient httpclient = new DefaultHttpClient();
 
 httpclient.getParams().setParameter(ClientPNames.DEFAULT_HOST,
  new HttpHost(www.google.com, 80, http));
 
 HttpGet req = new HttpGet(/);
 
 HttpResponse rsp = httpclient.execute(req);
 HttpEntity entity = rsp.getEntity();
 
 System.out.println();
 System.out.println(rsp.getStatusLine());
 System.out.println();
 
 if (entity != null) {
  System.out.println(EntityUtils.toString(entity));
 }
 -
 
 Hope this helps
 
 Oleg
 
 -
 To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
 For additional commands, e-mail: httpclient-users-h...@hc.apache.org
 
 
 

Fantastic, thanks a lot!
-- 
View this message in context: 
http://www.nabble.com/HttpClient-4.0-tp25531034p25641698.html
Sent from the HttpClient-User mailing list archive at Nabble.com.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Enviado desde mi dispositivo BlackBerry® de Orange.

NoHttpResponseException in the async client

[Joan Balagueró - ventusproxy]

Hello,

We have replaced the httpclient by the async client in our application.
Everything works fine, but the ‘NoHttpResponseException’ has disappeared
from our error statistics reports. So, or with the new async pool this error
does not occur for some reason (that I don’t know) or we are not catching
this error correctly (more likely).

We are using an ‘HttpAsyncResponseConsumer’ and overwriting the
‘responseReceived’, ‘consumeContent’ and ‘failed’ methods. We
understand that when a ‘NoHttpResponseException’ occurs,
‘responseReceived’ and ‘consumeContent’ are not called, and  the
‘failed’ method is the only one that is directly called.

Our ‘failed’ method looks like this:

@Override
public void failed(final Exception e)
{
  ProxyServletException pse = null;
  
  if (e.getClass() == java.net.SocketTimeoutException.class) pse = new
ProxyServletException(ErrorConstants.HTTP_RESPONSE_TIMEOUT, e);
  else if (e.getClass() == java.net.ConnectException.class) pse = new
ProxyServletException(ErrorConstants.HTTP_CONNECT_TIMEOUT, e);
  else if (e.getClass() == org.apache.http.NoHttpResponseException.class)
pse = new ProxyServletException(ErrorConstants.HTTP_NO_RESPONSE, e);←
the error is caugth here 
  else if (e.getClass() == java.io.IOException.class) pse = new
ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e);
  else if (e.getClass() ==
com.ventusproxy.proxy.servlet.ProxyServletException.class) pse =
(ProxyServletException)e;
  else if (e.getClass() ==
org.apache.http.conn.ConnectionPoolTimeoutException.class) pse = new
ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e);
  else if (e.getClass() == java.util.concurrent.TimeoutException.class) pse
= new ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e);

  pse = (pse != null ? pse : new
ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e));

  ( . . . )
}


Is this ok? Or I'm missing something?

Thanks,

Joan.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

RV: Migration from Async 4.1.3 to HttpClient 5

[Joan Balagueró - ventusproxy]

Hello Oleg,

Sorry, but I think I'm going to need a bit more help to finish understand this 
... This was my test:

a. A load of around 2000 req/s
b. Just 1 route = http://54.38.179.182:8080
c. Every time we change the max_connections value this method is executed:
public void setMaxConnections(int maxConnections)  {
  this.phccm.setMaxTotal(maxConnections);
  this.phccm.setDefaultMaxPerRoute(maxConnections);
 }
d. Printing stats every 1s:
public void printStats() {
 System.out.println("Stats total: " + 
this.phccm.getTotalStats().getLeased() + " / " + 
this.phccm.getTotalStats().getMax());
 System.out.println("Stats route: " + this.phccm.getStats(new 
HttpRoute(new HttpHost("54.38.179.182", 8080, "http"))).getLeased() + " / " + 
this.phccm.getStats(new HttpRoute(new HttpHost("54.38.179.182", 8080, 
"http"))).getMax());
}


TEST 1. Strict pool, max connections = 1, keep-alive = 1s, pool timeout = 1m
- almost all requests end up with max connections error
- Stats total = Stats route = 1 / 1
--> So test ok.

TEST 2. Strict pool, max connections = 5000 (value changed without restarting 
pool), keep-alive = 1s, pool timeout = 1m
- all requests processed ok
- Stats total = Stats route ~ 1030 / 5000
--> So test ok.

TEST 3. Strict pool, max connections = 1 (value changed without restarting 
pool), keep-alive = 1s, pool timeout = 1m
- some requests processed ok, some returning max connections error
- Stats total = Stats route ~ n / 1, with 'n' lowering slowly from 
1.030 to 
--> It seems that even with a maxConn = 1 the pool is reusing pooled 
connections.

TEST 4: Lax pool, max connections = 1, POOL RESTARTED before sending traffic, 
keep-alive = 1s, pool timeout = 1m
- almost all requests end up with max connections error
- Stats total = Stats route = 1 / 1 (sometimes 2 / 1, ok because it's 
lax).
--> So test ok. 

TEST 5. Lax pool, max connections = 5000 (value changed without restarting 
pool), keep-alive = 1s, pool timeout = 1m
- almost all requests end up with max connections error
- Stats total = Stats route = 1 / 1 (sometimes 2 / 1).


So my doubts are:

1. Is TEST 3 ok? Even having pooled connections to reuse, shouldn't the 
max_conn value have preference? 

2. Is TEST 5 ok? It seems the 'DefaultMaxPerRoute' cannot be applied on the fly 
in a lax pool. It should have a value of 5000 but it's preserving the previous 
value of 1 (test 4).



Thanks ,

Joan.



-Mensaje original-
De: Oleg Kalnichevski [mailto:ol...@apache.org] 
Enviado el: viernes, 9 de noviembre de 2018 15:31
Para: HttpClient User Discussion
Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5

On Fri, 2018-11-09 at 15:19 +0100, Joan Balagueró wrote:
> Ok, so if I have a defaultMaxPerRoute = 1, and all requests I'm 
> sending are using plain http to the same ip (without proxy) and only 
> using 4 different ports  (8080, 8081, 8082, 8083), than this means I 
> have 1 max connection for ip:8080, 1 for ip:8081, 1 for ip:80802 and
> 1 for ip:80803?
> Joan.
> 

Correct.

Oleg

> -Mensaje original-
> De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: viernes, 9 
> de noviembre de 2018 15:01
> Para: HttpClient User Discussion
> Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
> 
> On Fri, 2018-11-09 at 13:39 +0100, Joan Balagueró wrote:
> > Thanks Oleg. One more thing about the max connections with 
> > lax/strict pool. Our code to modify the number of max connections 
> > is:
> > 
> > public void setMaxConnections(int maxConnections)  {
> >   this.phccm.setMaxTotal(maxConnections);
> >   this.phccm.setDefaultMaxPerRoute(maxConnections);
> > }
> > 
> > If I modify (on the fly) the max connections in a strict pool it 
> > works. For example I set a very low value and I start to receive 
> > DeadlineTimeoutException, when I set a higher value the error 
> > disappears. If I print the pool.getMaxTotal() I get the right value.
> > 
> > But this does not work with a lax pool. I set up a lax pool with max 
> > connections = 1, and no DeadlineTimeoutException is thrown (with the 
> > same load). When I print the maxTotal and defaultMaxPerRoute I get
> > 0
> > and 1 (instead of 1 and 1).
> > 
> > Is this a bug or am I missing something?
> > 
> 
> Max total is not enforced by the lax pool, only max per route.
> 
> Oleg
> 
> 
> > Thanks,
> > 
> > Joan.
> > 
> > 
> > -Mensaje original-
> > De: Oleg Kalnichevski [mailto:ol...@apache.org] Enviado el: jueves,
> > 8
&