Re: HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
Hello Oleg,
We have one enabled using:
new DefaultHttpRequestRetryHandler(RETRY_COUNT, false)
RETRY_COUNT is set to 0 by default but can be changed.
Do you mean we should not set it to 0 ?
Regards
Philippe
On Sat, Feb 15, 2014 at 4:39 PM, Oleg Kalnichevski ol...@apache.org wrote:
On Sat, 2014-02-15 at 16:01 +0100, Philippe Mouawad wrote:
By default we disable retry but it can be controlled by a parameter.
I have say you may want to have a retry handler enabled by default if
the stale check is off. At the very least you may want to retry
idempotent requests or some specific type of exceptions (like 'server
failed to respond').
Oleg
Thanks for the info on ConnectionReuseStrategy, it will be a way to
handle
these kind of issues.
Regards
On Sat, Feb 15, 2014 at 3:12 PM, Oleg Kalnichevski
ol...@apache.orgjavascript:_e(%7B%7D,'cvml','ol...@apache.org');
wrote:
On Sat, 2014-02-15 at 14:46 +0100, Philippe Mouawad wrote:
Hello Oleg,
The problem is that is seems in this particular case, removing this
check
it leads to 50% error.
What kind of HttpRequestRetryHandler implementation does JMeter 2.11
use? Usually with the stale check disabled one should want to retry at
least some requests automatically.
I could be explained by a max number of requests per connection
setting,
as
per their documentation:
- http://aws.amazon.com/articles/1904
Also, don't overuse a connection. Amazon S3 will accept up to 100
requests
before it closes a connection (resulting in 'connection reset').
Rather
than having this happen, use a connection for 80-90 requests before
closing
and re-opening a new connection.
Is there by the way a configuration parameter in HttpClient to limit
the
number of requests per connection ?
There is no parameter but one can use a custom ConnectionReuseStrategy
to that effect
---
ConnectionReuseStrategy reuseStrategy = new
DefaultConnectionReuseStrategy() {
@Override
public boolean keepAlive(
final HttpResponse response, final HttpContext context) {
HttpConnection conn = (HttpConnection)
context.getAttribute(HttpClientContext.HTTP_CONNECTION);
long count = conn.getMetrics().getRequestCount();
if (count = 100) {
return false;
}
return super.keepAlive(response, context);
}
};
---
Hope this helps
Oleg
Thanks
Regards
Philippe
On Sat, Feb 15, 2014 at 12:49 PM, Oleg Kalnichevski
ol...@apache.orgjavascript:_e(%7B%7D,'cvml','ol...@apache.org');
wrote:
On Sat, 2014-02-15 at 11:23 +0100, Philippe Mouawad wrote:
Hello ,
yes that's it.
regards
Then, I am not sure I understand the problem. The stale connection
check
is about trading off some performance for fewer i/o errors or visa
versa.
Oleg
On Saturday, February 15, 2014, Oleg Kalnichevski
o...@ok2consulting.comjavascript:_e(%7B%7D,'cvml','
o...@ok2consulting.com');
wrote:
On Fri, 2014-02-14 at 15:42 +0100, Philippe Mouawad wrote:
Hello Oleg,
We set this configuration in JMeter 2.11, we got recently
this
bug
report
related to this change:
https://issues.apache.org/bugzilla/show_bug.cgi?id=56119
This issue seems to be faced by another person:
https://twitter.com/cfwhisperer/status/428278488349417472
Regards
Philippe
Philippe
I am not sure I remember the context. Is it about turning off
stale
connection checking?
Oleg
On Wed, Dec 11, 2013 at 9:50 AM, Oleg Kalnichevski
ol...@apache.org javascript:_e(%7B%7D,'cvml','ol...@apache.org');
javascript:;
wrote:
On Tue, 2013-12-10 at 21:38 +0100, Philippe Mouawad wrote:
Hello Oleg,
Thanks for answer.
Wouldn't it be interesting to add some threshold or delay
between
each
check instead of doing check on each request ?
Regards
Philippe
Philippe
I am not sure it is worth the trouble. In most
circumstances
the
stale
connection check should be turned off anyway.
I think it just needs to be better documented.
Oleg
On Mon, Dec 9, 2013 at 11:59 AM, Oleg Kalnichevski
ol...@apache.org javascript:_e(%7B%7D,'cvml','ol...@apache.org
');
javascript:;
wrote:
On Sun, 2013-12-08 at 21:32 +0100, Philippe Mouawad
wrote:
Hello,
Profiling JMeter, I noticed an important number of
SocketTimeoutException
being triggered
How can I abort an aynsc request using HttpAsyncClient
Hi I would like to use HttpAsyncClient in order to execute async requests. Is it possible to abort an executing request from another thread? Thanks, -- Yoram Dayagi (Gmail) Sent with Airmail
Redirect erases header?
Hi I’m using HttpAsyncClient to execute an async request. I noticed that when requesting a GET request for a URL that redirects to another, a header I put in the request (Range: bytes=0-1000) is ignored and all content is returned in the response. Notes: 1. When accessing directly the redirected URL the execution is as expected (only first 1000 bytes are returned) 2. The response of the original URL doesn’t contain a header Accept-Ranges: bytes”, where as the response of the redirected URL does contain this header. How can I overcome this problem? -- Yoram Dayagi (Gmail) Sent with Airmail
Re: HttpClient 4.2.6 : ManagedClientConnectionImpl way of checking stale connection and Performances
On Sun, 2014-02-16 at 14:15 +0100, Philippe Mouawad wrote: Hello Oleg, We have one enabled using: new DefaultHttpRequestRetryHandler(RETRY_COUNT, false) RETRY_COUNT is set to 0 by default but can be changed. Do you mean we should not set it to 0 ? This obviously needs to be a positive number if you want requests retried automatically. Oleg - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How to know which port is newly created for the connection ?
On 2/14/2014 2:15 PM, Kiran Chitturi wrote:
I am using http client 4.3.2 and I have a few questions.
1) I am making lot of concurrent requests to a server. When creating
requests, I want to debug and check which ports are newly created by the
client. Currently, I can see the ports in netstat as pasted below.
tcp4 0 0 172.16.11.138.7575 172.16.11.138.52014ESTABLISHED
I enabled the debug logging for MainClientExec class and it gives me the
below information. Here it does not exactly tell me what is the source port
that is being used for the connection. How can I use this ? Also, whenever
MainClientExec prints the below line, does it mean a new connection is
created with a new port or an existing port is being used ?
2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection
{}-http://172.16.11.138:8983
2) My code looks like this for creating a http Client. I pass the client to
SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ which
uses this client to make requests. SolrJ makes requests to a Solr server.
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(100);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
final CloseableHttpClient httpClient = httpBuilder.build();
This httpClient created above also throws some exceptions. Can I assume retry
is successful and does this happen because I enabled socketReuseAddress ?
2014-02-14 12:53:21,415 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:53:21,443 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@106] - Retrying request
When it comes to the HttpClient side of this, I don't have much idea
what's going on ... but I do have a little bit of experience with SolrJ.
I have run into a lot of problems when trying to fully migrate Solr to
HttpClient 4.3, most of which are likely a result of my own
inexperience. Most of SolrJ's interaction with HttpClient is now
deprecated.
https://issues.apache.org/jira/browse/SOLR-5604
For my own SolrJ code, I couldn't get it to work when I used
HttpClientBuilder to make the HttpClient object for SolrJ. I don't
remember what the exceptions looked like. What I ended up doing to
avoid deprecations in my own code was using the utilities built into
SolrJ. I basically kicked the problem upstream. Here's how I build an
HttpClient in my own SolrJ code now, one that is shared between all my
HttpSolrServer instances:
ModifiableSolrParams params = new ModifiableSolrParams();
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 300);
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, 5000);
httpClient = HttpClientUtil.createClient(params);
https://lucene.apache.org/solr/4_6_0/solr-solrj/org/apache/solr/client/solrj/impl/HttpClientUtil.html
Most of the options you have above in your code are available in the
Solr utility class. Note that the object built under all these layers
is currently the deprecated SystemDefaultHttpClient class, which is
modifiable after it is created. The new objects in HttpClient 4.3 are
NOT modifiable after creation.
There's still a lot of work left to do for SOLR-5604. Until that work
is done, I don't think you'll have much luck using the new classes
introduced by HttpClient 4.3.x.
Thanks,
Shawn
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How to know which port is newly created for the connection ?
On 2/16/14 8:23 PM, Shawn Heisey s...@elyograg.org wrote:
On 2/14/2014 2:15 PM, Kiran Chitturi wrote:
I am using http client 4.3.2 and I have a few questions.
1) I am making lot of concurrent requests to a server. When creating
requests, I want to debug and check which ports are newly created by the
client. Currently, I can see the ports in netstat as pasted below.
tcp4 0 0 172.16.11.138.7575 172.16.11.138.52014
ESTABLISHED
I enabled the debug logging for MainClientExec class and it gives me
the below information. Here it does not exactly tell me what is the
source port that is being used for the connection. How can I use this ?
Also, whenever MainClientExec prints the below line, does it mean a new
connection is created with a new port or an existing port is being used ?
2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection
{}-http://172.16.11.138:8983
2) My code looks like this for creating a http Client. I pass the
client to
SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ
which uses this client to make requests. SolrJ makes requests to a Solr
server.
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(100);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
final CloseableHttpClient httpClient = httpBuilder.build();
This httpClient created above also throws some exceptions. Can I assume
retry is successful and does this happen because I enabled
socketReuseAddress ?
2014-02-14 12:53:21,415 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:53:21,443 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@106] - Retrying request
When it comes to the HttpClient side of this, I don't have much idea
what's going on ... but I do have a little bit of experience with SolrJ.
I have run into a lot of problems when trying to fully migrate Solr to
HttpClient 4.3, most of which are likely a result of my own
inexperience. Most of SolrJ's interaction with HttpClient is now
deprecated.
https://issues.apache.org/jira/browse/SOLR-5604
For my own SolrJ code, I couldn't get it to work when I used
HttpClientBuilder to make the HttpClient object for SolrJ. I don't
remember what the exceptions looked like. What I ended up doing to
avoid deprecations in my own code was using the utilities built into
SolrJ. I basically kicked the problem upstream. Here's how I build an
HttpClient in my own SolrJ code now, one that is shared between all my
HttpSolrServer instances:
ModifiableSolrParams params = new ModifiableSolrParams();
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 300);
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, 5000);
httpClient = HttpClientUtil.createClient(params);
https://lucene.apache.org/solr/4_6_0/solr-solrj/org/apache/solr/client/sol
rj/impl/HttpClientUtil.html
Most of the options you have above in your code are available in the
Solr utility class. Note that the object built under all these layers
is currently the deprecated SystemDefaultHttpClient class, which is
modifiable after it is created. The new objects in HttpClient 4.3 are
NOT modifiable after creation.
Shawn,
Thank you so much for your reply. If I use the 'SystemDefaultHttpClient'
for creating http client, there are many options for configuring the http
client like socket reuse, disabling stale check, etc.. that are not
possible to configure through SolrJ. May be after SOLR-5604, there will be
more options through SolrJ for configuring or one could pass their own
http client with the configured parameters.
If I use Http client from 4.3.x series, do you think there is code inside
SolrJ that would make http client (4.3.2) incompatible with SolrJ?
Currently, my code looks like this:
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
socketConfig.setSoTimeout(1);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(300);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
LBHttpSolrServer lb = new LBHttpSolrServer(httpClient, parser)
CloudSolrServer server = new CloudSolrServer(zkConnect, lb);
I also faced the issue of stale connections (TIME_WAIT) when using 4.2.3
but not with 4.3.2 (may be because I enabled socket reuse). So far, I
haven't seen any exceptions with 4.3.x when using CSS but I will keep a
note on them so that I can switch to 4.2.3 if 4.3.2 proves
