Re: How to know which port is newly created for the connection ?
On Fri, 2014-02-14 at 21:15 +, Kiran Chitturi wrote:
Hi,
I am using http client 4.3.2 and I have a few questions.
1) I am making lot of concurrent requests to a server. When creating
requests, I want to debug and check which ports are newly created by the
client. Currently, I can see the ports in netstat as pasted below.
tcp4 0 0 172.16.11.138.7575 172.16.11.138.52014ESTABLISHED
I enabled the debug logging for MainClientExec class and it gives me the
below information. Here it does not exactly tell me what is the source port
that is being used for the connection. How can I use this ? Also, whenever
MainClientExec prints the below line, does it mean a new connection is
created with a new port or an existing port is being used ?
2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection
{}-http://172.16.11.138:8983
Kiran
Please raise a JIRA for this change request. I'll try to include it into
the upcoming 4.3.3 release.
2) My code looks like this for creating a http Client. I pass the client to
SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ which
uses this client to make requests. SolrJ makes requests to a Solr server.
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(100);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
final CloseableHttpClient httpClient = httpBuilder.build();
This httpClient created above also throws some exceptions. Can I assume retry
is successful and does this happen because I enabled socketReuseAddress ?
HttpClient should propagate I/O exceptions to the caller which it is
unable to re-try. If your code is not catching any exceptions it should
be safe to assume requests have been successfully re-tried.
Oleg
2014-02-14 12:53:21,415 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:53:21,443 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@106] - Retrying request
2014-02-14 12:53:47,299 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:53:47,327 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@106] - Retrying request
2014-02-14 12:54:24,740 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:54:24,763 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@106] - Retrying request
2014-02-14 12:54:39,121 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:54:39,157 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@106] - Retrying request
Please let me know your suggestions.
Many Thanks for your help,
--
Kiran Chitturi,
Software Engineer,
LucidWorks.
Cell Phone: 540-577-4852
Office Phone: 650-249-4452
Email: kiran.chitt...@lucidworks.com
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How to know which port is newly created for the connection ?
On 2/17/14 1:12 AM, Oleg Kalnichevski ol...@apache.org wrote:
On Fri, 2014-02-14 at 21:15 +, Kiran Chitturi wrote:
Hi,
I am using http client 4.3.2 and I have a few questions.
1) I am making lot of concurrent requests to a server. When creating
requests, I want to debug and check which ports are newly created by the
client. Currently, I can see the ports in netstat as pasted below.
tcp4 0 0 172.16.11.138.7575 172.16.11.138.52014
ESTABLISHED
I enabled the debug logging for MainClientExec class and it gives me
the below information. Here it does not exactly tell me what is the
source port that is being used for the connection. How can I use this ?
Also, whenever MainClientExec prints the below line, does it mean a new
connection is created with a new port or an existing port is being used ?
2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection
{}-http://172.16.11.138:8983
Kiran
Please raise a JIRA for this change request. I'll try to include it into
the upcoming 4.3.3 release.
I opened a JIRA at HTTPClient-1462
(https://issues.apache.org/jira/browse/HTTPCLIENT-1462).
Thanks,
Kiran.
2) My code looks like this for creating a http Client. I pass the
client to
SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ
which uses this client to make requests. SolrJ makes requests to a Solr
server.
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(100);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
final CloseableHttpClient httpClient = httpBuilder.build();
This httpClient created above also throws some exceptions. Can I assume
retry is successful and does this happen because I enabled
socketReuseAddress ?
HttpClient should propagate I/O exceptions to the caller which it is
unable to re-try. If your code is not catching any exceptions it should
be safe to assume requests have been successfully re-tried.
Oleg
2014-02-14 12:53:21,415 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:53:21,443 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@106] - Retrying request
2014-02-14 12:53:47,299 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:53:47,327 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@106] - Retrying request
2014-02-14 12:54:24,740 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:54:24,763 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@106] - Retrying request
2014-02-14 12:54:39,121 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:54:39,157 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@106] - Retrying request
Please let me know your suggestions.
Many Thanks for your help,
--
Kiran Chitturi,
Software Engineer,
LucidWorks.
Cell Phone: 540-577-4852
Office Phone: 650-249-4452
Email: kiran.chitt...@lucidworks.com
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How to know which port is newly created for the connection ?
On 2/17/2014 12:34 AM, Kiran Chitturi wrote: Thank you so much for your reply. If I use the 'SystemDefaultHttpClient' for creating http client, there are many options for configuring the http client like socket reuse, disabling stale check, etc.. that are not possible to configure through SolrJ. May be after SOLR-5604, there will be more options through SolrJ for configuring or one could pass their own http client with the configured parameters. That's part of my idea with that issue. Get rid of all HttpClient deprecations in Solr code and make it possible to use an HttpClient built with HttpClientBuilder. A secondary but important goal would be to fix HttpClientUtil to use the builder object, and expand the list of options it can configure. If I use Http client from 4.3.x series, do you think there is code inside SolrJ that would make http client (4.3.2) incompatible with SolrJ? After I upgraded to 4.3.1 in my SolrJ application, I saw all the deprecated HttpClient stuff. I tried switching it to HttpClientBuilder and using the client with HttpSolrServer, but my application no longer worked, and threw some ugly exceptions. I am a little bit surprised that you can get it to work, but I haven't tried CloudSolrServer. I also haven't tried again now that I'm on 4.3.2, which might be why it's working for you. More boring details, many of which have already been discussed: I switched to SolrJ's HttpClientUtil so I no longer had deprecations in my own code, and forgot about it for a few weeks. Then someone opened SOLR-5590. I tested HttpClient 4.3.1 (latest at the time) in the Lucene ivy config. The tests still passed, so I committed the change. Now Solr and one small part of Lucene are using deprecated classes and methods, so we have SOLR-5604. Every time I start trying to fix it, the errors balloon, so I think it's going to require a significant overhaul. With Solr 4.x, API compatibility with earlier 4.x releases is preferred, so it might only be possible to implement SOLR-5604 in trunk. Thanks, Shawn - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How to know which port is newly created for the connection ?
On 2/14/2014 2:15 PM, Kiran Chitturi wrote:
I am using http client 4.3.2 and I have a few questions.
1) I am making lot of concurrent requests to a server. When creating
requests, I want to debug and check which ports are newly created by the
client. Currently, I can see the ports in netstat as pasted below.
tcp4 0 0 172.16.11.138.7575 172.16.11.138.52014ESTABLISHED
I enabled the debug logging for MainClientExec class and it gives me the
below information. Here it does not exactly tell me what is the source port
that is being used for the connection. How can I use this ? Also, whenever
MainClientExec prints the below line, does it mean a new connection is
created with a new port or an existing port is being used ?
2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection
{}-http://172.16.11.138:8983
2) My code looks like this for creating a http Client. I pass the client to
SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ which
uses this client to make requests. SolrJ makes requests to a Solr server.
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(100);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
final CloseableHttpClient httpClient = httpBuilder.build();
This httpClient created above also throws some exceptions. Can I assume retry
is successful and does this happen because I enabled socketReuseAddress ?
2014-02-14 12:53:21,415 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:53:21,443 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@106] - Retrying request
When it comes to the HttpClient side of this, I don't have much idea
what's going on ... but I do have a little bit of experience with SolrJ.
I have run into a lot of problems when trying to fully migrate Solr to
HttpClient 4.3, most of which are likely a result of my own
inexperience. Most of SolrJ's interaction with HttpClient is now
deprecated.
https://issues.apache.org/jira/browse/SOLR-5604
For my own SolrJ code, I couldn't get it to work when I used
HttpClientBuilder to make the HttpClient object for SolrJ. I don't
remember what the exceptions looked like. What I ended up doing to
avoid deprecations in my own code was using the utilities built into
SolrJ. I basically kicked the problem upstream. Here's how I build an
HttpClient in my own SolrJ code now, one that is shared between all my
HttpSolrServer instances:
ModifiableSolrParams params = new ModifiableSolrParams();
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 300);
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, 5000);
httpClient = HttpClientUtil.createClient(params);
https://lucene.apache.org/solr/4_6_0/solr-solrj/org/apache/solr/client/solrj/impl/HttpClientUtil.html
Most of the options you have above in your code are available in the
Solr utility class. Note that the object built under all these layers
is currently the deprecated SystemDefaultHttpClient class, which is
modifiable after it is created. The new objects in HttpClient 4.3 are
NOT modifiable after creation.
There's still a lot of work left to do for SOLR-5604. Until that work
is done, I don't think you'll have much luck using the new classes
introduced by HttpClient 4.3.x.
Thanks,
Shawn
-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: How to know which port is newly created for the connection ?
On 2/16/14 8:23 PM, Shawn Heisey s...@elyograg.org wrote:
On 2/14/2014 2:15 PM, Kiran Chitturi wrote:
I am using http client 4.3.2 and I have a few questions.
1) I am making lot of concurrent requests to a server. When creating
requests, I want to debug and check which ports are newly created by the
client. Currently, I can see the ports in netstat as pasted below.
tcp4 0 0 172.16.11.138.7575 172.16.11.138.52014
ESTABLISHED
I enabled the debug logging for MainClientExec class and it gives me
the below information. Here it does not exactly tell me what is the
source port that is being used for the connection. How can I use this ?
Also, whenever MainClientExec prints the below line, does it mean a new
connection is created with a new port or an existing port is being used ?
2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection
{}-http://172.16.11.138:8983
2) My code looks like this for creating a http Client. I pass the
client to
SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ
which uses this client to make requests. SolrJ makes requests to a Solr
server.
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(100);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
final CloseableHttpClient httpClient = httpBuilder.build();
This httpClient created above also throws some exceptions. Can I assume
retry is successful and does this happen because I enabled
socketReuseAddress ?
2014-02-14 12:53:21,415 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address
already in use
2014-02-14 12:53:21,443 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@106] - Retrying request
When it comes to the HttpClient side of this, I don't have much idea
what's going on ... but I do have a little bit of experience with SolrJ.
I have run into a lot of problems when trying to fully migrate Solr to
HttpClient 4.3, most of which are likely a result of my own
inexperience. Most of SolrJ's interaction with HttpClient is now
deprecated.
https://issues.apache.org/jira/browse/SOLR-5604
For my own SolrJ code, I couldn't get it to work when I used
HttpClientBuilder to make the HttpClient object for SolrJ. I don't
remember what the exceptions looked like. What I ended up doing to
avoid deprecations in my own code was using the utilities built into
SolrJ. I basically kicked the problem upstream. Here's how I build an
HttpClient in my own SolrJ code now, one that is shared between all my
HttpSolrServer instances:
ModifiableSolrParams params = new ModifiableSolrParams();
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 300);
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, 5000);
httpClient = HttpClientUtil.createClient(params);
https://lucene.apache.org/solr/4_6_0/solr-solrj/org/apache/solr/client/sol
rj/impl/HttpClientUtil.html
Most of the options you have above in your code are available in the
Solr utility class. Note that the object built under all these layers
is currently the deprecated SystemDefaultHttpClient class, which is
modifiable after it is created. The new objects in HttpClient 4.3 are
NOT modifiable after creation.
Shawn,
Thank you so much for your reply. If I use the 'SystemDefaultHttpClient'
for creating http client, there are many options for configuring the http
client like socket reuse, disabling stale check, etc.. that are not
possible to configure through SolrJ. May be after SOLR-5604, there will be
more options through SolrJ for configuring or one could pass their own
http client with the configured parameters.
If I use Http client from 4.3.x series, do you think there is code inside
SolrJ that would make http client (4.3.2) incompatible with SolrJ?
Currently, my code looks like this:
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
socketConfig.setSoTimeout(1);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(300);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
LBHttpSolrServer lb = new LBHttpSolrServer(httpClient, parser)
CloudSolrServer server = new CloudSolrServer(zkConnect, lb);
I also faced the issue of stale connections (TIME_WAIT) when using 4.2.3
but not with 4.3.2 (may be because I enabled socket reuse). So far, I
haven't seen any exceptions with 4.3.x when using CSS but I will keep a
note on them so that I can switch to 4.2.3 if 4.3.2 proves
How to know which port is newly created for the connection ?
Hi,
I am using http client 4.3.2 and I have a few questions.
1) I am making lot of concurrent requests to a server. When creating requests,
I want to debug and check which ports are newly created by the client.
Currently, I can see the ports in netstat as pasted below.
tcp4 0 0 172.16.11.138.7575 172.16.11.138.52014ESTABLISHED
I enabled the debug logging for MainClientExec class and it gives me the below
information. Here it does not exactly tell me what is the source port that is
being used for the connection. How can I use this ? Also, whenever
MainClientExec prints the below line, does it mean a new connection is created
with a new port or an existing port is being used ?
2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection
{}-http://172.16.11.138:8983
2) My code looks like this for creating a http Client. I pass the client to
SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ which uses
this client to make requests. SolrJ makes requests to a Solr server.
HttpClientBuilder httpBuilder = HttpClientBuilder.create();
Builder socketConfig = SocketConfig.custom();
socketConfig.setSoReuseAddress(true);
httpBuilder.setDefaultSocketConfig(socketConfig.build());
httpBuilder.setMaxConnTotal(100);
httpBuilder.setMaxConnPerRoute(100);
httpBuilder.disableRedirectHandling();
httpBuilder.useSystemProperties();
final CloseableHttpClient httpClient = httpBuilder.build();
This httpClient created above also throws some exceptions. Can I assume retry
is successful and does this happen because I enabled socketReuseAddress ?
2014-02-14 12:53:21,415 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@93] - I/O exception (java.net.SocketException)
caught when processing request: Address already in use
2014-02-14 12:53:21,443 - INFO [CloudSolrServer
ThreadPool-1-thread-59:RetryExec@106] - Retrying request
2014-02-14 12:53:47,299 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:53:47,327 - INFO [CloudSolrServer
ThreadPool-1-thread-203:RetryExec@106] - Retrying request
2014-02-14 12:54:24,740 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:54:24,763 - INFO [CloudSolrServer
ThreadPool-1-thread-263:RetryExec@106] - Retrying request
2014-02-14 12:54:39,121 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@93] - I/O exception
(java.net.SocketException) caught when processing request: Address already in
use
2014-02-14 12:54:39,157 - INFO [CloudSolrServer
ThreadPool-1-thread-142:RetryExec@106] - Retrying request
Please let me know your suggestions.
Many Thanks for your help,
--
Kiran Chitturi,
Software Engineer,
LucidWorks.
Cell Phone: 540-577-4852
Office Phone: 650-249-4452
Email: kiran.chitt...@lucidworks.com
