Re: How Do I Set SNI(Server Name Indentification)

2023-08-11 Thread Shawn Heisey

On 8/10/23 14:03, Petar Tahchiev wrote:

Hi Jochen,
I don't have 2 different SSL certificates.
I have no idea what  SNI is but that seems to be the only difference in the
log from curl and httpclient5.


https://en.wikipedia.org/wiki/Server_Name_Indication

Basically it's a feature of TLS that allows a client to send a hint to a 
server so it can decide which certificate to send.  With HTTPS, the SNI 
value is typically the same as the Host header value that is later sent 
over the encrypted channel.  With httpclient implementations, the SNI 
value is usually extracted from the URL that has been requested.  So a 
request for "https://www.example.com/some/path; would set the SNI and 
Host header to www.example.com.


This issue seems to be a case where the SNI value is missing, or maybe 
sent or interpreted as the literal string "null".


It seems odd that SNI could affect a server that doesn't have more than 
one certificate.  Unless the server is deciding to not proceed with the 
connection at all because it doesn't have a certificate that matches the 
missing or incorrect SNI value.


I have seen that things can often get fuzzy with Java software and TLS, 
because Sun wrote their own implementation of TLS for Java, and it 
sometimes does not behave exactly the same as other implementations. 
I'm not trying to say that their implementation is wrong, but it does 
behave differently than another implementation like openssl.


I hope you can get the info you need to work around the difficulty.

Thanks,
Shawn

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Sockets not closing

2022-07-07 Thread Shawn Heisey

On 7/7/22 11:58, Gordon Ross wrote:

I’ve tried doing the bare minimum of:

InputStream is = response.getEntity().getContent();
is.close();
response.close();


I remember having some problems with connections staying open, leading 
the client to run out.  I was advised to make sure the entity is fully 
consumed.  Try this code instead of those three lines:


HttpEntity entity = response.getEntity();
InputStream is = entity.getContent();
is.close();
EntityUtils.consumeQuietly(entity);
response.close();

I am not in any way an expert or even very knowledgeable about 
httpclient, but this looks like what I ended up doing on advice from 
this list, which did fix my problem.  I can't find the code where I did 
this, so I can't verify.


https://hc.apache.org/httpcomponents-core-5.1.x/current/httpcore5/apidocs/org/apache/hc/core5/http/io/entity/EntityUtils.html#consumeQuietly(org.apache.hc.core5.http.HttpEntity)

Based on the javadoc, that might close the InputStream too, but someone 
who knows more than I do will have to confirm.


Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: OS

2019-04-18 Thread Shawn Heisey

On 4/18/2019 2:03 AM, liname...@outlook.com wrote:

Hello, I am doing an investigation.
Does Windows Server 2019 support the following products:

Apache HttpComponents  3.1
Apache HttpComponents  4.0.1
Apache HttpComponents  4.3.5


I have seen this question from you on several Apache mailing lists for 
different software components.


Just like all the other projects where I have seen your question, the 
HttpComponents projects are Java-based software.  The only requirement 
is a JVM.  If you can obtain Java for Windows Server 2019, which I think 
is likely because I know it's available for 64-bit Windows, you can use 
any java library that is compatible with the JVM version you have installed.


Also be sure to carefully read the license for the JVM you are installing.

Thanks,
Shawn

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: httpCore5 - ClassicGetExecutionExample

2018-08-09 Thread Shawn Heisey

On 8/8/2018 2:36 PM, Tommy Pham wrote:

Ah.  I forgot about Apache migrating to Git a while back.  I wasn't sure if
all the projects are migrated.  Thank you, again.


As far as I know, there is no mandate at Apache to switch everything to 
git.  It's up to each individual project to decide what source control 
software they want to use.  Many projects are still using subversion for 
their primary source repositories.


There is a general trend among projects (both open source and other) 
towards switching to git.  It's the shiny new toy that everyone wants.  
Many Apache projects have switched to git, and are happy with that 
choice.  In another ten years, there might be something new that 
attracts the world's attention.


It would be amusing to me if the subversion project changed to something 
else to house their source code.  That's probably never going to happen, 
but if it did, I'd have to smile about it.


Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Fwd: HttpClient5 : simple example on how to configure timeout after build()

2018-05-17 Thread Shawn Heisey

On 5/17/2018 3:51 AM, Oleg Kalnichevski wrote:

HttpClientBuilder in HC4 got overloaded with so many connection
management parameters which could easily get rendered ineffective by
explicitly passing an instance of HttpClientConnectionManager to the
builder.

The same could with HC5 would look like that.


Thanks for the information on how to set the parameters I was looking 
for.  I've upgraded to HC5 in one app, and all the tests are still 
passing.  It also seems to work with real traffic.


I'm explicitly creating the inner objects rather than building them on 
the fly like your code example does.  Probably no real difference in 
what actually happens on execution, I just find this code style easier 
to read.


        final SocketConfig sc = 
SocketConfig.custom().setSoTimeout(Timeout.ofSeconds(10)).build();
        final HttpClientConnectionManager cm = 
PoolingHttpClientConnectionManagerBuilder.create()

.setMaxConnTotal(1000).setMaxConnPerRoute(200).setDefaultSocketConfig(sc).build();
        final RequestConfig rc = 
RequestConfig.custom().setConnectionTimeout(Timeout.ofSeconds(5))

                .build();
        final CloseableHttpClient client = 
HttpClientBuilder.create().setConnectionManager(cm)

                .setDefaultRequestConfig(rc).build();

Are there any plans to make the creation of builder objects more 
uniform?  Sometimes it's custom(), sometimes it's create().It would be 
nice if there was consistency.


Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Fwd: HttpClient5 : simple example on how to configure timeout after build()

2018-05-16 Thread Shawn Heisey
On 5/16/2018 8:42 AM, Shawn Heisey wrote:
>   RequestConfig rc = RequestConfig.custom().setConnectTimeout(15000)
>           .setSocketTimeout(12).build();
>   httpClient = HttpClients.custom().setDefaultRequestConfig(rc)
> .setMaxConnPerRoute(300).setMaxConnTotal(5000).disableAutomaticRetries()
>           .build();

I have attempted to upgrade httpclient in one of my projects from 4.5.x
to the latest 5.0 beta release, and this code no longer compiles.  On
RequestConfig.Builder, the setSocketTimeout method no longer exists.  On
HttpClientBuilder, the setMaxConnPerRoute and setMaxConnTotal methods no
longer exist.  These methods were not deprecated in 4.5.x.

Is it still possible to set a socket timeout default?  Is it still
possible to increase the maximum number of connections that a client can
make?  If so, how is it done?

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Fwd: HttpClient5 : simple example on how to configure timeout after build()

2018-05-16 Thread Shawn Heisey

On 5/16/2018 8:09 AM, / wrote:
I am looking for an example on how to configure HttpClient5 after it 
has been built and how to extract/print some of its configuration.


Once I have an HttpClient object, how do I go about and change some of 
its settings, for example connection timeout or user-agent-string or 
even cookie jar?


I am looking for the most straight-forward and efficient way to do 
this. I don't care about "fluent" APIs neither about streams etc.


something like:

myclient.setParameter(connection_timeout, 1000); 


For the most part, you can't change settings on an existing HttpClient 
object.  Since about 4.3, the objects and methods that allow clients to 
be changed after creation are all deprecated. That capability is 
completely gone in 5.x.  Default settings are managed with builder 
objects using fluent methods, then you create the client object with the 
indicated settings.  Here's how I create a client object with explicit 
defaults using non-deprecated code in the 4.5 version:


  RequestConfig rc = RequestConfig.custom().setConnectTimeout(15000)
          .setSocketTimeout(12).build();
  httpClient = HttpClients.custom().setDefaultRequestConfig(rc)
.setMaxConnPerRoute(300).setMaxConnTotal(5000).disableAutomaticRetries()
          .build();

The httpClient field is an instance of HttpClient.  I do not know what 
kind of adjustments might need to be made for 5.x, but that should give 
you an idea about how things are done since the way you're trying to do 
it is no longer available.


Many of the settings you might be interested in can also be changed at 
the request level.  I do not know HOW to do this, only that it CAN be 
done.  I think this is what Oleg was referring to in his reply.


Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Are there any "Upgrading from 4.x" notes for the alpha/beta 5.0 releases? (trying to update Solr)

2017-12-13 Thread Shawn Heisey
I'm trying to prepare Solr for the release of the 5.0 versions of
httpcomponents.  When I update the ivy config to pull down the
alpha/beta 5.0 releases, there are naturally a lot of errors in the code.

The "Organize Imports" function in eclipse can find many of the classes
whose packages have changed, but I am also finding it difficult to
locate replacement classes for things that seem to have simply
disappeared.  One example is org.apache.http.message.AbstractHttpMessage.

Is there anyplace I can go to find documentation that can help me figure
out how to alter an existing project for the new version?

Side note:  It is likely that Solr is using paradigms and methods that
have been discouraged by this project for a long time.  Is there an IRC
channel for this project where I can have a more realtime discussion
with an expert?  I can open an issue in Jira on the SOLR project, but
I'm hoping for something a little less formal for the bulk of the
discussion.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



maxPerRoute -- what *exactly* defines a route?

2017-11-01 Thread Shawn Heisey
One of the settings that you can set when creating HttpClient objects is 
the maximum connections per route on the connection manager.  Increasing 
this value is part of making HttpClient capable of handling many 
threads/connections at once.


The exact definition of a route is not stated in the javadoc for the 
builder class, and I haven't been able to find it anywhere else either.


I did find this SO post, but I don't see an answer there that's clear. 
The HttpRoute javadoc may contain the answer, but if it does, it's not 
straightforward.


https://stackoverflow.com/questions/12028635/what-is-the-meaning-of-per-route-basis-in-poolingclientconnectionmanager

A section of documentation from an entirely different project (nodejs) 
was given.  A different project may have a very different definition of 
a route than THIS project.


So I'm looking for what the definition of a route is for httpcomponents, 
and would like to see that information added to the javadoc on the 
Builder class(es) and the connection manager class(es).


If I were to open an issue for this, would it go on HTTPCLIENT or 
HTTPCORE?  Should I open an issue?


I suspect that the definition of a route is probably the 
"protocol://host:port" combination, and based on the info about nodejs 
that was given, may also include the type of request 
(GET/POST/HEAD/OPTIONS/etc).  It might only take the host into account 
and ignore the protocol and port.  It might have more information, such 
as a defined proxy server, authentication parameters, etc.  Can anyone 
confirm or deny?  If the host is involved, I would also need to know 
whether it is compared as given, or if any conversion (lowercasing, IP 
address resolution, finding FQDN, etc.) happens before comparison.


If a detailed answer to this question is in a FAQ somewhere, perhaps 
that should be referenced by javadoc and in other places.


Thanks,
Shawn

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Can the connection pool be shut down in any way other than explicitly closing?

2017-07-18 Thread Shawn Heisey
I have a user on the solr-user mailing list who is running into an
exception from HttpClient:

Caused by: java.lang.IllegalStateException: Connection pool shut down at
org.apache.http.util.Asserts.check(Asserts.java:34) at
org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:184)
at
org.apache.http.pool.AbstractConnPool.lease(AbstractConnPool.java:217)
at
org.apache.http.impl.conn.PoolingClientConnectionManager.requestConnection(PoolingClientConnectionManager.java:184)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:415)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:447)
... 24 more

They are using a SolrJ object called CloudSolrClient.  This in turn uses
LBHttpSolrClient, which uses HttpSolrClient, and inside THAT class,
HttpClient is used.

What I'd like to know is whether the connection pool can ever be shut
down *without* explicitly calling close/shutdown ... by some kind of
error, perhaps.  I suspect that they are calling an explicit action to
shut down either the solr client or the HttpClient, but I don't want to
say that before I ask whether it can happen any other way.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Support HTTP/2 protocol

2017-01-30 Thread Shawn Heisey
On 1/26/2017 8:15 AM, Oleg Kalnichevski wrote:
> ALPN will be supported as soon as it is supported by the Java platform
> (which is not going to happen until Java 9). 

I see evidence that the other Java http implementations have ALPN
support already ... but those systems implement both server and client. 
Could it be that those ALPN implementations are server-side only?  I
can't seem to easily locate anything saying for sure.

> ALPN can be used to advertise server protocol capabilities at the time
> of SSL handshake and allow clients to pick the desired protocol from
> the list of supported protocols. If one knows supported protocols
> beforehand ALPN is completely useless. Clients can go straight to using
> HTTP/2 if the server is known to support it.
>
> In the next release of HttpCore I would like to add protocol detection
> logic to enable endpoints to detect HTTP protocol version by examining
> the first packet received from the opposite endpoint. This in my
> opinion would be a much more practical feature. ALPN presently is very
> low on my priority list.
>
> Having said that ALPN support contribution would be welcome if someone
> is willing to develop it.

Knowing in advance that HTTP/2 support is available may be problematic. 
I can imagine a situation where servers are upgraded gradually, and the
client may not know whether the one it's connecting to can support the
new protocol.  Can HTTP/2 detection be reliable without ALPN, even in
situations where connecting to the same host/port may support HTTP/2 on
one connection, but not the next?  TCP load balancing is relatively
common with SSL.  If such detection can be reliable, then there won't be
anything to worry about.

For my webserver installations, I am hoping to get HTTP/2 support
enabled in the load balancer and worry about support on the back end
later.  I'm expecting the back-end LAN to be fast enough that multiple
connections can easily be established while the Internet-facing side
works through the inherent packet latency.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: HttpClient behavior on close with active connections

2016-11-10 Thread Shawn Heisey
On 11/10/2016 3:21 AM, Oleg Kalnichevski wrote:
> On Wed, 2016-11-09 at 13:43 -0700, Shawn Heisey wrote:
>> What happens to a long-lived HTTP connection if another thread calls
>> close() on the HttpClient? Does the connection immediately die and
>> throw an exception, or would the connection finish as expected in its
>> own time? 
> Yes, it does. CloseableHttpClient#close immediately purges the
> internal pool of all connections both active (leased) and inactive
> (kept alive). HC 5.0 will support graceful shutdown option whereby one
> can give connections some grace period to complete ongoing exchanges
> prior to shutting down. 

Thank you!  I prefer to have everything shut down immediately, so this
is good news.  If the exception that happens in the background thread is
different enough from exceptions indicating "real" problems, I will be
able to detect that situation and log a short warning, instead of a full
stacktrace.  I will be on the lookout for this when I've written enough
of the code to test it.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



HttpClient behavior on close with active connections

2016-11-09 Thread Shawn Heisey
I'm using SolrJ 6.2.1 in the program I'm writing, which pulls in
httpclient/httpmime 4.5.2 and httpcore 4.4.5 as dependencies.

One of the things that my SolrJ code does takes over an hour to
complete.  The HTTP connection is kept open for all that time.  I'd like
to find a way for the Solr server to respond immediately and continue
the operation in the background, but currently there doesn't seem to be
a way to do that.

Now I am attempting to make it possible to gracefully shut down
everything related to what my code does, turn everything into garbage,
reload the configuration, and build it all back up.  Therefore I need to
know how certain things behave at close/shutdown, assuming everything is
not idle.

What happens to a long-lived HTTP connection if another thread calls
close() on the HttpClient?  Does the connection immediately die and
throw an exception, or would the connection finish as expected in its
own time?  If the connection remains open until completion, does the
close() call return immediately, or wait until that connection ends,
whether successful or not?  Are all object references (threads in
particular) properly released and turned into garbage?  I tried to
follow the code, but got lost quickly.  If it's documented somewhere,
please point me at that documentation.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Controlling releaseConnection

2016-10-17 Thread Shawn Heisey
On 10/17/2016 3:22 PM, Pellerin, Clement wrote:
> Our customer needs to delay the release of the connection until the response 
> is fully processed.
> They want to turn off the early automatic release of the connection and do it 
> manually later.
>
> This is the problematic code in MainClientExec
> // check for entity, release connection if possible
> final HttpEntity entity = response.getEntity();
> if (entity == null || !entity.isStreaming()) {
> // connection not needed and (assumed to be) in re-usable 
> state
> connHolder.releaseConnection();
> return new HttpResponseProxy(response, null);
> } else {
> return new HttpResponseProxy(response, connHolder);
> }

Mostly an end-user here, with no status to speak of in this project.  I
do have status on another Apache project that utilizes HttpClient, but I
don't know much about that part of the code.  I have written some
HttpClient code for a completely unrelated project of my own, but that
code is VERY simple.

When I read the code above, what I see is this: It only releases the
connection if the entity is nonexistent (null) or the entity is NOT a
type that uses streaming.

I will fully admit that my experience with HttpClient is limited, but I
think the chance is very small that the HttpComponents committers have
made a mistake here.  I think this particular code has probably been
discussed and examined, then ultimately validated as correct.  Here's
why I think they didn't make a mistake:

If the entity object is null, then the response probably doesn't HAVE an
entity (response body), so it will be entirely self-contained,
consisting of headers only, and the connection doesn't have anything
further to send.  If the entity exists but doesn't utilize streaming,
then I think it's likely that the entity was received in its entirety
and has been incorporated into the response object already, and once
again, the connection isn't needed.  If my limited understanding of
non-streaming entities is correct, they have the potential to be very
dangerous from a memory consumption perspective, and my own usage of
HttpClient (where I did not set anything related to the entity type)
suggests that streaming entities are used by default.

Restating in another way:  In the first situation that results in a
released connection, there's nothing to consume, you just need the
response object that you already have.  In the second situation, the
entity you will consume is probably already available within the
response object and doesn't need the connection.  The comment on the
release call in the code quoted above implies that this is how things work.

In these situations, why do you need the connection to stick around?  I
think it can't do anything else that's useful for that request.  I would
imagine that if the connection utilizes keepalive/pipelining, that it
will typically remain open after release and can be utilized again for a
different request.

Someone with more direct knowledge of HttpClient's internal
implementation will need to confirm whether or not I'm correct in what
I've written.  My understanding could be wrong.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: HttpClient is not proving to be thread-safe and reusable -- probably user error

2016-06-21 Thread Shawn Heisey
On 6/21/2016 8:56 AM, Pete Keyes wrote:
> You failed to consume the response. Apache-HC will never return the
> connection to the pool. Once you've hit the 500th thread all
> connections are leased. In the run() method simply add a finally to
> your try/catch and move the response object into scope.

Thank you.  Added that, and now it all works.  I haven't needed to
actually obtain the data in the response, so I never looked that part up
in the documentation.  In hindsight, it makes sense that the response
needs handling.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



HttpClient is not proving to be thread-safe and reusable -- probably user error

2016-06-21 Thread Shawn Heisey
At the paste URL below is the code I'm using in a test.  The test is
checking for race conditions in some server code I've written:

http://apaste.info/Vs6

This code will stop working correctly during the second loop.  On the
first loop, it creates 400 threads and requests the URL once in each
thread.  On the second loop, it tries to do it again, but hangs after
the 100th thread is created.  When I create the client, i set
maxConnPerRoute to 500, so it is related to that.  It seems that each
connection isn't being removed from the internal tracking after it
completes.

If I move the httpclient creation inside the for loop and uncomment the
"client.close();" at the end of the loop, then everything works ... but
it seems like Ishould be able to use one client for this entire test.

What have I done wrong in my code?

threads is a Set.
firstFailedLoop and loopNum are AtomicInteger.
numThreadsPerRun is set to 400.

If anything in the code is unclear, please let me know what needs
clarification.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Provide auth username/password with system properties or environment?

2015-11-19 Thread Shawn Heisey
On 11/19/2015 3:03 AM, Oleg Kalnichevski wrote:
> On Wed, 2015-11-18 at 13:17 -0700, Shawn Heisey wrote:
>> In the meantime, I am hoping that there is a way that Solr users can
>> provide credentials to HttpClient via another means, like system
>> properties or environment variables.  Is there anything like that?
> Shawn
>
> One can always use a custom credentials provider instead of the default
> ones.
>
> http://hc.apache.org/httpcomponents-client-4.5.x/httpclient/xref/org/apache/http/impl/client/BasicCredentialsProvider.html
> http://hc.apache.org/httpcomponents-client-4.5.x/httpclient/xref/org/apache/http/impl/client/SystemDefaultCredentialsProvider.html

I'm hoping to give users of the existing code a way to provide
authentication details to HttpClient.  The users aren't creating Java
code, they are using bits that we provide -- a shell script or Windows
command script that executes a binary SolrCLI class.  That class
currently doesn't do anything with authentication.

>From what I can tell, there isn't a way, and users will need to wait for
us to extend our script and SolrCLI class to handle credentials.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Provide auth username/password with system properties or environment?

2015-11-18 Thread Shawn Heisey
Recent versions of Solr have added basic authentication.  The Solr
client functionality uses HttpClient.

The start script included with Solr has some additional functionality
for manipulating the running server, and the class that this script
calls uses HttpClient to talk to Solr.  We have an issue to add the
ability to provide user/password to this script:

https://issues.apache.org/jira/browse/SOLR-8048

In the meantime, I am hoping that there is a way that Solr users can
provide credentials to HttpClient via another means, like system
properties or environment variables.  Is there anything like that?

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Getting SocketTimeoutException after a very short amount of time

2015-10-13 Thread Shawn Heisey
On 9/4/2015 3:18 PM, Shawn Heisey wrote:
> On 9/4/2015 12:36 PM, Oleg Kalnichevski wrote:
>> There is nothing wrong with your code. Feel free to run it outside
>> Solr without any extra dependencies to make sure it works as intended.
>> I _strongly_ suspect there is an older version of HttpClient /
>> HttpCore on your classpath.
> 
> I'm willing to believe almost anything the evidence supports at this
> point, but I haven't seen anything to support that theory.

Closing out this discussion:

The problem turned out, unsurprisingly, to be a bug in my own code.  As
much as I hate to learn that I've done something wrong, I'm glad to see
that the problem is not in HttpClient, and therefore cannot affect other
people.

Part of my code creates a few threads to do indexing on multiple Solr
shards in parallel, then uses join() to wait for those threads.  The
code that uses join() was incorrect, and not waiting long enough.  The
main loop of my program continued on its merry way, unaware of the
problem, while one of those threads was running in the background,
eventually reaching socket timeout and sending an alarm email.

Now that I know where the problem is, I can come up with a solution.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Code with 4.x deprecations already removed?

2015-09-14 Thread Shawn Heisey
On 9/14/2015 1:45 AM, Oleg Kalnichevski wrote:
> You should not be using 5.0 unless you intent to participate in
> development of HttpClient. The trunk is currently unstable and I have
> no idea how long it may take for it to get stable enough for a GA
> release. Please use 4.5.x branch for now. It will still have all
> deprecated functionality intact.

I would only use 5.0 for finding and removing HC deprecations from Solr,
I would not commit an update to an unreleased version of any dependency,
even if such a change would compile, and in this case it definitely will
NOT work.

My experimentation with simply marking all deprecated code use as an
error revealed over 3000 uses of deprecated code.  As for HC, we use it
in such a way that deprecation removal will be very much a nontrivial
effort.  I'm not sure I actually have the time or understanding to
complete the work.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Getting SocketTimeoutException after a very short amount of time

2015-09-04 Thread Shawn Heisey
On 9/4/2015 12:36 PM, Oleg Kalnichevski wrote:
> There is nothing wrong with your code. Feel free to run it outside
> Solr without any extra dependencies to make sure it works as intended.
> I _strongly_ suspect there is an older version of HttpClient /
> HttpCore on your classpath.

I'm willing to believe almost anything the evidence supports at this
point, but I haven't seen anything to support that theory.

I will try to come up with a test program using HttpClient only that
calls one of our optimize URLs to see whether I get the same timeout. 
If it works as expected, I will take this problem to the Solr list.

The classpath calculation is simple.  Here's the "env.sh" that my start
script sources to set the classpath:

root@bigindy5:/opt/sparkbuild# cat env.sh
#!/bin/sh


for i in \
 -Xms4M \
 -Xmx1024M \
 -XX:+UseG1GC \
 -XX:+ParallelRefProcEnabled \
 -XX:G1HeapRegionSize=8m \
 -XX:MaxGCPauseMillis=200 \
 -XX:+UseLargePages \
 -XX:+AggressiveOpts \
# -Dcom.sun.management.jmxremote \
# -Dcom.sun.management.jmxremote.port=3001 \
# -Dcom.sun.management.jmxremote.ssl=false \
# -Dcom.sun.management.jmxremote.authenticate=false \

do
  OPTS="${OPTS} $i"
done
export OPTS

CLASSPATH="bin:resources"
for i in lib/*.jar
do
  CLASSPATH=${CLASSPATH}:${i}
done
export CLASSPATH

At the following URL is the entire "lsof" output for the running copy of
my program.  I only see one location for the HC jars.  They do show up
twice, but it's the same location each time.  The only change I made in
this output was to redact our domain name from the TCP entries.  Solr is
listening on 8982, which explains the connections to that port in
CLOSE_WAIT.

http://apaste.info/38Z

Separate from my apparent inability to get a socket timeout set, but
possibly related, I'm somewhat mystified by the fact that I get a socket
timeout exception after 30 seconds when typical Java and Linux defaults
are no socket timeout and the DEBUG entries from httpclient are saying
that the connection can be kept alive indefinitely.  I haven't changed
any timeout settings in the Ubuntu OS, so unless Ubuntu sets a default
socket timeout, I don't know where this is coming from.  The same thing
happens on the production servers running CentOS 6, and I have not set
any OS-level timeouts there either.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Getting SocketTimeoutException after a very short amount of time

2015-09-03 Thread Shawn Heisey
On 9/3/2015 10:57 AM, Shawn Heisey wrote:
> I don't see anything in the file my start script creates by
> redirecting stdout, though. If everything were working right,
> shouldn't I see output from setting the timeout in my own code?

I replaced the client construction using Solr's HttpClientUtil with the
following, which I came up with after reading the
ClientConfiguration.java example on the HC website:

  /* Trying out a new way of creating clients */
  RequestConfig requestConfig = RequestConfig.custom()
  .setSocketTimeout(90)
  .setConnectTimeout(15000)
  .build();
  httpClient = HttpClients.custom()
  .setDefaultRequestConfig(requestConfig)
  .setMaxConnPerRoute(300)
  .setMaxConnTotal(5000)
  .build();
  RequestConfig optimizeRequestConfig = RequestConfig.custom()
  .setSocketTimeout(720)
  .setConnectTimeout(15000)
  .build();
  optimizeHttpClient = HttpClients.custom()
  .setDefaultRequestConfig(optimizeRequestConfig)
  .build();

The httpClient and optimizeHttpClient instances are defined like this:

  /**
   * A static http client to use on Solr client objects.
   */
  private static HttpClient httpClient = null;

  /**
   * A static http client to use on Solr client objects. This one is for
doing
   * optimizes, will use a much longer socket timeout.
   */
  private static HttpClient optimizeHttpClient = null;

I am still seeing nothing in stdout, even though there's an added
System.out.println in BHttpResponseBase.java at the recommended location.

Am I doing something wrong in my attempts to debug this problem?

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Getting SocketTimeoutException after a very short amount of time

2015-08-31 Thread Shawn Heisey
On 8/31/2015 3:02 AM, Oleg Kalnichevski wrote:
> The socket timeout in blocking i/o is handled by the JRE. We have no
> control over it. 
> --- 
> Caused by: java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> ---
> There likely to be something that sets SO_TIMEOUT to a lower value. You
> should be able to trace it by placing a breakpoint or good ol'
> System.out.println at this line:
> http://hc.apache.org/httpcomponents-core-4.4.x/httpcore/xref/org/apache/http/impl/BHttpConnectionBase.html#277
> 
> Naturally it would make things easier if Solr used no deprecated
> methods.

I once tried to work out a patch for Solr to remove deprecated classes
implementing HttpClient.  I quickly got myself bogged down in issues I
didn't know how to fix.  I wasn't even attempting to fix the deeper-down
usage using deprecated methods, just the HttpClient objects themselves.

I'm not sure how to set up a debugging environment with breakpoints, so
I think println is going to be my friend.  Now I need to figure out how
to pull the project into eclipse for editing, and then how to build
httpcore.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Getting SocketTimeoutException after a very short amount of time

2015-08-30 Thread Shawn Heisey
I am seeing SocketTimeoutException in the log for my SolrJ program. 
SolrJ uses HttpClient.

I'm starting with the HC list for this problem because that seems like
the most likely place for a problem, but I know that it could be in
SolrJ or my own code.

The index cycle that timed out began at 06:13:30.003.  This starts out
talking to a MySQL database to work out any index changes that need to
be made, and if any are found, proceeds to make them.  The loop failed
with a socket timeout error at 2015-08-30 06:14:00.232, so it looks like
it only waited 30 seconds ... but the SO_TIMEOUT on the httpclient
should have been set to 15 minutes.

Here is the full exception, slightly redacted to obscure my company name:

ERROR - 2015-08-30 06:14:00.232;   977; 2; chain.b: error committing shard 2
com.REDACTED.idxbuild.util.BuildException: shard.b.2.spark2live commit
failed
at com.REDACTED.idxbuild.solr.Core.commit(Core.java:496)
at com.REDACTED.idxbuild.index.Shard.commit(Shard.java:454)
at com.REDACTED.idxbuild.index.Chain$2.run(Chain.java:969)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout
occured while waiting response from server at:
http://idxb3.REDACTED.com:8981/solr
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:570)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)
at
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:135)
at
org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:523)
at com.REDACTED.idxbuild.solr.Core.commit(Core.java:484)
... 2 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:466)
... 7 more

Here's my code that creates the two HttpClient objects which are shared
among multiple HttpSolrClient instances.  I create a client with a 15
minute socket timeout for general use, and a client with a two hour
timeout for doing index optimizes:

  synchronized (firstInstance)
  {
  if (firstInstance)
  {
  ModifiableSolrParams params = new ModifiableSolrParams();
  params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 300);
  params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, 5000);
  params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, 15000);
  params.add(HttpClientUtil.PROP_SO_TIMEOUT, 90);
  httpClient = HttpClientUtil.createClient(params);
  params.clear();
  params.add(HttpClientUtil.PROP_CONNECTION_TIMEOUT, 15000);
  params.add(HttpClientUtil.PROP_SO_TIMEOUT, 720);
  optimizeHttpClient = HttpClientUtil.createClient(params);
  firstInstance = false;
  }
  }

httpClient and optimizeHttpClient are static instances of HttpClient.

The code for SolrJ's 

Re: [ANNOUNCEMENT] HttpComponents Client 4.5 GA Released

2015-06-05 Thread Shawn Heisey
On 6/5/2015 3:24 AM, Oleg Kalnichevski wrote:
 The Apache HttpComponents project is pleased to announce 4.5 GA release
 of HttpComponents HttpClient.

 HttpClient 4.5 (GA) is a minor feature release that includes several
 incremental enhancements to the existing functionality such as support
 for private domains in the Mozilla Public Suffix List.

 Users of HttpClient 4.x are advised to upgrade.

Will there be any other components with a 4.5 release, or just the
httpclient?  I'm specifically interested in httpcore and httpmime.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: [ANNOUNCEMENT] HttpComponents Core 4.4.1 GA released

2015-03-20 Thread Shawn Heisey
On 3/20/2015 11:24 AM, Oleg Kalnichevski wrote:
 The Apache HttpComponents project is pleased to announce 4.4.1 GA
 release of HttpComponents Core.

You probably don't hear this enough ... thank you to anyone who works on
this project.  You make a lot of other software possible.

For the sake of stability, I decided to wait for the 4.4.1 release
before upgrading the HttpClient dependency in the Solr project.  These
are our current versions sent to ivy:

/org.apache.httpcomponents/httpclient = 4.3.1
/org.apache.httpcomponents/httpcore = 4.3
/org.apache.httpcomponents/httpmime = 4.3.1

Will any of those components be remaining at the 4.4 release while the
others advance to 4.4.1?  I'd like to know when I should proceed with
the dependency upgrade.

https://issues.apache.org/jira/browse/SOLR-6865

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: HTTP 2 and Apache HTTP client

2015-01-09 Thread Shawn Heisey
On 1/9/2015 2:41 AM, Oleg Kalnichevski wrote:
 Shawn,

 I can work on HC in my spare time only. Even if I quit my day job and
 divorce my wife I am not sure HTTP/2.0 in HC by the end of February is
 realistic. The best case scenario would be having BETA quality support
 for HTTP/2.0 by the end of year.

 Oleg

I would never make any actual demands of your time, I'm just conveying
my opinion that I think it's really important.  Thank you for everything
you have done and anything you may do in the future.  If there is any
way my limited http skillset and nonexistent understanding of the
codebase can be helpful, I am willing.

Other people are likely to be more insistent and demanding once the spec
is published. :)

Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: HTTP 2 and Apache HTTP client

2015-01-08 Thread Shawn Heisey
On 1/8/2015 5:28 PM, Stefan Magnus Landrø wrote:
 Maybe consider jetty instead? 
 
 https://github.com/eclipse/jetty.project/blob/jetty-http2/jetty-http2/http2-client/src/test/java/org/eclipse/jetty/http2/client/Client.java

For an Apache project like Solr (which uses HttpClient and is my primary
reason for being here), I see two problems with that idea:

One problem is the eating your own dog food argument -- if software
from an Apache project fits all of the requirements and has no
significant disadvantages compared to a competitor, we should choose to
use the software from Apache.  Both projects are likely to benefit.

http://en.wikipedia.org/wiki/Eating_your_own_dog_food

The other problem is that switching to a completely different http
client would likely involve significant development time, both for the
switch itself and for fixing the inevitable bugs that are a direct result.

Seeing that competing projects are already focusing effort on HTTP/2 is
even more reason to accelerate HttpClient towards HTTP/2.

Even Oracle has fairly concrete plans for a new client in Java itself:

https://bugs.openjdk.java.net/browse/JDK-8042950

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: HTTP 2 and Apache HTTP client

2015-01-08 Thread Shawn Heisey
On 1/8/2015 7:20 AM, Oleg Kalnichevski wrote:
 There is no concrete roadmap for HTTP/2.0 yet. An immediate objective is
 full compliance with the latest HTTP/1.1 spec (RFC 7230 and related)

 https://wiki.apache.org/HttpComponents/HttpComponentsRoadmap

Recently I read through the draft RFC for HTTP/2.0.  Once everybody has
stable and optimized implementations, the average Internet experience
(and a lot of behind-the-scenes infrastructure that use HTTP) will
improve quite a lot.  It looks really awesome.

It will be a major development effort to reach a stable 2.0 implementation.

The current schedule says that the final RFC will be published in
February 2015, which is quite a lot sooner than I thought.

http://en.wikipedia.org/wiki/HTTP/2#Development_milestones

IMHO, work on the implementation should be underway already, so that
there is an initial implementation available by the time the RFC is
published.  That's easy for me to say, since I am not qualified for the
work.  If I had any idea how to write code that functions at such a low
level, I would have already sent some ideas to Oleg!

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: [ANNOUNCEMENT] HttpComponents Core 4.4 GA released

2014-12-19 Thread Shawn Heisey
On 12/19/2014 6:39 AM, Oleg Kalnichevski wrote:
 It is a GA release of core components HttpClient is based up. HttpClient
 4.4 GA is not ready yet.

Got it.  Missed that little detail.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: [ANNOUNCEMENT] HttpComponents Core 4.4 GA released

2014-12-18 Thread Shawn Heisey
On 12/18/2014 1:41 AM, Oleg Kalnichevski wrote:
 The Apache HttpComponents project is pleased to announce 4.4 GA release
 of HttpComponents Core. 

I'm trying to upgrade the Lucene-Solr codebase to use HC 4.4, but ivy
can't find it.  It's missing from the maven repo that ivy tried to use
for downloading:

http://uk.maven.org/maven2/org/apache/httpcomponents/httpclient/

When do you think the maven repo will have the new version, so that ivy
can find it?

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: HttpClient v4.x deprecations

2014-09-24 Thread Shawn Heisey
On 9/24/2014 3:45 PM, Todd W Lainhart wrote:
 Is there a policy regarding when deprecated APIs are finally removed from 
 subsequent distributions?

 E.G. If I find a field marked as @deprecated as of 4.2, when would I 
 expect it to get pulled, if at all?  Or is the policy not to expect any 
 further development (e.g. bug fixes) for deprecated API?  Perhaps a bit of 
 both?

I am curious about this as well, but I think I know at least part of the
answer.  It's probably the same as Lucene/Solr, which is a fellow Apache
project to HttpComponents.

On Lucene/Solr, deprecations on one major release get removed from the
next major release.  For Lucene/Solr, the first number in the version
refers to the major release.  In real terms, things that are currently
being deprecated in 4.x releases on that project will be completely
removed from the 5.0 release, when it happens.

If I'm right in my general assessment, this is an important question:
Does the HttpComponents project consider 4.4 to be the next major
release, or 5.0?

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Fwd: Connection timeout doesn't behave the same on windows and linux

2014-03-03 Thread Shawn Heisey

On 3/3/2014 9:00 AM, Bratislav Stojanovic wrote:
It seems that connection timeout parameter doesn't work on Linux Mint 
16 (which is

basically Ubuntu). *Code runs perfectly fine on Windows.*Here's my setup :


snip


*.setConnectTimeout(1000)*


snip

*The problem : when I run this code on linux, there's no 1 sec delay 
between consecutive attempts. It just

*
*goes over 10 attempts and finished within ~200ms. Why?

*
*Am I doing something wrong or this is a bug? Please help.
*
*
P.S. I'm using http client 4.3.2.*


I assume that the service it's trying to connect to is down or in some 
way unavailable.  What is the exact status?  If the server is reachable 
on that port but the http server is not running, a connection attempt 
will be instantly refused, there's no reason to wait for the timeout period.


In order for a full second to pass, the port must be completely 
unreachable, which means the server either needs to be down or it needs 
to be firewalled in a way that offers no response at all rather than a 
refusal.  Is this the case with the server that the Windows version is 
trying to contact?


Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: How to know which port is newly created for the connection ?

2014-02-17 Thread Shawn Heisey
On 2/17/2014 12:34 AM, Kiran Chitturi wrote:
 Thank you so much for your reply. If I use the 'SystemDefaultHttpClient'
 for creating http client, there are many options for configuring the http
 client like socket reuse, disabling stale check, etc.. that are not
 possible to configure through SolrJ. May be after SOLR-5604, there will be
 more options through SolrJ for configuring or one could pass their own
 http client with the configured parameters.

That's part of my idea with that issue.  Get rid of all HttpClient
deprecations in Solr code and make it possible to use an HttpClient
built with HttpClientBuilder.  A secondary but important goal would be
to fix HttpClientUtil to use the builder object, and expand the list of
options it can configure.

 If I use Http client from  4.3.x series, do you think there is code inside
 SolrJ that would make http client (4.3.2) incompatible with SolrJ?

After I upgraded to 4.3.1 in my SolrJ application, I saw all the
deprecated HttpClient stuff.  I tried switching it to HttpClientBuilder
and using the client with HttpSolrServer, but my application no longer
worked, and threw some ugly exceptions.  I am a little bit surprised
that you can get it to work, but I haven't tried CloudSolrServer.  I
also haven't tried again now that I'm on 4.3.2, which might be why it's
working for you.

More boring details, many of which have already been discussed:

I switched to SolrJ's HttpClientUtil so I no longer had deprecations in
my own code, and forgot about it for a few weeks.  Then someone opened
SOLR-5590.  I tested HttpClient 4.3.1 (latest at the time) in the Lucene
ivy config.  The tests still passed, so I committed the change.

Now Solr and one small part of Lucene are using deprecated classes and
methods, so we have SOLR-5604.  Every time I start trying to fix it, the
errors balloon, so I think it's going to require a significant overhaul.
 With Solr 4.x, API compatibility with earlier 4.x releases is
preferred, so it might only be possible to implement SOLR-5604 in trunk.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: How to know which port is newly created for the connection ?

2014-02-16 Thread Shawn Heisey
On 2/14/2014 2:15 PM, Kiran Chitturi wrote:
 I am using http client 4.3.2 and I have a few questions.
 
 1)  I am making lot of concurrent requests to a server. When creating 
 requests, I want to debug and check which ports are newly created by the 
 client. Currently, I can see the ports in netstat as pasted below.
 
 tcp4   0  0  172.16.11.138.7575 172.16.11.138.52014ESTABLISHED
 
 I enabled the debug logging for MainClientExec class and it gives me the 
 below information. Here it does not exactly tell me what is the source port 
 that is being used for the connection. How can I use this ? Also, whenever 
 MainClientExec prints the below line, does it mean a new connection is 
 created with a new port or an existing port is being used ?
 
 2014-02-14 13:01:01 DEBUG MainClientExec:217 - Opening connection 
 {}-http://172.16.11.138:8983
 
 2) My code looks like this for creating a http Client. I pass the client to 
 SolrJhttps://cwiki.apache.org/confluence/display/solr/Using+SolrJ which 
 uses this client to make requests. SolrJ makes requests to a Solr server.
 
 HttpClientBuilder httpBuilder = HttpClientBuilder.create();
 Builder socketConfig =  SocketConfig.custom();
 socketConfig.setSoReuseAddress(true);
 httpBuilder.setDefaultSocketConfig(socketConfig.build());
 httpBuilder.setMaxConnTotal(100);
 httpBuilder.setMaxConnPerRoute(100);
 httpBuilder.disableRedirectHandling();
 httpBuilder.useSystemProperties();
 final CloseableHttpClient httpClient = httpBuilder.build();
 
 This httpClient created above also throws some exceptions. Can I assume retry 
 is successful and does this happen because I enabled socketReuseAddress ?
 
 2014-02-14 12:53:21,415 - INFO  [CloudSolrServer 
 ThreadPool-1-thread-59:RetryExec@93] - I/O exception 
 (java.net.SocketException) caught when processing request: Address already in 
 use
 2014-02-14 12:53:21,443 - INFO  [CloudSolrServer 
 ThreadPool-1-thread-59:RetryExec@106] - Retrying request

When it comes to the HttpClient side of this, I don't have much idea
what's going on ... but I do have a little bit of experience with SolrJ.

I have run into a lot of problems when trying to fully migrate Solr to
HttpClient 4.3, most of which are likely a result of my own
inexperience.  Most of SolrJ's interaction with HttpClient is now
deprecated.

https://issues.apache.org/jira/browse/SOLR-5604

For my own SolrJ code, I couldn't get it to work when I used
HttpClientBuilder to make the HttpClient object for SolrJ.  I don't
remember what the exceptions looked like.  What I ended up doing to
avoid deprecations in my own code was using the utilities built into
SolrJ.  I basically kicked the problem upstream.  Here's how I build an
HttpClient in my own SolrJ code now, one that is shared between all my
HttpSolrServer instances:

ModifiableSolrParams params = new ModifiableSolrParams();
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS_PER_HOST, 300);
params.add(HttpClientUtil.PROP_MAX_CONNECTIONS, 5000);
httpClient = HttpClientUtil.createClient(params);

https://lucene.apache.org/solr/4_6_0/solr-solrj/org/apache/solr/client/solrj/impl/HttpClientUtil.html

Most of the options you have above in your code are available in the
Solr utility class.  Note that the object built under all these layers
is currently the deprecated SystemDefaultHttpClient class, which is
modifiable after it is created.  The new objects in HttpClient 4.3 are
NOT modifiable after creation.

There's still a lot of work left to do for SOLR-5604.  Until that work
is done, I don't think you'll have much luck using the new classes
introduced by HttpClient 4.3.x.

Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org



Re: Migrating to HttpClient 4.3.1 from 4.2.6 - Solr

2014-01-09 Thread Shawn Heisey

On 1/5/2014 9:03 AM, Oleg Kalnichevski wrote:
That is intended. HttpClient instances are expected to be immutable 
(not their dependencies though). This helps make them thread safe 
without incurring an overhead of synchronization. One should customize 
individual requests or execution contexts and never meddle with 
HttpClient instances. 


Thank you for your reply.

This is why I say it will be quite a paradigm shift for Solr.  Solr is 
*almost* there in that usually those settings are only changed when the 
server object is first created, but typically it's done by creating the 
HttpClient object and then modifying its defaults.  Now the create the 
SolrServer and/or HttpClient with appropriate defaults paradigm will 
need to be 100% explicit.


How much of a bad idea would it be to create a new HttpClient object 
when a default parameter at the SolrServer level needs to change, 
turning the old one into collectable garbage?  I don't like the idea, 
just curious.


There is no migration guide but HttpClient programming principles are 
covered here 
http://hc.apache.org/httpcomponents-client-4.3.x/tutorial/html/index.html 


I think it's going to take me a while before I can fully wrap my head 
around what to do here.  I look forward to figuring it all out and 
learning a lot in the process.


Thanks,
Shawn


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org