Re: How to implement httpPatch in httpCLient 4.1.3

[Oleg Kalnichevski]

On Mon, 2017-02-20 at 13:14 +, Khare, Aparna wrote:
> Dear Users,
> 
>   I have a requirement where I want to implement HTTPPatch method.
> The problem is we are still on httpClient 4.1.3 which does not have a
> default method for PATCH.
> 
> I need help if there are ways I can do the same.
> 
> Looking forward to the experts.
> 

---
public class HttpPatch extends HttpEntityEnclosingRequestBase {

public HttpPatch() {
super();
}

public HttpPatch(final URI uri) {
super();
setURI(uri);
}

public HttpPatch(final String uri) {
super();
setURI(URI.create(uri));
}


@Override
public String getMethod() {
return "PATCH";
}
}
---

Hope this helps

Oleg

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents HttpAsyncClient 4.1.3 GA Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.1.3 GA
release of HttpComponents HttpAsyncClient.

This is a maintenance release that fixes a number of minor issues
discovered since 4.1.2 and upgrades HttpCore and HttpClient
dependencies.

---
Download - 


Release notes -


HttpComponents site -


---
About Apache HttpAsyncClient

Although the java.net package provides basic functionality for
accessing resources via HTTP, it doesn't provide the full flexibility
or functionality needed by many applications. HttpAsyncClient seeks to
fill this void by providing an efficient, up-to-date, and feature-rich
package with an event-driven programming interface based on a non-
blocking I/O model.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Unknown/invalid Content-Encoding response header

[Oleg Kalnichevski]

On Mon, 2017-02-06 at 12:10 +0100, Oliver Stöneberg wrote:
> A few days ago I wrote some test for some code I am using HttpClient 
> in. The current code is using 4.5.2. After I finsihed the tests I 
> backported those to anolder version of the compomonent I am using
> the 
> library in which still uses 4.3.x. The tests worked fine with the 
> latest version, but with the older one I got the following exception:
> 
> Caused by: org.apache.http.HttpException: Unsupported Content-Coding: 
> UTF-8
>   at
> org.apache.http.client.protocol.ResponseContentEncoding.process(Respo
> nseContentEncoding.java:98)
>   at
> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttp
> Processor.java:139)
>   at
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java
> :200)
>   at
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
>   at
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java
> :108)
>   at
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttp
> Client.java:184)
>   ... 37 more
> 
> It turns out some part of the code was using "Content-Encoding" to 
> specify the charset of the response content. It feels a bit strange 
> to me that a newer version would not report such a serious error. I 
> looked at the code in question and I can see that 
> ResponseContentEncoding.java change quite a bit and now has an 
> "ignoreUnknown" flag for these cases - unfortunately that appears to 
> be hard-coded to "true" now and cannot be configured.
> 
> I browsed the release notes for some insight on this and it appears 
> to be that the fix for 
> 
> * [HTTPCLIENT-1550] Fixed 'deflate' zlib header check.
>   Contributed by Oleg Kalnichevski 
> 
> caused this change in behavior. Seems to me like the fix went a bit 
> too far on this.
> 

See HTTPCLIENT-1643 for details

https://issues.apache.org/jira/browse/HTTPCLIENT-1643

Oleg

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: I/O thread blocking under high concurrent load

[Oleg Kalnichevski]

On February 2, 2017 7:05:07 AM GMT+01:00, Sudheera Palihakkara 
 wrote:
>Hi all,
>
>I'm currently using httpcore-nio 4.4 for a proxy-like application and
>make
>use of the file channels when content transferring. Recently I have
>profiled my code under a high concurrent load and observed few unusual
>thread blocking occurrences. Stack traces are listed below.
>
>*I/O threads*
>
>S-I/O-dispatcher-1 [BLOCKED]
>sun.nio.ch.EPollSelectorImpl.doSelect(long) EPollSelectorImpl.java:88
>java.lang.Thread.run() Thread.java:745
>
>S-I/O-dispatcher-5 [BLOCKED]
>org.apache.http.impl.nio.reactor.IOSessionImpl.setEvent(int)
>IOSessionImpl.java:164
>org.apache.http.impl.nio.DefaultNHttpClientConnection.submitRequest(HttpRequest)
>DefaultNHttpClientConnection.java:339
>
>*Worker threds*
>
>pool-2-thread-99 [BLOCKED]
>sun.nio.ch.EPollArrayWrapper.setInterest(int, int)
>EPollArrayWrapper.java:213
>org.apache.http.nio.protocol.HttpAsyncService$HttpAsyncExchangeImpl.submitResponse(HttpAsyncResponseProducer)
>HttpAsyncService.java:1037
>
>pool-2-thread-96 [BLOCKED]
>sun.nio.ch.EPollSelectorImpl.wakeup() EPollSelectorImpl.java:191
>org.apache.http.nio.protocol.HttpAsyncService$HttpAsyncExchangeImpl.submitResponse(HttpAsyncResponseProducer)
>HttpAsyncService.java:1037
>
>Previously I have used httpcore-nio 4.2.5 for the same application and
>did
>not encountered any such thread blockings. Can you help me out to
>figure
>out what might be the issue here? Thank you very much.

Synchronized sections of code can cause threads to block on the synchronization 
object under load. This does not necessarily mean there is a problem.

Oleg
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Support HTTP/2 protocol

[Oleg Kalnichevski]

On Mon, 2017-01-30 at 08:43 -0700, Shawn Heisey wrote:
> On 1/26/2017 8:15 AM, Oleg Kalnichevski wrote:
> > ALPN will be supported as soon as it is supported by the Java
> > platform
> > (which is not going to happen until Java 9). 
> 
> I see evidence that the other Java http implementations have ALPN
> support already ... but those systems implement both server and
> client. 
> Could it be that those ALPN implementations are server-side only?  I
> can't seem to easily locate anything saying for sure.
> 

No, it could not. HttpCore also implements both server and client side
transports.

Please take a closer look at the existing ALPN implementations, for
instance, that provided by Jetty. It requires a custom agent that works
with specific versions of OpenJDK only.

If one does not have a problem going that route there is nothing
stopping them from using Jetty ALPN agent to build a custom protocol
negotiator for HttpClient. 


> > ALPN can be used to advertise server protocol capabilities at the
> > time
> > of SSL handshake and allow clients to pick the desired protocol
> > from
> > the list of supported protocols. If one knows supported protocols
> > beforehand ALPN is completely useless. Clients can go straight to
> > using
> > HTTP/2 if the server is known to support it.
> > 
> > In the next release of HttpCore I would like to add protocol
> > detection
> > logic to enable endpoints to detect HTTP protocol version by
> > examining
> > the first packet received from the opposite endpoint. This in my
> > opinion would be a much more practical feature. ALPN presently is
> > very
> > low on my priority list.
> > 
> > Having said that ALPN support contribution would be welcome if
> > someone
> > is willing to develop it.
> 
> Knowing in advance that HTTP/2 support is available may be
> problematic. 
> I can imagine a situation where servers are upgraded gradually, and
> the
> client may not know whether the one it's connecting to can support
> the
> new protocol.  Can HTTP/2 detection be reliable without ALPN, even in
> situations where connecting to the same host/port may support HTTP/2
> on
> one connection, but not the next?  TCP load balancing is relatively
> common with SSL.  If such detection can be reliable, then there won't
> be
> anything to worry about.
> 

I see no reason why it could not, given that HTTP/2 connection preface
message is essentially a specially crafted HTTP/1.1 compatible request
message.

Oleg

> For my webserver installations, I am hoping to get HTTP/2 support
> enabled in the load balancer and worry about support on the back end
> later.  I'm expecting the back-end LAN to be fast enough that
> multiple
> connections can easily be established while the Internet-facing side
> works through the inherent packet latency.
> 
> Thanks,
> Shawn
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Client 4.5.3 GA Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.5.2 GA
release of HttpComponents HttpClient.

HttpClient 4.5.3 is a maintenance release that fixes a number of minor
bugs reported since 4.5.2.

Download - 
Release notes -

HttpComponents site - 

About HttpComponents HttpClient

The Hyper-Text Transfer Protocol (HTTP) is perhaps the most significant
protocol used on the Internet today. Web services, network-enabled
appliances and the growth of network computing continue to expand the
role of the HTTP protocol beyond user-driven web browsers, while
increasing the number of applications that require HTTP support.

Although the java.net package provides basic functionality for
accessing resources via HTTP, it doesn't provide the full flexibility
or functionality needed by many applications. HttpClient seeks to fill
this void by providing an efficient, up-to-date, and feature-rich
package implementing the client side of the most recent HTTP standards
and recommendations.

Designed for extension while providing robust support for the base HTTP
protocol, HttpClient may be of interest to anyone building HTTP-aware
client applications such as web browsers, web service clients, or
systems that leverage or extend the HTTP protocol for distributed
communication.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Support HTTP/2 protocol

[Oleg Kalnichevski]

On Thu, 2017-01-26 at 14:32 +0100, Philippe Mouawad wrote:
> Hello ,
> Oleg kindly proposed to help JMeter project in adding HTTP/2 support.
> 
> We have started this thread to work on design.
> 
> As per Andrei remark, it seems ALPN is not yet supported by current
> HTTPClient 5.x version.
> Is there some visibility on its support ?
> 


ALPN will be supported as soon as it is supported by the Java platform
(which is not going to happen until Java 9). 

ALPN can be used to advertise server protocol capabilities at the time
of SSL handshake and allow clients to pick the desired protocol from
the list of supported protocols. If one knows supported protocols
beforehand ALPN is completely useless. Clients can go straight to using
HTTP/2 if the server is known to support it.

In the next release of HttpCore I would like to add protocol detection
logic to enable endpoints to detect HTTP protocol version by examining
the first packet received from the opposite endpoint. This in my
opinion would be a much more practical feature. ALPN presently is very
low on my priority list.

Having said that ALPN support contribution would be welcome if someone
is willing to develop it.

Oleg 



> Thanks for your help.
> Regards
> 
> On Thu, Jan 26, 2017 at 2:13 PM, Andrey Pokhilko  wrote:
> 
> > Hi,
> > 
> > From my experiments, I see that lack of two specific features make
> > it
> > not useful. According to https://hc.apache.org/news.html:
> > 
> >   * No ALPN support yet
> >   * no connection upgrade
> > 
> > Especially ALPN part is crucial for protocol functioning. Is there
> > any
> > ETA from Oleg when it will become available?
> > 
> > 
> > In general, we can start designing the "synchronous way" solution.
> > From
> > my understanding, it is doable and will be good enough for the
> > beginning.
> > 
> > 
> > Andrey Pokhilko
> > 
> > On 25.01.2017 23:38, Philippe Mouawad wrote:
> > > Hello
> > > I'd like to start a thread on this particular item for which an
> > 
> > enhancement
> > > exists:
> > > 
> > >    - https://bz.apache.org/bugzilla/show_bug.cgi?id=59847
> > > 
> > > The aim of this thread is to discuss, throw ideas on how we could
> > 
> > implement
> > > this in JMeter.
> > > 
> > > Oleg K. from HttpComponents project has nicely proposed to help
> > > on it.
> > > 
> > > I see at least 2 parts in this item:
> > > 
> > >    - The Sampler
> > >    - The Recorder
> > > 
> > > 
> > > 
> > > *Sampler:*
> > > We have 2 options:
> > > 
> > >    - build a usual "synchronous" sampler similar to HTTP:
> > >   - Is this realistic ?
> > >   - Does it perform well ?
> > >   - + : It should not be too complex
> > >    - build a new "Asynchronous sampler":
> > >   - Is this realistic ?
> > >   - + We could gain more performance
> > >   - - It is a huge piece of work as we need to change JMeter
> > > model
> > > 
> > > *Recorder:*
> > > 
> > > I think we need to introduce a new more generic Recorder as the
> > > current
> > > Test Script Recorder is too tightly linked to HTTP 1.X protocol
> > > 
> > > 
> > > Regards
> > > Philippe M.
> > > @philmdot
> > > 
> > 
> > 
> 
> 

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: PoolingClientConnectionManager lease/release connection problem with SSLConnectionFactory

[Oleg Kalnichevski]

On Fri, 2017-01-20 at 15:17 +0800, sdy wrote:
> hi, folks,
>   I'm working with httpclient (4.2.1) on suse linux
> enterprise server 11(x86_64),  and construct the connectionpool
> manager code as follow:
> 
> 
>  KeyStore keyStore = getKeyStore(KEY_STORE_CLIENT_PATH,
> KEY_STORE_CLIENT_PASS);
> KeyStore trustStore = getKeyStore(KEY_STORE_TRUST_PATH,
> KEY_STORE_TRUST_PASS);
> socketFactory = new SSLSocketFactory(keyStore, KEY_STORE_CLIENT_PASS,
> trustStore);
> socketFactory.setHostnameVerifier(SSLSocketFactory.ALLOW_ALL_HOSTNAME
> _VERIFIER);
> connManager = new PoolingClientConnectionManager();  
> connManager.setMaxTotal(100);  
> connManager.setDefaultMaxPerRoute(100);  
> connManager.getSchemeRegistry().register(new Scheme("https", 443,
> socketFactory));
> params = new BasicHttpParams();
> params.setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT,5000);
> params.setParameter(CoreConnectionPNames.SO_TIMEOUT, 5000);
> httpClient = new DefaultHttpClient(connManager, params);
> 
> 
> 
> 
> and do the post request with the following method:
> 
> 
> private String doPost(String url, String payload) {
> HttpPost httpPost = null;
> String result = null;
> HttpResponse response = null;
> try {
> httpPost = new HttpPost(url);
> StringEntity postEntity = new StringEntity(payload);
> httpPost.setEntity(postEntity);
> httpPost.addHeader("Content-Type", "application/json");
> httpPost.addHeader("Authorization", "Basic " + this.authString);
> response = this.httpClient.execute(httpPost);
> result = EntityUtils.toString(response.getEntity());
> EntityUtils.consume(response.getEntity());
> } catch (Exception e) { 
> e.printStackTrace();
> return null;
> }finally {
> if(response != null){
> if(response.getEntity() != null){
> InputStream input;
> try {
> input = response.getEntity().getContent();
> if(input != null){
>    input.close();
>    }
> } catch (IllegalStateException | IOException e) {
> // TODO Auto-generated catch block
> e.printStackTrace();
> }
> }
> }
> }
> return result;
> }
> 
> 
> the problem for me is that I want to use the keep-alive ability
> of PoolingClientConnectionManager, but the httpclient close the
> connection after the single post request.  the debug log as follow.
> Who konws 
> what's wrong? Is it the bug of the httpclient ? or I have done
> something wrong with my code?
> 
> 


There is nothing wrong. Connections associated with a particular SSL
principal are considered state-full. Please see this section of the
tutorial 

http://hc.apache.org/httpcomponents-client-4.5.x/tutorial/html/advanced
.html#stateful_conn

Oleg


> DEBUG [org.apache.http.impl.conn.PoolingClientConnectionManager]
> Connection request: [route: {s}->https://api.svip.jiguang.cn][total
> kept alive: 100; route allocated: 100 of 100; total allocated: 100 of
> 100]
> DEBUG [org.apache.http.impl.conn.DefaultClientConnection] Connection
> 0.0.0.0:37394<->183.240.12.100:443 closed
> DEBUG [org.apache.http.impl.conn.PoolingClientConnectionManager]
> Connection leased: [id: 100][route: {s}->https://api.svip.jiguang.cn]
> [total kept alive: 99; route allocated: 100 of 100; total allocated:
> 100 of 100]
> DEBUG [org.apache.http.impl.conn.DefaultClientConnectionOperator]
> Connecting to api.svip.jiguang.cn:443
> DEBUG [org.apache.http.client.protocol.RequestAddCookies] CookieSpec
> selected: best-match
> DEBUG [org.apache.http.client.protocol.RequestAuthCache] Auth cache
> not set in the context
> DEBUG [org.apache.http.client.protocol.RequestProxyAuthentication]
> Proxy auth state: UNCHALLENGED
> DEBUG [org.apache.http.impl.client.DefaultHttpClient] Attempt 1 to
> execute request
> DEBUG [org.apache.http.impl.conn.DefaultClientConnection] Sending
> request: POST /v3/push HTTP/1.1
> DEBUG [org.apache.http.headers] >> POST /v3/push HTTP/1.1
> DEBUG [org.apache.http.headers] >> Content-Type: application/json
> DEBUG [org.apache.http.headers] >> Authorization: Basic
> ZGQxMjEwYWJhYTNhZWZmYTFkMDZjNDhlOjA5MjRiZThhMDdlNWJiZjViYTdkYTNmZA==
> DEBUG [org.apache.http.headers] >> Content-Length: 471
> DEBUG [org.apache.http.headers] >> Host: api.svip.jiguang.cn
> DEBUG [org.apache.http.headers] >> Connection: Keep-Alive
> DEBUG [org.apache.http.impl.conn.DefaultClientConnection] Receiving
> response: HTTP/1.1 200 OK
> DEBUG [org.apache.http.headers] << HTTP/1.1 200 OK
> DEBUG [org.apache.http.headers] << Server: nginx
> DEBUG [org.apache.http.headers] << Date: Fri, 20 Jan 2017 07:05:02
> GMT
> DEBUG [org.apache.http.headers] << Content-Type: application/json
> DEBUG [org.apache.http.headers] << Transfer-Encoding: chunked
> DEBUG [org.apache.http.headers] << Connection: keep-alive
> DEBUG [org.apache.http.headers] << X-Rate-Limit-Limit: 33000
> DEBUG [org.apache.http.headers] << X-Rate-Limit-Remaining: 32799
> DEBUG [org.apache.http.headers] << X-Rate-Limit-Reset: 4
> DEBUG 

Re: Location based RedirectStrategy

[Oleg Kalnichevski]

On Mon, 2017-01-16 at 13:22 +0100, Lucas Ventura Carro wrote:
> I wanted to develop an specific RedirectStrategy, which will accept
> the
> redirect based on the location URI returned in response.
> 
> So my idea was to override the 'isRedirected' method and, retrieve
> the URI
> with 'getLocationURI', decide redirection based on this value and a
> fixed
> set -- as method 'isRedirectable' doesn't fit my needs.
> 
> But when reviewing the code from
> DefaultRedirectStrategy.getLocation[1]
> I've realized that each time is called, it will store the URI in the
> current request context.
> Is this correct? Should not this be done on the caller method
> 'getRedirect'
> instead? As 'getLocation' seems more calculating, than doing the
> redirection itself.
> 
> [1]:
> https://hc.apache.org/httpcomponents-client-ga/httpclient/xref/org/ap
> ache/http/impl/client/DefaultRedirectStrategy.html#122
> 
> --
> Lucas

Hi Lucas 

You are right. #getRedirect might be a better place for circular
redirect detection logic. Feel free to raise a JIRA with a change
request or raise PR at GitHub or both.

Oleg 

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 4.4.6 GA released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.4.6 GA
release of HttpComponents Core. 

This is a maintenance release that fixes a number of issues discovered
since 4.4.5.

Please note that as of 4.4 HttpCore requires Java 1.6 or newer.

Download - 
Release notes -

HttpComponents site - 

About HttpComponents Core

HttpCore is a set of low level HTTP transport components that can be
used to build custom client and server side HTTP services with a
minimal footprint. HttpCore supports two I/O models: a blocking I/O
model based on the classic Java I/O and a non-blocking, event driven
I/O model based on Java NIO.


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Parse cookie with HttpClient?

[Oleg Kalnichevski]

On Sun, 2017-01-01 at 23:57 -0500, davidjesse...@aol.com wrote:
> I am working with a web service that returns cookies in the response
> http body instead of the response headers.
> I need to add these cookies to the HtmlUnit WebClient instance.
> I have extracted the cookie string. Are there any methods that
> automatically parse the cookie string? or would I have to parse the
> cookie manually and separate it by domain, name, value, etc...?

CookeSpec#parse is your friend.

Oleg

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HttpClient BUFFER_WARN_TRIGGER_LIMIT replacement in 4.x

[Oleg Kalnichevski]

On Thu, 2016-12-29 at 10:59 +, Müller, Stephan wrote:
> Ahoi!
> 
> I started migrating a legacy codebase using httpclient 3.x to use
> httclient 4.5.2.
> 
> Our code used to set 
> params.setParameter("http.method.response.buffer.warnlimit",responseB
> ufferWarnLimit);
> on HttpConnectionParams from a HttpConnectionManager
> 
> I am unable to find any hints on what happened to this parameter in
> 4.x or how to "simulate" it.
> What to do now? Is there still something similar? 
> 
> Any help is appreciated!
> 
> Regards,
> Stephan

Stephan


Unlike HC 3.1 HC 4.x never buffers message content in memory. This
parameter is simply no longer applicable.

Oleg

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: http client sample failing for Digest authentication

[Oleg Kalnichevski]

On Wed, 2016-12-28 at 00:13 +, David Thielen wrote:
> I am running the sample for digest authentication. I didn't change anything, 
> just using the provided sample and I am getting the error "HTTP/1.1 401 
> UNAUTHORIZED"
> 
> If I go direct to http://httpbin.org/digest-auth/auth/user/passwd in prompts 
> me for user/passwd and then provides the page. So the website is working 
> right.
> 
> Looking at Fiddler, there are 2 requests and the second has for auth:
> 
> No Proxy-Authorization Header is present.
> Authorization Header is present: Digest username="user", 
> realm="m...@kennethreitz.com", nonce="b2c603bb7c93cfa197945553a1044283", 
> uri="/digest-auth/auth/user/passwd", 
> response="5bc193bd90e3acb1cb9242138e6bd100", 
> opaque="a6766136f0585f6a7daeb124cdcdbd45", qop=auth, nc=0001, 
> cnonce="9609c2bb4a21dd0a"
> 
> No idea where that realm came from - that is not me.
> 
> What am I doing wrong?
> 
> Thanks - dave

Dave

(1) If you are referring to ClientPreemptiveDigestAuthentication it is
not meant to be used without modification. One is expected to provide
correct realm and nounce values.

(2) More importantly something has changed on the httpbin.org side and
their digest authentication logic no longer works with HttpClient. I had
no bandwidth to find out the cause of the problem. Volunteers are
welcome.

Oleg  


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 5.0 alpha2 released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 5.0-alpha2
release of HttpComponents Core. 

This is a major release that renders HttpCore API incompatible with the
stable 4.x branch and upgrades HTTP/1.1 and HTTP/2 protocol conformance
to the requirements and recommendations of the latest protocol
specification.

Notable changes and features included in the 5.0 series are:

* Partial support for HTTP/2 protocol and partial conformance to
requirements and recommendations of the latest HTTP/2 protocol
specification (RFC 7540, RFC 7541)

  Supported features:

** HPACK header compression
** stream multiplexing
** flow control
** response push (client and server)
** message trailers
** expect-continue handshake
** connection validation (ping)
** TLS 1.2 features

   Unsupported features:

** application-layer protocol negotiation (ALPN)
** padding of outgoing frames
** stream priority
** plain connection HTTP/1.1 upgrade
** CONNECT method
** TLS renegotiation and compression cannot be disabled with Java
1.7 JSSE APIs

* Improved conformance to requirements and recommendations of the latest
HTTP/1.1 protocol specification (RFC 7230, RFC 7231)

* New asynchronous HTTP transport APIs consistent for both HTTP/1.1 and
HTTP/2 transport.

* Improved HTTP/1.1 and HTTP/2 requester and server implementations.

* Redesigned connection pool implementation with reduced pool lock
contention.

* Support for HTTP/1.1 protocol switch / upgrade.

* Package name space changed to 'org.apache.hc.core5'

* Maven group id changed to 'org.apache.httpcomponents.core5' 

HttpCore 5.0 releases can be co-located with earlier versions.

Please note that as of 5.0 HttpCore requires Java 1.7 or newer.

Please note that at this point 5.0 APIs are considered API experimental
and unstable and are expected to change in the coming releases without
providing a migration path.

Download - 
Release notes -

HttpComponents site - 

About HttpComponents Core

HttpCore is a set of HTTP/1.1 and HTTP/2 transport components that can
be used to build custom client and server side HTTP services with a
minimal footprint.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Using HttpClientContext to get a server proxy's certificates

[Oleg Kalnichevski]

On Sat, 2016-12-17 at 18:05 -0500, Qiang Cao wrote:
> Hi Everyone,
> 
> I'm running my client to talk to a service that is behind a proxy. The
> channel between the client and the server proxy is established through SSL.
> I wish to retain the certificates of the proxy.
> 
> Client --- SSL socket (HTTPS) ---> SSL PROXY --- Plain socket (HTTP) --->
> Server
> 
> The way I do it is to create a context for each request and grab the SSL
> session info from the context after each request is executed.
> 
> ..
> HttpClientContext clientContext = HttpClientContext.create();
> CloseableHttpResponse response = httpclient.execute(target, httppost,
> clientContext);
> ManagedHttpClientConnection conn = clientContext.getConnection(
> ManagedHttpClientConnection.class);
> 
> if(conn.isOpen()) {
>   SSLSession sslsession = conn.getSSLSession();
>X509Certificate[] peerCertChain = sslsession.getPeerCertificateChain();
> }
> 
> The code works fine when there is no proxy and the server runs SSL. In that
> case, I'm able to get the server certificates.  However, with the SSL proxy
> in the middle, the connection (ManagedHttpClientConnection) I got from the
> context is always NOT open. With that, I sort of ran out of ideas to get
> the SSL session for the request.  Any thoughts?
> 
> Thanks in advance!
> 
> -Qiang

Hi Qiang

Connection socket factory should be a better injection point for any
custom SSL logic 

---
CloseableHttpClient client = HttpClientBuilder.create()
.setSSLSocketFactory(new 
SSLConnectionSocketFactory(SSLContexts.createSystemDefault()) {

@Override
public Socket createLayeredSocket(
final Socket socket,
final String target,
final int port,
final HttpContext context) throws IOException {
final SSLSocket layeredSocket = (SSLSocket) 
super.createLayeredSocket(socket, target, port, context);
SSLSession sslsession = layeredSocket.getSession();
X509Certificate[] peerCertChain = 
sslsession.getPeerCertificateChain();
return layeredSocket;
}
})
.build();
---

Hope this helps

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HttpClient bounded download size

[Oleg Kalnichevski]

On Mon, 2016-12-12 at 17:47 -0500, Joseph Naegele wrote:
> Hi all, back again.
> 
> Now I'm wondering how to properly limit the number of bytes downloaded by the 
> synchronous HttpClient (v4.5, i.e. most recent versions).
> 
> Like the HttpAsyncClient, even if I only read the desired number of bytes 
> from a response entity's InputStream, closing the entity results in the 
> remainder of the response entity being downloaded from the connection. I 
> found this in ContentLengthInputStream's close method, which is wrapped by 
> EofSensorInputStream.
> 
> Again, my goal is to intentionally collect a truncated response if it's 
> larger than my limit.
> 
> Thanks!
> Joe Naegele
> 

Call CloseableHttpResponse#close without closing the entity. 

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HttpAsyncClient bounded download size

[Oleg Kalnichevski]

On Tue, 2016-12-06 at 11:21 -0500, Joseph Naegele wrote:
> Hi folks,
> 
> How can I limit the amount of data downloaded for a request executed by the 
> HttpAsyncClient and still process the response as "completed" in the 
> registered FutureCallback? The use case is a large scale web crawler that 
> truncates resources deemed too large.
> 
> I started by limiting the amount of data read from the response entity's 
> InputStream, however this doesn't work with the default 
> BasicAsyncResponseConsumer, because it uses the dynamically expanding 
> SimpleInputBuffer to download the entire response entity.
> 
> I implemented my own HttpAsyncResponseConsumer, similar to the 
> BasicAsyncResponseConsumer, and tried using IOControl to signal shutdown once 
> the I've read maximum desired number of bytes, however this triggers a 
> ConnectionClosedException. This is undesirable because I can't distinguish it 
> from other causes of ConnectionClosedExceptions, and I want to treat 
> "truncated" responses as completed in the registered FutureCallback (where I 
> post-process the response).
> 
> Is there another method of implementing my desired functionality?
> 
> Thanks,
> Joe Naegele
> 

Hi Joe

Make your custom HttpAsyncResponseConsumer throw a custom exception,
like IHadEnoughException or some such, and see if it gets correctly
propagated to the result callback. If it does not, please raise an issue
in JIRA.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: DNS Round Robin

[Oleg Kalnichevski]

On Fri, 2016-12-02 at 12:55 -0500, Murat Balkan wrote:
> Hi all,
> 
> When I do httpclient.execute to the url (
> https://mc.yandex.ru/metrika/watch.js) the the execute call hangs more than
> the timeout parameters.  I have socket timeout set to 15 seconds and
> connect timeout set to 15 seconds.
> 
> That site has a DNS round robin and it seems that Httpclient automatically
> tries the next IP address in the resolved address list.
> 
> Is it possible to cancel that behavior?
> 
> Thanks a lot,
> Murat

Murat

You can do so by using a custom DnsResolver

CloseableHttpClient httpclient = HttpClients.custom()
.setDnsResolver(host ->  new InetAddress[] { 
InetAddress.getByName(host) })
.build();

Hope this helps

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HttpContext To Url

[Oleg Kalnichevski]

On Tue, 2016-11-29 at 16:19 -0500, Murat Balkan wrote:
> Hi,
> Whats the best way to reach the request target Url from the HttpContext
> object?
> Thank you,
> Murat

http://hc.apache.org/httpcomponents-core-4.4.x/httpcore/apidocs/org/apache/http/protocol/HttpCoreContext.html#getTargetHost()

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: RFC 6265 and non-prefix cookie path

[Oleg Kalnichevski]

On Fri, 2016-11-25 at 16:02 +0100, Schulz-Hildebrandt, Ole wrote:
> Hi,
> 
> I'm using httpclient 4.5.2 and the CookieSpec Standard.
> 
> If the response to a http request to http://.../abc contains a cookie for the 
> path /def this cookie is rejected by httpclient. This is the correct behavior 
> in case of RFC 2109 (cf. 
> https://issues.apache.org/jira/browse/HTTPCLIENT-1043). But RFC 6265 (as far 
> as I know) does not state that a cookie path must be a prefix of the request 
> uri path. In 8.6 it is even mentioned as a "security problem" that 'an HTTP 
> response to a request for http://example.com/foo/bar can set a cookie with a 
> Path attribute of "/qux"'.
> 
> I know that I can workaround my problem by using a custom cookie policy. I 
> just wondered if this behavior of httpclient is correct with respect to RFC 
> 6265.
> 
> Best regards
> Ole

Hi Ole

I skimmed through the RFC and also could not find a statement supporting
this behavior. This is likely to be a left over from earlier
implementations of cookie specs.

Please feel free to raise am issue in JIRA for this defect.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: scala: apache httpclient in multi-threaded environment. confusion on correct way of use

[Oleg Kalnichevski]

On Sat, 2016-11-12 at 20:16 +0530, Nishant Kumar wrote:
> Hi,
> 
> I am writing a singleton class (Object in scala) which uses apache
> httpclient(4.5.2) to send some file or some json content to a server and
> return status to caller. Here is my class -
> 
> object HttpUtils{
>   protected val retryHandler = new HttpRequestRetryHandler() {
> def retryRequest(exception: IOException, executionCount: Int,
> context: HttpContext): Boolean = {
>   //retry logic
>   true
> }
>   }
>   private val connectionManager = new
> PoolingHttpClientConnectionManager()
> 
>   // Reusing same client for each request that might be coming from
> different threads .
>   // Is it correct 

Yes.

>   val httpClient = HttpClients.custom()
> .setConnectionManager(connectionManager)
> .setRetryHandler(retryHandler)
> .build()
> 
>   def restApiCall (url : String, rDD: RDD[SomeMessage]) : Boolean = {
> // Creating new context for each request
> val httpContext: HttpClientContext = HttpClientContext.create
> val post = new HttpPost(url)
> 
> // convert RDD to text file using rDD.collect
> 
> // add this file as MultipartEntity to post
> 
> var response = None: Option[CloseableHttpResponse] // Is it correct
> way of using it ???

Not really. HttpClient never returns null response messages. The
'response' instance is non-optional.

> try {
>   response = Some(httpClient.execute(post, httpContext))
>   val responseCode = response.get.getStatusLine.getStatusCode
>   EntityUtils.consume(response.get.getEntity) // Is it require ???

Not required but advised if you want to ensure re-use of persistent
connections.

>   if (responseCode == 200) true
>   else false
> }
> finally {
>   if (response.isDefined) response.get.close
>   post.releaseConnection() // Is it require ???

No, it is not.

> }
>   }
>   def onShutDown = {
> connectionManager.close()
> httpClient.close()

CloseableHttpClient#close also shuts down the connection pool associated
with it.

Oleg

>   }
> }
> 
> 
> Multiple threads (More specifically from spark streaming context) are
> calling `restApiCall` method. I am relatively new to `scala` and `apache
> httpClient`. I have to make frequent connections to only few fixed server
> (i.e. 5-6 fixed URL's with different request parameters). I have also
> written comments in above code as question. Please address them as well.
> 
> I went through multiple online resource but still not confident about it.
> 
>  - Is it the best way to use http client in multi-threaded environment?
>  - Is it possible to keep live connections and use it for various requests
> ? Will it be beneficial in this case ?
>  - Am i using/releasing all resources efficiently ? If not please suggest.
>  - Is it good to use it in Scala or there exist some better library ?
> 
>  Thanks for your help in advance.
> 
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SSL Handshake : timing and timeout

[Oleg Kalnichevski]

On Fri, 2016-11-11 at 11:01 +0100, Stefan Magnus Landrø wrote:
> True. The way SSL sockets are created is complex stuff. They are layered
> and I believe the handshake is async - maybe Oleg could clarify?
> 

There is nothing special or unusual about SSL/TLS handshakes. They
involve multiple IP packet exchanges over a TCP connection. Standard
socket settings fully apply.

Oleg

> Anyways - We ran into an issue a few years ago with an SSL handshake taking
> for ever (a switch was dropping packets of a certain size ... ), and
> tracked our hanging threads down to blocking because the SoTimeout was set
> to 0 by default. Changing this prevented threads from hanging for ever.
> 
> Ref javadoc:
> 
> Determines the default socket timeout value for non-blocking I/O operations.
> 
> 2016-11-11 10:01 GMT+01:00 Philippe Mouawad :
> 
> > Thanks but I see not property related to Ssl Handshake timeout.
> >
> > Regards
> >
> > On Friday, November 11, 2016, Stefan Magnus Landrø <
> > stefan.lan...@gmail.com>
> > wrote:
> >
> > > https://hc.apache.org/httpcomponents-client-ga/
> > > httpclient/apidocs/org/apache/http/impl/client/HttpClientBuilder.html#
> > > setDefaultSocketConfig(org.apache.http.config.SocketConfig)
> > >
> > > 2016-11-11 9:34 GMT+01:00 Philippe Mouawad  > > >:
> > >
> > > > Thank you Stefan
> > > > Could you point me to some doc ?
> > > >
> >



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: RetryHandler Per GetMethod

[Oleg Kalnichevski]

On Thu, 2016-11-10 at 12:54 -0500, Murat Balkan wrote:
> Hi,
> So I need to put to the Context an attribute and read it in the handler?
> Thank you
> 

Yes, you do. 

Oleg

> On 10 November 2016 at 05:38, Bernd Eckenfels <e...@zusammenkunft.net>
> wrote:
> 
> > In fact the retry handlers (some implementations)  already look at the
> > method and only retry idempotent methods
> >
> > Gruss
> > Bernd
> > --
> > http://bernd.eckenfels.net
> >
> >
> >
> >
> > On Thu, Nov 10, 2016 at 11:15 AM +0100, "Oleg Kalnichevski" <
> > ol...@apache.org> wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, 2016-11-09 at 09:05 -0500, Murat Balkan wrote:
> > > Hello,
> > >
> > > Is it possible to set a retry handler per HttpGet? The following code
> > taken
> > > from the documentation only applies to 3x and seems to be deprecated.
> > >
> > >
> > > httpget.getParams().
> > > setParameter(HttpMethodParams.RETRY_HANDLER, myretryhandler);
> > >
> > >
> > > What I want to achieve is to set disableAutomaticRetries on the
> > httpclient
> > > level and set them at the httpget level.
> > >
> > > Is this supported?
> > >
> >
> > No it is not, but one can use HttpContext attributes to customize the
> > handler's behavior on the per request basis.
> >
> > Oleg
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >
> >
> >
> >
> >
> >
> >
> 
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: NoHttpResponseException in the async client

[Oleg Kalnichevski]

On Thu, 2016-11-10 at 11:31 +0100, Joan Balagueró - ventusproxy wrote:
> Hello,
> 
> We have replaced the httpclient by the async client in our application.
> Everything works fine, but the ‘NoHttpResponseException’ has disappeared
> from our error statistics reports. So, or with the new async pool this error
> does not occur for some reason (that I don’t know) or we are not catching
> this error correctly (more likely).

Async HC does not throw this exception. It is specific to the classic
(blocking) HC.

Oleg 

> 
> We are using an ‘HttpAsyncResponseConsumer’ and overwriting the
> ‘responseReceived’, ‘consumeContent’ and ‘failed’ methods. We
> understand that when a ‘NoHttpResponseException’ occurs,
> ‘responseReceived’ and ‘consumeContent’ are not called, and  the
> ‘failed’ method is the only one that is directly called.
> 
> Our ‘failed’ method looks like this:
> 
> @Override
> public void failed(final Exception e)
> {
>   ProxyServletException pse = null;
> 
>   if (e.getClass() == java.net.SocketTimeoutException.class) pse = new
> ProxyServletException(ErrorConstants.HTTP_RESPONSE_TIMEOUT, e);
>   else if (e.getClass() == java.net.ConnectException.class) pse = new
> ProxyServletException(ErrorConstants.HTTP_CONNECT_TIMEOUT, e);
>   else if (e.getClass() == org.apache.http.NoHttpResponseException.class)
> pse = new ProxyServletException(ErrorConstants.HTTP_NO_RESPONSE, e);  ←
> the error is caugth here 
>   else if (e.getClass() == java.io.IOException.class) pse = new
> ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e);
>   else if (e.getClass() ==
> com.ventusproxy.proxy.servlet.ProxyServletException.class) pse =
> (ProxyServletException)e;
>   else if (e.getClass() ==
> org.apache.http.conn.ConnectionPoolTimeoutException.class) pse = new
> ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e);
>   else if (e.getClass() == java.util.concurrent.TimeoutException.class) pse
> = new ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e);
> 
>   pse = (pse != null ? pse : new
> ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e));
> 
>   ( . . . )
> }
> 
> 
> Is this ok? Or I'm missing something?
> 
> Thanks,
> 
> Joan.
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HttpClient behavior on close with active connections

[Oleg Kalnichevski]

On Wed, 2016-11-09 at 13:43 -0700, Shawn Heisey wrote:
> I'm using SolrJ 6.2.1 in the program I'm writing, which pulls in
> httpclient/httpmime 4.5.2 and httpcore 4.4.5 as dependencies.
> 
> One of the things that my SolrJ code does takes over an hour to
> complete.  The HTTP connection is kept open for all that time.  I'd like
> to find a way for the Solr server to respond immediately and continue
> the operation in the background, but currently there doesn't seem to be
> a way to do that.
> 
> Now I am attempting to make it possible to gracefully shut down
> everything related to what my code does, turn everything into garbage,
> reload the configuration, and build it all back up.  Therefore I need to
> know how certain things behave at close/shutdown, assuming everything is
> not idle.
> 
> What happens to a long-lived HTTP connection if another thread calls
> close() on the HttpClient?  Does the connection immediately die and
> throw an exception, or would the connection finish as expected in its
> own time? 

Hi Shawn

Yes, it does. CloseableHttpClient#close immediately purges the internal
pool of all connections both active (leased) and inactive (kept alive).
HC 5.0 will support graceful shutdown option whereby one can give
connections some grace period to complete ongoing exchanges prior to
shutting down. 
  
Hope this helps

Oleg

>  If the connection remains open until completion, does the
> close() call return immediately, or wait until that connection ends,
> whether successful or not?  Are all object references (threads in
> particular) properly released and turned into garbage?  I tried to
> follow the code, but got lost quickly.  If it's documented somewhere,
> please point me at that documentation.
> 
> Thanks,
> Shawn
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: RetryHandler Per GetMethod

[Oleg Kalnichevski]

On Wed, 2016-11-09 at 09:05 -0500, Murat Balkan wrote:
> Hello,
> 
> Is it possible to set a retry handler per HttpGet? The following code taken
> from the documentation only applies to 3x and seems to be deprecated.
> 
> 
> httpget.getParams().
> setParameter(HttpMethodParams.RETRY_HANDLER, myretryhandler);
> 
> 
> What I want to achieve is to set disableAutomaticRetries on the httpclient
> level and set them at the httpget level.
> 
> Is this supported?
> 

No it is not, but one can use HttpContext attributes to customize the
handler's behavior on the per request basis.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: NoHttpResponseException Timer?

[Oleg Kalnichevski]

On Tue, 2016-11-08 at 10:48 -0500, Murat Balkan wrote:
> Hi Oleg,
> 
> From the documentation , I understand that this exception is thrown after
> the TCP socket is opened and client sends GET / request. We then start
> waiting the response from the server, but it never responds but drops the
> connection after 50 seconds in my case (which I believe a server parameter)
> 
> My question is, shouldn't socket timeout timer kick in as soon as the
> socket is open? Does it need at least 1 packet to arrive at the client?
> 

Yes, socket (read) timeout setting should apply.

Oleg

> Thanks
> 
> On 8 November 2016 at 07:35, Oleg Kalnichevski <ol...@apache.org> wrote:
> 
> > On November 7, 2016 9:16:19 PM GMT+01:00, Murat Balkan <mrbal...@gmail.com>
> > wrote:
> > >Hi,
> > >
> > >Which timer does the NoHttpResponseException use? My socket timeout is
> > >set
> > >around 30 and connection timeout is set to 20. However, I see that in
> > >some
> > >cases, this exception is thrown after more than 50 seconds.
> > >
> > >Is there a timeout for org.apache.http.NoHttpResponseException?
> > >
> > >Thanks
> > >Murat
> >
> > There is no timer. This exception is thrown if the server closes the
> > connection on its end without any response what so ever
> >
> > Oleg
> > --
> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
> >
> 
> 
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: NoHttpResponseException Timer?

[Oleg Kalnichevski]

On November 7, 2016 9:16:19 PM GMT+01:00, Murat Balkan  
wrote:
>Hi,
>
>Which timer does the NoHttpResponseException use? My socket timeout is
>set
>around 30 and connection timeout is set to 20. However, I see that in
>some
>cases, this exception is thrown after more than 50 seconds.
>
>Is there a timeout for org.apache.http.NoHttpResponseException?
>
>Thanks
>Murat

There is no timer. This exception is thrown if the server closes the connection 
on its end without any response what so ever

Oleg 
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: java.util.concurrent.TimeoutException with the async client

[Oleg Kalnichevski]

On Thu, 2016-10-27 at 15:13 +0200, Joan Balagueró wrote:
> Hi Oleg,
> 
> Sorry yes, reviewing emails I already asked about TimeoutException. I've been 
> reading again the httpcore tutorial because think I had a misunderstanding 
> about how the async client works.
> 
> In the previous case we had about 1000 requests per second, and the average 
> response time was about 20ms. So Stefan proposed a pool of 50 connections 
> with a pool timeout of 10-20ms.
> 
> In the current case we're having not more than 100 requests per second, but 
> the average response time is 1 second.
> 
> When a connection is established, if no packets are being received, the 
> IOReactor dispatch thread is released to do another task, but the http 
> connection continues established. So in the current case, where I have less 
> concurrency but much longer response times I should set up a higher max 
> connections. Right?
> 
> If so, just two questions:
> 
> 1. Why, exactly in the same load conditions, a pool with 600 connections is 
> more than enough for the blocking client and not enough for the async one?
> 

I cannot answer this question unless I can analyze and re-run the
benchmark used to load test both clients.


> 2. Regarding the  'ConnectionRequestTimeout', you told me that the reason was 
> probably due to lock contention. Is this lock contention provoked by this 
> timeout of only 1ms? Do you always recommend a higher value as Stefan said 
> previously?
> 

What is the point of having such a low request timeout? What is it
exactly you intent to achieve by making requests fail if the pool
manager fails to lease a connection after 1 ms? 

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: java.util.concurrent.TimeoutException with the async client

[Oleg Kalnichevski]

On Thu, 2016-10-27 at 12:23 +0200, Joan Balagueró wrote:
> Hello,
> 
> After working in production with the async client (instead of the blocking
> one) everything seems ok, except that now I can see this error
> intermitently:
> 
> java.util.concurrent.TimeoutException
> at
> org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(AbstractN
> IOConnPool.java:364)
> at
> org.apache.http.nio.pool.AbstractNIOConnPool.processNextPendingRequest(Abstr
> actNIOConnPool.java:344)
> at
> org.apache.http.nio.pool.AbstractNIOConnPool.release(AbstractNIOConnPool.jav
> a:318)
> at
> org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.releaseCon
> nection(PoolingNHttpClientConnectionManager.java:303)
> at
> org.apache.http.impl.nio.client.AbstractClientExchangeHandler.releaseConnect
> ion(AbstractClientExchangeHandler.java:239)
> at
> org.apache.http.impl.nio.client.MainClientExec.responseCompleted(MainClientE
> xec.java:387)
> at
> org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCom
> pleted(DefaultClientExchangeHandlerImpl.java:168)
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAs
> yncRequestExecutor.java:436)
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRe
> questExecutor.java:326)
> at
> org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNH
> ttpClientConnection.java:265)
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODi
> spatch.java:81)
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODi
> spatch.java:39)
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODis
> patch.java:114)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:1
> 62)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIORe
> actor.java:337)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOR
> eactor.java:315)
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor
> .java:276)
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:10
> 4)
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(Abs
> tractMultiworkerIOReactor.java:588)
> at java.lang.Thread.run(Thread.java:745)
> 
> It seems that the pool is exhausted at this point.
> 
> The setup of both pools is similar. MaxConnections to 600,
> ConnectionRequestTimeout' is set to 1 (because I don't want to wait for a
> pooling connection to be free), an IdleConnectionsHandler, etc.
> 

This looks wrong to me. I think I already explained that.

> I'm not sure if some connections are not being closed. With the blocking
> client one manually released the connection to the pool, but with the async
> one I'm not closing connections explicitly (so I suppose this is done
> automatically by the async client).
> 
> The point is this error was not present with the blocking client. If I'm not
> wrong, with the async client the possibility to exhaust the pool should be
> more difficult because every time the server is not sending tcp packets the
> thread is released and used for other connections. 
> 
> Well, whatever the reason I'm going to enable http traces. Since my client
> only permitts updates once a week, I want to be sure that I'm enabling the
> http traces correctly (otherwise I will have to wait one more week).
> 
> With the blocking client I enabled traces putting these lines in the
> log4j.xml file (and it worked). Is the same procedure with the async client?
> 
> 
> 
>  
> 
> 
> 
> 
> 

Logging configuration should be the same as for blocking HC.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: next request after a 204 response fails

[Oleg Kalnichevski]

On Thu, 2016-10-27 at 11:55 +0200, Marc Sluiter wrote:
> On 26 October 2016 at 22:28:28, Oleg Kalnichevski (ol...@apache.org) wrote:
> On Wed, 2016-10-26 at 13:01 +0200, Marc Sluiter wrote:  
> > Hi all,  
> >  
> > during testing a Restlet based web app with JMeter I came across a strange 
> > issue, which I stripped down to an issue with the Apache HttpClient:  
> >  
> > when a request gets a 204 (success no content) response, the NEXT request 
> > fails with a "org.apache.http.NoHttpResponseException: The target server 
> > failed to respond” exception (see stacktrace below), but only if the 
> > request follows immediately (like 200ms) after the 1st request. With some 
> > time (e.g. 2s) between the requests the 2nd request succeeds.  
> >  
> > The only interesting fact I found so far is, that Restlet adds a 
> > “Content-Length: 0” header to the 204 response. There seems to be some 
> > disussion if that is allowed or not. While I agree that the header does 
> > make much sense for a 204, it seems not to be strictly forbidden.  
> >  
> > Find example code at https://github.com/slintes/httpclienttest with 3 
> > components:  
> >  
> > - the Restlet server with 2 endpoints, “test” which returns a 200 with 
> > response body, and “empty” which returns a 204 without body (but 
> > automatically added Content-Length header)  
> > - an Apache HttpClient based test client, which request “test”, “empty”, 
> > “test”, “test”: the 3rd request fails, unless you increase the sleep in 
> > line 35. The 4th request is always successful.  
> > - for comparison a Vertx HttpClient based test client, which does the same 
> > and always succeeds.  
> >  
> > Is this a bug? And if not, why not?  
> >  
> > Complete stacktrace of the failing request:  
> >  
> > org.apache.http.NoHttpResponseException: The target server failed to 
> > respond  
> > at 
> > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
> >   
> > at 
> > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
> >   
> > at 
> > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
> >   
> > at 
> > org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
> >   
> > at 
> > org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
> >   
> > at 
> > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
> >   
> > at 
> > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
> >   
> > at 
> > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
> >   
> > at 
> > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)  
> > at 
> > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)  
> > at 
> > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
> >   
> > at 
> > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
> >   
> > at 
> > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:220)
> >   
> > at 
> > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
> >   
> > at 
> > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
> >   
> >  
> > Kind Regards,  
> >  
> > Marc Sluiter  
> 
> Marc  
> 
> As far as I can tell HttpClient does everything correctly. It treats 204  
> as a message without a response body and correctly keeps the connection  
> alive. However on the subsequent request the server drops the  
> connection, which looks like a server side problem to me.  
> 
> See the wire log.  
> 
> Just because VertX sweeps the problem under the carpet and quietly  
> retries the request does not meant the problem does not exit.  
> 
> Oleg  
> 
> ---  
> [DEBUG] RequestAddCookies - CookieSpec selected: default  
> [DEBUG] RequestAuthCache - Auth cache not set in the context  
> [DEBUG] PoolingHttpClientConnectionManager - Connection request: [route: 
> {}->http://localhost:9090][total kept alive: 0; route allocated: 0 of 2; 
> total allocated: 0 of 20]  
> [DEBUG] PoolingHttpClientConnectionManager - Connection leased: [id: 
> 0][route: {}->http://localhost:9090][total kept alive: 0; rou

Re: next request after a 204 response fails

[Oleg Kalnichevski]

On Wed, 2016-10-26 at 13:01 +0200, Marc Sluiter wrote:
> Hi all,
> 
> during testing a Restlet based web app with JMeter I came across a strange 
> issue, which I stripped down to an issue with the Apache HttpClient:
> 
> when a request gets a 204 (success no content) response, the NEXT request 
> fails with a "org.apache.http.NoHttpResponseException: The target server 
> failed to respond” exception (see stacktrace below), but only if the request 
> follows immediately (like 200ms) after the 1st request. With some time (e.g. 
> 2s) between the requests the 2nd request succeeds.
> 
> The only interesting fact I found so far is, that Restlet adds a 
> “Content-Length: 0” header to the 204 response. There seems to be some 
> disussion if that is allowed or not. While I agree that the header does make 
> much sense for a 204, it seems not to be strictly forbidden.
> 
> Find example code at https://github.com/slintes/httpclienttest with 3 
> components:
> 
> - the Restlet server with 2 endpoints, “test” which returns a 200 with 
> response body, and “empty” which returns a 204 without body (but 
> automatically added Content-Length header)
> - an Apache HttpClient based test client, which request “test”, “empty”, 
> “test”, “test”: the 3rd request fails, unless you increase the sleep in line 
> 35. The 4th request is always successful.
> - for comparison a Vertx HttpClient based test client, which does the same 
> and always succeeds.
> 
> Is this a bug? And if not, why not?
> 
> Complete stacktrace of the failing request:
> 
> org.apache.http.NoHttpResponseException: The target server failed to respond
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
>   at 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
>   at 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
>   at 
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
>   at 
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
>   at 
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
>   at 
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
>   at 
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
>   at 
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
>   at 
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
>   at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
>   at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:220)
>   at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
>   at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
> 
> Kind Regards,
> 
> Marc Sluiter

Marc

As far as I can tell HttpClient does everything correctly. It treats 204
as a message without a response body and correctly keeps the connection
alive. However on the subsequent request the server drops the
connection, which looks like a server side problem to me.

See the wire log. 

Just because VertX sweeps the problem under the carpet and quietly
retries the request does not meant the problem does not exit.  

Oleg

---
[DEBUG] RequestAddCookies - CookieSpec selected: default
[DEBUG] RequestAuthCache - Auth cache not set in the context
[DEBUG] PoolingHttpClientConnectionManager - Connection request: [route: 
{}->http://localhost:9090][total kept alive: 0; route allocated: 0 of 2; total 
allocated: 0 of 20]
[DEBUG] PoolingHttpClientConnectionManager - Connection leased: [id: 0][route: 
{}->http://localhost:9090][total kept alive: 0; route allocated: 1 of 2; total 
allocated: 1 of 20]
[DEBUG] MainClientExec - Opening connection {}->http://localhost:9090
[DEBUG] DefaultHttpClientConnectionOperator - Connecting to 
localhost/127.0.0.1:9090
[DEBUG] DefaultHttpClientConnectionOperator - Connection established 
127.0.0.1:42579<->127.0.0.1:9090
[DEBUG] MainClientExec - Executing request GET /test HTTP/1.1
[DEBUG] MainClientExec - Target auth state: UNCHALLENGED
[DEBUG] MainClientExec - Proxy auth state: UNCHALLENGED
[DEBUG] headers - http-outgoing-0 >> GET /test HTTP/1.1
[DEBUG] headers - http-outgoing-0 >> Host: localhost:9090
[DEBUG] headers - http-outgoing-0 >> Connection: Keep-Alive
[DEBUG] headers - http-outgoing-0 >> User-Agent: Apache-HttpClient/4.5.2 
(Java/1.8.0_60)
[DEBUG] headers - http-outgoing-0 >> Accept-Encoding: gzip,deflate
[DEBUG] wire - http-outgoing-0 >> "GET /test HTTP/1.1[\r][\n]"
[DEBUG] wire - http-outgoing-0 >> "Host: localhost:9090[\r][\n]"
[DEBUG] wire - 

Re: Including Content-Length request header

[Oleg Kalnichevski]

On Sat, 2016-10-22 at 13:04 -0400, davidjesse...@aol.com wrote:
> I understand that HttpClient automatically includes the Content-Length 
> request header when passing a StringEntity. But I would like to set the 
> position of the header. But when I include the request header myself I get 
> the following exception 
> 
> 
> "org.apache.http.ProtocolException: Content-Length header already present" 
> 
> 
> 
> 
> Is there anyway to bypass the auto inclusion of the Content-Length header?
> 

If are really convinced this is a good idea you will have to remove
RequestContent interceptor from the protocol processing chain. 

http://hc.apache.org/httpcomponents-client-4.5.x/httpclient/xref/org/apache/http/impl/client/HttpClientBuilder.html#1079

The only way to do so it set a custom HttpProcessor when building
HttpClient instance with HttpClientBuilder. This can be done but one
needs to be careful when assembling a custom HttpProcessor. There are
several protocol interceptors whose absence in the protocol processing
chain is very likely to cause unexpected behavior. 

Proceed at your own risk.

Oleg  



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Throwing socketTimeoutException within HttpAsyncResponseConsumer consumeContent method

[Oleg Kalnichevski]

On Thu, 2016-10-20 at 12:11 +0200, Joan Balagueró wrote:
> Hello,
> 
> In my async client pool, I’ve currently  set up a select interval of 1000ms.
> But I need to control response timeouts of 50ms, 100ms, etc.
> 
> So I’have tried to implement something like this in the 'consumeContent'
> method of my 'HttpAsyncResponseConsumer':
> 
> public void consumeContent(final ContentDecoder decoder, final IOControl
> ioctrl) throws IOException 
>  {
>   int numBytesRead;
> 
>   if (this.bbuf == null) 
>   {
>this.bbuf  = ByteBuffer.allocate(32768);
>this.vbaos = new VentusByteArrayOutputStream();
>   }
> 
>   // If the difference between now and the starting point of the request
> (when I sent it to the server) is greater than 8ms, throw a
> SocketTimeoutException.
>   long diff = System.currentTimeMillis() - this.startTime;
>   if (diff >= 8) throw new java.net.SocketTimeoutException("Socket timeout
> :: consumeContent :: [ " + diff + "ms > " + "8ms ]");
> 
>   while ( (numBytesRead = decoder.read(this.bbuf)) > 0 ) 
>   {
>this.vbaos.write(this.bbuf.array(), 0, numBytesRead);
>this.bbuf.clear();
>   }
>  }
> 
> And it seems to work. The exception is thrown, the consumeContent is no
> longer called, the 'responseCompleted' method is not called, and the async
> client calls the 'failed' method that logs the exception correctly.
> 
> My question is: can I be sure that the response reading was stopped, the
> connection was aborted (even without completing the response reading)

Yes, you can.

>  and
> the thread was released?
> 

I am not sure I understand what thread you are referring to.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Controlling releaseConnection

[Oleg Kalnichevski]

On Wed, 2016-10-19 at 10:14 -0400, Pellerin, Clement wrote:
> I agree with you this feature is strange and need not be added to the library.
> As I said before, I need to preserve this feature for backwards compatibility.
> It has something to do with delaying the emission of an MDN in an AS2 gateway
> so it represents the status of the receiver instead of the man in the middle.
> Again, please don't ask me to change the architecture of the solution.
> We don't own the application and we can't change it. Nevertheless it must
> continue to work after we port to HttpClient 4.5.2
> 
> As for the work-around, I have something working but it duplicates too much
> code for my taste. I think this is a nice opportunity for HttpClient
> to learn what it is like to try to subclass its engine.
> 
> I am subclassing HttpClientBuilder to override createMainExec()
> to return my own implementation of the ClientExecChain.
> This is nice.
> 
> Unfortunately, our subclass of ClientExecChain must be in the
> org.apache.http.impl.execchain package because support classes
> in that package are all package private. This is annoying.
> 
> I modified the problematic code in ClientExecChain.execute()
> but that was not enough. I need to trap the EofSensorWatcher
> of the response HttpEntity. That requires duplicating
> HttpResponseProxy and ResponseEntityProxy.
> Again this must be done in the apache package.
> 
> The release should go through the ConnectionHolder since that's
> where the "AtomicBoolean released" field is stored.
> I thought of providing my own implementation of ConnectionHolder
> with an empty releaseConnection() method, but that does not work
> when ResponseEntityProxy calls cleanup(), this becomes
> releaseConnection(false) which closes the connection completely.
> 

Those classes are package private so we would not keep them API
compatible between minor releases.

I intend to re-evaluate the execution pipeline code in 5.0 and attempt
to provide a public API based on the chain of responsibility pattern. 

> I would appreciate if you have better suggestions for the work-around.
> It would be nice to make subclassing easier too.
> 
> By the way, should ResponseEntityProxy.enchance()
> be called enhance() instead?
> Compare this to RequestEntityProxy.enhance()
> 

Yes, it should be.

Please fork HttpClient on Github and raise a PR with changes you are
proposing. We can add methods to MainClientExec as long as they remain
package private.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Controlling releaseConnection

[Oleg Kalnichevski]

On Tue, 2016-10-18 at 10:12 -0400, Pellerin, Clement wrote:
> I agree that code is correct. I never said there was a bug.
> I am asking how to override the behavior.
> We are porting our product from HttpClient 4.1 to 4.5.2
> and we need to preserve that feature because it is used by our customers.

Clement

I'll be happy to suggest a work-around but I am still unable to
understand what it is exactly you are trying to achieve by keeping a
connection for messages without a message body, which is quite uncommon
(status 204, status 304, response to HEAD, and that is it). 

What is exactly the point of doing so in th first place?

Oleg 


> It would be a shame to duplicate all of MainClientExec because
> of a lack of modularity in that class.
> 
> We tried to subclass the PoolingHttpClientConnectionManager
> but that does not work because the connection state is actually
> stored in the ConnectionHolder created directly in MainClientExec.execute().
> final ConnectionHolder connHolder = new ConnectionHolder(this.log, 
> this.connManager, managedConn);
> 
> Basically, I'm asking if the maintainers could consider a method like
>   protected boolean isEarlyRelease(HttpEntity entity) {
>   return entity == null || !entity.isStreaming();
>   }
> 
> Or a release strategy interface if this is a common requirement.
> I am also interested in any other work-around.
> 
> -Original Message-
> From: Shawn Heisey [mailto:apa...@elyograg.org] 
> Sent: Monday, October 17, 2016 9:24 PM
> To: HttpClient User Discussion
> Subject: Re: Controlling releaseConnection
> 
> On 10/17/2016 3:22 PM, Pellerin, Clement wrote:
> > Our customer needs to delay the release of the connection until the 
> > response is fully processed.
> > They want to turn off the early automatic release of the connection and do 
> > it manually later.
> >
> > This is the problematic code in MainClientExec
> > // check for entity, release connection if possible
> > final HttpEntity entity = response.getEntity();
> > if (entity == null || !entity.isStreaming()) {
> > // connection not needed and (assumed to be) in re-usable 
> > state
> > connHolder.releaseConnection();
> > return new HttpResponseProxy(response, null);
> > } else {
> > return new HttpResponseProxy(response, connHolder);
> > }
> 
> Mostly an end-user here, with no status to speak of in this project.  I do 
> have status on another Apache project that utilizes HttpClient, but I don't 
> know much about that part of the code.  I have written some HttpClient code 
> for a completely unrelated project of my own, but that code is VERY simple.
> 
> When I read the code above, what I see is this: It only releases the 
> connection if the entity is nonexistent (null) or the entity is NOT a type 
> that uses streaming.
> 
> I will fully admit that my experience with HttpClient is limited, but I think 
> the chance is very small that the HttpComponents committers have made a 
> mistake here.  I think this particular code has probably been discussed and 
> examined, then ultimately validated as correct.  Here's why I think they 
> didn't make a mistake:
> 
> If the entity object is null, then the response probably doesn't HAVE an 
> entity (response body), so it will be entirely self-contained, consisting of 
> headers only, and the connection doesn't have anything further to send.  If 
> the entity exists but doesn't utilize streaming, then I think it's likely 
> that the entity was received in its entirety and has been incorporated into 
> the response object already, and once again, the connection isn't needed.  If 
> my limited understanding of non-streaming entities is correct, they have the 
> potential to be very dangerous from a memory consumption perspective, and my 
> own usage of HttpClient (where I did not set anything related to the entity 
> type) suggests that streaming entities are used by default.
> 
> Restating in another way:  In the first situation that results in a released 
> connection, there's nothing to consume, you just need the response object 
> that you already have.  In the second situation, the entity you will consume 
> is probably already available within the response object and doesn't need the 
> connection.  The comment on the release call in the code quoted above implies 
> that this is how things work.
> 
> In these situations, why do you need the connection to stick around?  I think 
> it can't do anything else that's useful for that request.  I would imagine 
> that if the connection utilizes keepalive/pipelining, that it will typically 
> remain open after release and can be utilized again for a different request.
> 
> Someone with more direct knowledge of HttpClient's internal implementation 
> will need to confirm whether or not I'm correct in what I've written.  My 
> understanding could be wrong.
> 
> 

Re: Disabling pooling of PoolingConnectionManager

[Oleg Kalnichevski]

On Mon, 2016-10-17 at 22:06 -0600, Bhowmik, Bindul wrote:
> Murat,
> 
> On Mon, Oct 17, 2016 at 8:11 PM, Murat Balkan  wrote:
> > I see. I think that also means that I cannot share the ClosableHttpClient
> > instance among multiple threads as each client can refer to one connection
> > manager instance.
> >
> > Can connectionreusestrategy be used so that the pooling connection manager
> > will always return a new connection regardless of the route provided?
> 
> I did not think about that, guess you could use the NoConnectionReuseStrategy
> 

That would be one option. Another option is to manually evict persistent
connections from the pool after each transaction. Another option is to
set connection TTL (total time to live) to some very low value. Another
option is to simply use 'connection: close' request header.

Oleg  

> - Bindul
> 
> >
> > Regards.
> > Murat
> >
> > On Mon, Oct 17, 2016 at 5:05 PM, Bhowmik, Bindul 
> > wrote:
> >
> >> Murat,
> >>
> >> On Mon, Oct 17, 2016 at 12:58 PM, Murat Balkan  wrote:
> >> > Hi Bindul,
> >> > Thanks for the answer.
> >> > I was thinking that using a shared connection manager will increase the
> >> > performance. What will be the implications of reusing the same
> >> > BasicHttpClientConnectionManager instance?
> >>
> >> If you see the documentation for the BasicHttpClientConnectionManager
> >> [1], you will see that it only maintains one active connection. If you
> >> share the the instance, your requests will be waiting for the
> >> connection to be available and that will be your bottleneck.
> >>
> >> I would also recommend reading the connection management section of
> >> the Http Client documentation [2]
> >>
> >> - Bindul
> >>
> >> [1] http://hc.apache.org/httpcomponents-client-ga/
> >> httpclient/apidocs/org/apache/http/impl/conn/
> >> BasicHttpClientConnectionManager.html
> >> [2] http://hc.apache.org/httpcomponents-client-4.5.x/
> >> tutorial/html/connmgmt.html
> >>
> >> > Regards,
> >> > Murat
> >> >
> >> > On Mon, Oct 17, 2016 at 2:31 PM, Bhowmik, Bindul <
> >> bindulbhow...@gmail.com>
> >> > wrote:
> >> >
> >> >> Murat,
> >> >>
> >> >> On Mon, Oct 17, 2016 at 11:12 AM, Murat Balkan 
> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > We are using PoolingHttpClientConnectionManager to open up
> >> connections
> >> >> to
> >> >> > multiple URL's in different threads (via different HttpGet objects).
> >> >> >
> >> >> > The only reason we are using the PoolingHttpClientConnectionManager
> >> is
> >> >> its'
> >> >> > performance in multi-thread environments (as suggested by the
> >> >> > documentation).
> >> >> >
> >> >> > However, we are not interested in the actual "pooling" functionality.
> >> >> > That's to say, we want to open up a brand new connection even if the
> >> >> route
> >> >> > is the same.
> >> >>
> >> >> The performance enhancements you achieve from
> >> >> PoolingHttpClientConnectionManager are due to its connection pooling
> >> >> feature, that saves you to cost of establishing the connection when
> >> >> another request goes to the same route.
> >> >>
> >> >> >
> >> >> > How can we achieve this? We tried to set maxPerroute to 1 but it
> >> seems it
> >> >> > is not the correct way.
> >> >>
> >> >> I have not tested, but setting maxPerRoute to 1 would degrade
> >> >> performance for you as you will have a number of Http clients waiting
> >> >> for the single connection.
> >> >>
> >> >> If you do not want to use pooled connections, you can use
> >> >> BasicHttpClientConnectionManager and not share it.
> >> >>
> >> >> >
> >> >> > Regards,
> >> >> > Murat
> >> >>
> >> >> -
> >> >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> >> >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >> >>
> >> >>
> >> >
> >> >
> >> > --
> >> > Murat Balkan
> >>
> >> -
> >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >>
> >>
> >
> >
> > --
> > Murat Balkan
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: AW: Controlling releaseConnection

[Oleg Kalnichevski]

On Tue, 2016-10-18 at 01:46 +0200, e...@zusammenkunft.net wrote:
> Hello,
> 
> Can younspecify why you need to delay it? Do wou want to make some kind of 
> rate limit with this or optimize pipelining?
> 
> Gruss
> Bernd

I am not quite sure I understand the problem. HttpClient releases
connection back to the pool only when done reading content from it. Why
would one want to hold onto the connection once the response message has
been fully consumed? Am I missing something?

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Memory leak in CloseableHttpAsyncClient?

[Oleg Kalnichevski]

On Sun, 2016-10-16 at 21:54 +, Rob Griffin wrote:
> We are using HTTPAsynchClient to send data to our web site from a Java 
> client. 
> 
> We call CloseableHttpAsyncClient.execute() to execute HTTP PUTs at the rate 
> of several hundred per minute. Sometimes our web site slows down and does not 
> respond quickly enough and when this occurs the requests back up. We have 
> code that detects this and cancels the Future returned from the execute 
> method when the request has waited too long. If this happens too often the 
> application crashes with an out of memory error. 
> 
> Analysis of a dump showed that there were more 108,000 instances of 
> org.apache.http.nio.pool.LeaseRequest along with a similar number of 
> instances of other HTTP Client classes. Inspecting one of these objects 
> showed that its future variable is not cancelled but that by tracing though 
> the callback variables there is a cancelled Future further up the chain. That 
> cancelled Future object is one returned by execute because its callback is 
> one of our classes. To me it appears that the library is unaware that cancel 
> has been called on the Future returned by execute() and so keeps a reference 
> to it.
> 
> See 
> [url=https://postimg.org/image/j6zfdrquf/][img]https://s15.postimg.org/qa7atdwa3/Screen_Shot010.jpg[/img][/url][url=https://postimage.org/]image
>  url[/url] 
> 
> Regards,
> 
> Rob Griffin

Hi Rob

Currently cancellation of the response future does not immediately
results in cancellation pending connection lease request or termination
of the request execution.

Feel free to raise an improvement request in Jira.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Cookies in the response body

[Oleg Kalnichevski]

On Sun, 2016-10-09 at 17:19 -0400, Arya F wrote:
> I am working with a rest server that the cookies are returned in the
> response body and not the response headers, which makes it impossible for
> HttpClient to automatically deal with the cookies.
> 
> I have extracted the cookies into POJO classes and they look like this
> 
>   "T" :
> "z=a3S.XBahmAYBSVvVHamKFS2NDA3MwY1TjUwNDROMDFPMDROMU=QAE=DAAB0fOdrGnTGN=EAAHQk2lLIRRoBAqtAEvrDGHQ--~E=EAAcDL4YtVi4VAabIF_d2mUEA--~F=c2wBTXpjd05BRXlPVEkzTXpNNU56WTROek01Tmprek1nLS0BYQFRQUUBZwFWR0JEUDdEUkoyVzRPT0w3UVJWSjRBNFpTUQFzY2lkAWtIM3VJdzNTdUpIYVZsWlF5ZGc4QjhjcXBfRS0BYWMBQUZiOVZpYU5sWldLZ1pvLQF0aXABanBfVnhDAXNjAXltb2JpbGVtYWlsAWZzAXVjZ2JwV0ZYLlMzYQF6egFhM1MuWEJnZEM-;
> path=/; domain=.MYDOAMIN.com; HttpOnly",
> 
> 
> My question is
> 
> Do I need to break the value of the cookie as well manually? like
> separating the value, domain and path before giving it to HttpClient? Or
> does HttpClient have a method which I can pass the full value below and
> HttpClient would automatically extract the domain, path and value?
> 
> 
> z=a3S.XBahmAYBSVvVHamKFS2NDA3MwY1TjUwNDROMDFPMDROMU=QAE=DAAB0fOdrGnTGN=EAAHQk2lLIRRoBAqtAEvrDGHQ--~E=EAAcDL4YtVi4VAabIF_d2mUEA--~F=c2wBTXpjd05BRXlPVEkzTXpNNU56WTROek01Tmprek1nLS0BYQFRQUUBZwFWR0JEUDdEUkoyVzRPT0w3UVJWSjRBNFpTUQFzY2lkAWtIM3VJdzNTdUpIYVZsWlF5ZGc4QjhjcXBfRS0BYWMBQUZiOVZpYU5sWldLZ1pvLQF0aXABanBfVnhDAXNjAXltb2JpbGVtYWlsAWZzAXVjZ2JwV0ZYLlMzYQF6egFhM1MuWEJnZEM-;
> path=/; domain=.MYDOAMIN.com; HttpOnly

http://hc.apache.org/httpcomponents-client-4.5.x/httpclient/apidocs/org/apache/http/impl/cookie/RFC2965Spec.html#parse(org.apache.http.Header,%20org.apache.http.cookie.CookieOrigin)

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool

[Oleg Kalnichevski]

On Mon, 2016-10-10 at 19:16 +0200, Claudio Degioanni wrote:
> Hi All,
> 
> I'm using the library Axis 2 version  1.7.3 with HttpClient 4.4.1, and
> after 200 request the library throw a TimeoutException from connection pool.
> 
> From stacktrace I found the exeption is throw by the class
> PoolingClientConnectionManager,
> as reported below
> 
> Caused by: org.apache.axis2.AxisFault: Timeout waiting for connection from
> pool
> at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
> ~[axis2-kernel-1.7.3.jar:1.7.3]
> at org.apache.axis2.transport.http.impl.httpclient4.
> HTTPSenderImpl.sendViaPost(HTTPSenderImpl.java:242)
> ~[axis2-transport-http-1.7.3.jar:1.7.3]
> at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:121)
> ~[axis2-transport-http-1.7.3.jar:1.7.3]
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.
> writeMessageWithCommons(CommonsHTTPTransportSender.java:403)
> ~[axis2-transport-http-1.7.3.jar:1.7.3]
> at org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(
> CommonsHTTPTransportSender.java:234) ~[axis2-transport-http-1.7.3.jar:1.7.3]
> at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:431)
> ~[axis2-kernel-1.7.3.jar:1.7.3]
> at 
> org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:399)
> ~[axis2-kernel-1.7.3.jar:1.7.3]
> at 
> org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
> ~[axis2-kernel-1.7.3.jar:1.7.3]
> at org.apache.axis2.client.OperationClient.execute(OperationClient.java:150)
> ~[axis2-kernel-1.7.3.jar:1.7.3]
> 
> Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout
> waiting for connection from pool
> at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(
> PoolingClientConnectionManager.java:226) ~[httpclient-4.4.1.jar:4.4.1]
> at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(
> PoolingClientConnectionManager.java:195) ~[httpclient-4.4.1.jar:4.4.1]
> at org.apache.http.impl.client.DefaultRequestDirector.execute(
> DefaultRequestDirector.java:423) ~[httpclient-4.4.1.jar:4.4.1]
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> ~[httpclient-4.4.1.jar:4.4.1]
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
> ~[httpclient-4.4.1.jar:4.4.1]
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> ~[httpclient-4.4.1.jar:4.4.1]
> at org.apache.axis2.transport.http.impl.httpclient4.
> HTTPSenderImpl.executeMethod(HTTPSenderImpl.java:873)
> ~[axis2-transport-http-1.7.3.jar:1.7.3]
> at org.apache.axis2.transport.http.impl.httpclient4.
> HTTPSenderImpl.sendViaPost(HTTPSenderImpl.java:238)
> ~[axis2-transport-http-1.7.3.jar:1.7.3]
> ... 15 common frames omitted
> 
> 
> Search into code of library i found the configuration of connection pool:
> 
>  connManager = new PoolingClientConnectionManager(schemeRegistry);
>  ((PoolingClientConnectionManager)connManager).setMaxTotal(200);
>  ((PoolingClientConnectionManager)connManager).setDefaultMaxPerRoute(200);
>   configContext.setProperty(HTTPConstants.MULTITHREAD_HTTP_CONNECTION_MANAGER,
> connManager);
> 
> The connection pool it's configured to have 200 connection for route, I
> have made some test and my application fail after until 200 web service
> request.
> 
> From log i see this line, to confim my idea:
> 
> 2016-09-19 16:01:06| DEBUG   | Connection leased: [id: 199][route:
> {s}-http://:443][total kept alive: 0; route allocated: 200 of 200;
> total allocated: 200 of 200]
> 2016-09-19 16:01:09| DEBUG   | Connection request: [route:
> {s}->http://:443][total kept alive: 0; route allocated: 200 of
> 200; total allocated: 200 of 200]
> 
> I think the connection pool use all available connections without release
> it, after that throwing the exception.
> 
> I tried to write on Axis2 mail list without luck.
> 
> Any suggestions/test to solve the my problem ?
> 
> regs

Fix code that leaks connections.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Get response header from 204 response code

[Oleg Kalnichevski]

On Sun, 2016-10-09 at 00:59 -0400, Arya F wrote:
> I have also asked about this on Stackoverflow and have not gotten any
> solutions. Here is the link to the question but I will also explain it here
> 
> http://stackoverflow.com/questions/39939509/get-location-header-from-204-
> response
> 
> 
> I am working with a request which responses with a 204 status code which
> means no response. But the response does contain headers. I am trying to
> get the value of the "Location" response header. I have tried the
> context.REDIRECT_LOCATIONS and it is null which always worked when the
> response code was not 204.
> 
> How can I retrieve the value of the Location response header?

Header location = response.getFirstHeader("location");

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: AW: Why are many TCP open connections created on OS (Linux)?

[Oleg Kalnichevski]

On Wed, 2016-10-05 at 12:16 +, Gnerlich, Manuel (IPS PROJECTS GMBH)
wrote:
> Hello Bernd,
> 
> thank you for your reply. 
> 
> I see them as ESTABLISHED and TIME_WAIT in netstat. The connections with 
> ESTABLISHED state are ok and most under 100, but the connections with 
> TIME_WAIT state are always more than 200. I mean on the client. On the server 
> is ok and has little open connections. 

Please take a look at this resource:

http://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions

Oleg

> 
> Gruss ;) Manuel
> 
> Here some important  code, how I set it up:
> 
> public class NetworkClient implements AutoCloseable {
> ...
> public NetworkClient() {
> ConnectingIOReactor ioReactor;
> try {
>   // Create I/O reactor configuration
>   IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
>   .setIoThreadCount(Runtime.getRuntime().availableProcessors())
>   
> .setConnectTimeout(config.getConnectTimeout()).setSoTimeout(config.getSoTimeout())
>   .build();
>   ioReactor = new DefaultConnectingIOReactor(ioReactorConfig);
> 
>   PoolingNHttpClientConnectionManager cm = new 
> PoolingNHttpClientConnectionManager(ioReactor);
>   cm.setMaxTotal(config.getConnectionMaxTotal());
>   cm.setDefaultMaxPerRoute(config.getConnectionDefaultMaxPerRoute());
> 
>   RequestConfig globalConfig = 
> RequestConfig.custom().setCookieSpec(CookieSpecs.IGNORE_COOKIES)
>   .build();
> 
>   HttpProcessor httpprocessor = HttpProcessorBuilder.create().build();
>   httpClient = 
> HttpAsyncClients.custom().setDefaultRequestConfig(globalConfig)
>   .setHttpProcessor(httpprocessor).setConnectionManager(cm).build();
> } catch (IOReactorException e) {
>   LOGGER.error("Could not create ConnectionIOReactor", e);
> }
> 
> initClientTransport();
>   }
> 
>   private void initClientTransport() {
> httpClient.start();
>   }
> 
> 
> 
> public Integer send(String destination, int port, HttpRequest httpRequest, 
> String operationName) {
> Assert.notNull(destination);
> Assert.notNull(httpRequest);
> 
> startTime = startTime == 0 ? System.nanoTime() : startTime;
> 
> HttpHost target = new HttpHost(destination, port);
> 
> NcResponse ncResponse = getNewNcResponse(operationName);
> 
> NcResponseProducer responseProducer = new NcResponseProducer(target, 
> httpRequest, ncResponse,
> requestCounter);
> NcResponseConsumer responseConsumer = new NcResponseConsumer(ncResponse);
> 
> httpClient.execute(responseProducer, responseConsumer, new 
> ClientCallback(responseCounter,
> failCounter, connectionErrorMessageList, 
> getRequestLine(httpRequest)));
> 
> long clientId = requestClientId.getAndIncrement();
> return ((Long) clientId).intValue();
>   }
> ..
> }
> 
> public class NcResponseProducer extends BasicAsyncRequestProducer {
> 
>   private final NcResponse ncResponse;
> 
>   private static final Logger LOGGER = LogManager.getLogger();
> 
>   private final AtomicLong requestCounter;
> 
>   private final Header[] allHeaders;
> 
>   public NcResponseProducer(HttpHost target, HttpRequest request, NcResponse 
> ncResponse,
>   AtomicLong requestCounter) {
> super(target, request);
> this.allHeaders = request.getAllHeaders();
> this.ncResponse = ncResponse;
> this.requestCounter = requestCounter;
>   }
> 
>   @Override
>   public void requestCompleted(HttpContext context) {
> super.requestCompleted(context);
> ncResponse.setStartTimeInNano(System.nanoTime());
> LOGGER.debug("Sent request to client {}. request ID={} headers={} 
> count={}",
> getTarget().toHostString(), ncResponse.getRequestID(), 
> Arrays.asList(allHeaders),
> requestCounter.incrementAndGet());
>   }
> 
> }
> 
> public class NcResponseConsumer extends BasicAsyncResponseConsumer {
> 
>   private final NcResponse ncResponse;
> 
>   public NcResponseConsumer(NcResponse ncResponse) {
> this.ncResponse = ncResponse;
>   }
> 
>   @Override
>   protected void onResponseReceived(HttpResponse response) throws IOException 
> {
> super.onResponseReceived(response);
> ncResponse.setEndTimeInNano(System.nanoTime());
> ncResponse.setReturnCode(response.getStatusLine().getStatusCode());
>   }
> 
> }
> 
> 
> -Ursprüngliche Nachricht-
> Von: Bernd Eckenfels [mailto:e...@zusammenkunft.net] 
> Gesendet: Dienstag, 4. Oktober 2016 21:19
> An: Gnerlich, Manuel (IPS PROJECTS GMBH) 
> Cc: httpclient-users@hc.apache.org
> Betreff: Re: Why are many TCP open connections created on OS (Linux)?
> 
> Am Tue, 4 Oct 2016 14:00:38 +
> schrieb "Gnerlich, Manuel (IPS PROJECTS GMBH)"
> :
> 
> > I am using Apache AsychHttpClient with connection pool to send more 
> > than 1 mio requests and configured maxinum number of connection (200) 
> > and of connection per Host (100).
> > 
> > However there are 

Re: I/O reactor status: STOPPED

[Oleg Kalnichevski]

On Tue, 2016-10-04 at 14:29 +0200, Joan Balagueró wrote:
> Within the HttpAsyncResponseConsumer?
> 

Most likely.

You can use #getAuditLog() method of the I/O reactor to find out exactly
what exception caused it to terminate.

http://hc.apache.org/httpcomponents-core-4.4.x/httpcore-nio/apidocs/org/apache/http/impl/nio/reactor/AbstractMultiworkerIOReactor.html#getAuditLog()

Oleg


> Joan.
> 
> -Mensaje original-----
> De: Oleg Kalnichevski [mailto:o...@ok2consulting.com] 
> Enviado el: martes, 4 de octubre de 2016 13:59
> Para: HttpClient User Discussion; Joan Balagueró
> Asunto: Re: I/O reactor status: STOPPED
> 
> On October 4, 2016 12:52:42 PM GMT+02:00, "Joan Balagueró" 
> <joan.balagu...@grupoventus.com> wrote:
> >Hello,
> >
> >I'm occassionally receiving this error, and the async pool stops
> >working:
> >
> >java.lang.IllegalStateException: Request cannot be executed; I/O 
> >reactor
> >status: STOPPED
> >at org.apache.http.util.Asserts.check(Asserts.java:46)
> >at
> >org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunn
> >ing(C
> >loseableHttpAsyncClientBase.java:90)
> >at
> >org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(Interna
> >lHttp
> >AsyncClient.java:123)
> >at
> >org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(Closea
> >bleHt
> >tpAsyncClient.java:68)
> >
> >
> >I'm using an HttpAsyncResponseConsumer, and I know that one of the 
> >reasons for this error is to throw an exception on the "failed" method. 
> >That's why this method looks like as follows:
> >
> >@Override
> > public void failed(final Exception e)
> > {
> >  try { this.setError(e); }
> >  catch (Throwable t) {}
> > }
> >
> >Even with this, sometimes the IO reactor stops. Is there anything else 
> >I must take into account in order to avoid this error? Any other method 
> >I should control? What else can cause the IO reactor to be stopped?
> >
> >Thanks,
> >
> >Joan.
> 
> Most likely due to a runtime thrown by application code.
> 
> Oleg
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 
> 
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: I/O reactor status: STOPPED

[Oleg Kalnichevski]

On October 4, 2016 12:52:42 PM GMT+02:00, "Joan Balagueró" 
 wrote:
>Hello,
>
>I'm occassionally receiving this error, and the async pool stops
>working:
>
>java.lang.IllegalStateException: Request cannot be executed; I/O
>reactor
>status: STOPPED
>at org.apache.http.util.Asserts.check(Asserts.java:46)
>at
>org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(C
>loseableHttpAsyncClientBase.java:90)
>at
>org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttp
>AsyncClient.java:123)
>at
>org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(CloseableHt
>tpAsyncClient.java:68)
>
>
>I'm using an HttpAsyncResponseConsumer, and I know that one of the
>reasons
>for this error is to throw an exception on the "failed" method. That's
>why
>this method looks like as follows:
>
>@Override
> public void failed(final Exception e)
> {
>  try { this.setError(e); }
>  catch (Throwable t) {}
> }
>
>Even with this, sometimes the IO reactor stops. Is there anything else
>I
>must take into account in order to avoid this error? Any other method I
>should control? What else can cause the IO reactor to be stopped?
>
>Thanks,
>
>Joan.

Most likely due to a runtime thrown by application code.

Oleg
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: TimeoutException in async pool

[Oleg Kalnichevski]

On Mon, 2016-10-03 at 10:45 +0200, Joan Balagueró wrote:
> Thanks Stefan and yes, that's right. And I already told my client to modify 
> these settings. But I still have the doubt about why a Timeout exception in a 
> pool that has no upper limit in the number of pooled connections. 
> 
> 

This is because of the lock contention.

Oleg

> -Mensaje original-
> De: Stefan Magnus Landrø [mailto:stefan.lan...@gmail.com] 
> Enviado el: lunes, 3 de octubre de 2016 9:00
> Para: HttpClient User Discussion
> Asunto: Re: TimeoutException in async pool
> 
> So - first I'd set pool size to 50 since that's all you need on average. Then 
> set acquisition timeout to something that makes sense. 10-20ms?
> 
> Sendt fra min iPhone
> 
> > Den 3. okt. 2016 kl. 07.32 skrev Joan Balagueró 
> > :
> > 
> > Hello,
> > 
> > I’m using the async PoolingNHttpClientConnectionManager (v 4.1.2).
> > 
> > Occasionally I’m getting this error:
> > 
> > java.util.concurrent.TimeoutException
> >at
> > org.apache.http.nio.pool.AbstractNIOConnPool.processPendingRequest(Abs
> > tractN
> > IOConnPool.java:364)
> >at
> > org.apache.http.nio.pool.AbstractNIOConnPool.lease(AbstractNIOConnPool.java:
> > 276)
> >at
> > org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.requ
> > estCon
> > nection(PoolingNHttpClientConnectionManager.java:266)
> >at
> > org.apache.http.impl.nio.client.AbstractClientExchangeHandler.requestC
> > onnect
> > ion(AbstractClientExchangeHandler.java:363)
> >at
> > org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.start
> > (Defau
> > ltClientExchangeHandlerImpl.java:125)
> >at
> > org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(Intern
> > alHttp
> > AsyncClient.java:141)
> >at
> > org.apache.http.impl.nio.client.CloseableHttpAsyncClient.execute(Close
> > ableHt
> > tpAsyncClient.java:68)
> > 
> > 
> > It seems that all connections in pool are busy. But my setup says:
> > 
> > this.phccm = new PoolingNHttpClientConnectionManager(…);
> > this.phccm.setMaxTotal(Integer.MAX_VALUE);
> > this.phccm.setDefaultMaxPerRoute(Integer.MAX_VALUE);
> > 
> > 
> > I don't have more than 1000 requests per second, and the average 
> > response time is less than 20ms.
> > 
> > What could be the reason for this error?
> > 
> > Thanks,
> > Joan.
> > 
> > 
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> > 
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: UrlEncodedFormEntity and parameter value encoding

[Oleg Kalnichevski]

On Sat, 2016-09-24 at 10:50 +0200, Philippe Mouawad wrote:
> Hi Oleg,
> If you look at :
> https://bz.apache.org/bugzilla/show_bug.cgi?id=60120
> You can see that this encoding can break some applications.
> 
> If I read https://en.wikipedia.org/wiki/Percent-encoding#The_
> application.2Fx-www-form-urlencoded_type
> 
> 
> Which references https://tools.ietf.org/html/rfc1630
> I understand @ is safe.
> 

Maybe, but Oracle seems to disagree.

System.out.println(URLEncoder.encode("@", "US-ASCII"));

> %40

Oleg 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Retries with the async client

[Oleg Kalnichevski]

On Wed, 2016-09-21 at 13:51 +0200, Joan Balagueró wrote:
> Hello,
> 
> I’m trying to retry ‘NoHttpResponse’ exceptions with the async client. The
> point is I’m unable to generate this kind of error from my server. So my
> question is: does this kind of exceptions that can be retried reach the
> callback? In my case the  HttpAsyncResponseConsumer.responseReceived (so
> they could be retried in this method),or the exception is thrown in the
> CloseableHttpAsyncClient.execute method? 
> 

Joan

No, exceptions cannot be retried from the callback. You probably will
have to create a dedicated thread that can put those requests back into
the execution queue. I understand this sounds cumbersome. I intent to
make this process easier (or automatic) in HC 5.0.

Oleg 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: UrlEncodedFormEntity and parameter value encoding

[Oleg Kalnichevski]

On Sun, 2016-09-18 at 16:27 +0200, Philippe Mouawad wrote:
> On Sunday, September 18, 2016, Oleg Kalnichevski <ol...@apache.org> wrote:
> 
> > On Sun, 2016-09-18 at 16:12 +0200, Philippe Mouawad wrote:
> > > So is it a bug ? or a regular behaviour ?
> > >
> >
> > We back to where we started. Why do you think it should not be encoded?
> 
> it does not seem to be when using browser.
> It should if it's in url but I am not sure it is reserved when located in
> form parameter.
> I don't fully understand the rfc documentation.
> 

I can be easily wrong here but I am not aware of any RFC that defines
composition of application/x-www-form-urlencoded content. 

As far as I know HttpClient presently emulates the behavior of
java.net.URLEncoder. It also seems safer to me to rather encode too much
and to encode too little.

Oleg  

> Regards
> 
> >
> > Oleg
> >
> > > thanks
> > >
> > > On Sunday, September 18, 2016, Oleg Kalnichevski <ol...@apache.org
> > <javascript:;>> wrote:
> > >
> > > > On Sun, 2016-09-18 at 15:11 +0200, Philippe Mouawad wrote:
> > > > > Hi Oleg,
> > > > > Thanks for rapid answer.
> > > > > Correct me if I am wrong but:
> > > > > - URLENCODER is built from an or on UNRESERVED (which contains '_',
> > '-',
> > > > > '.', '*' and a-z, A-Z, 0-9.
> > > > >
> > > > > URLENCODER is passed as safechars in
> > > > > private static String urlEncode(
> > > > > final String content,
> > > > > final Charset charset,
> > > > > final BitSet safechars,
> > > > > final boolean blankAsPlus)
> > > > >
> > > > >
> > > > > @ is not part of the chars, so you end up in this part of algorithms
> > > > right
> > > > > ?:
> > > > > } else {
> > > > > buf.append("%");
> > > > > final char hex1 =
> > > > > Character.toUpperCase(Character.forDigit((b >> 4) & 0xF, RADIX));
> > > > > final char hex2 =
> > > > > Character.toUpperCase(Character.forDigit(b & 0xF, RADIX));
> > > > > buf.append(hex1);
> > > > > buf.append(hex2);
> > > > > }
> > > > >
> > > > > And thus it gets URL encoded no ?
> > > > >
> > > >
> > > > Yes, I was mistaken. Sorry about that.
> > > >
> > > > Oleg
> > > >
> > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > <javascript:;>
> > > > <javascript:;>
> > > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> > <javascript:;>
> > > > <javascript:;>
> > > >
> > > >
> > >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > <javascript:;>
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> > <javascript:;>
> >
> >
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: UrlEncodedFormEntity and parameter value encoding

[Oleg Kalnichevski]

On Sun, 2016-09-18 at 16:12 +0200, Philippe Mouawad wrote:
> So is it a bug ? or a regular behaviour ?
> 

We back to where we started. Why do you think it should not be encoded?

Oleg

> thanks
> 
> On Sunday, September 18, 2016, Oleg Kalnichevski <ol...@apache.org> wrote:
> 
> > On Sun, 2016-09-18 at 15:11 +0200, Philippe Mouawad wrote:
> > > Hi Oleg,
> > > Thanks for rapid answer.
> > > Correct me if I am wrong but:
> > > - URLENCODER is built from an or on UNRESERVED (which contains '_', '-',
> > > '.', '*' and a-z, A-Z, 0-9.
> > >
> > > URLENCODER is passed as safechars in
> > > private static String urlEncode(
> > > final String content,
> > > final Charset charset,
> > > final BitSet safechars,
> > > final boolean blankAsPlus)
> > >
> > >
> > > @ is not part of the chars, so you end up in this part of algorithms
> > right
> > > ?:
> > > } else {
> > > buf.append("%");
> > > final char hex1 =
> > > Character.toUpperCase(Character.forDigit((b >> 4) & 0xF, RADIX));
> > > final char hex2 =
> > > Character.toUpperCase(Character.forDigit(b & 0xF, RADIX));
> > > buf.append(hex1);
> > > buf.append(hex2);
> > > }
> > >
> > > And thus it gets URL encoded no ?
> > >
> >
> > Yes, I was mistaken. Sorry about that.
> >
> > Oleg
> >
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > <javascript:;>
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> > <javascript:;>
> >
> >
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: UrlEncodedFormEntity and parameter value encoding

[Oleg Kalnichevski]

On Sun, 2016-09-18 at 15:11 +0200, Philippe Mouawad wrote:
> Hi Oleg,
> Thanks for rapid answer.
> Correct me if I am wrong but:
> - URLENCODER is built from an or on UNRESERVED (which contains '_', '-',
> '.', '*' and a-z, A-Z, 0-9.
> 
> URLENCODER is passed as safechars in
> private static String urlEncode(
> final String content,
> final Charset charset,
> final BitSet safechars,
> final boolean blankAsPlus)
> 
> 
> @ is not part of the chars, so you end up in this part of algorithms right
> ?:
> } else {
> buf.append("%");
> final char hex1 =
> Character.toUpperCase(Character.forDigit((b >> 4) & 0xF, RADIX));
> final char hex2 =
> Character.toUpperCase(Character.forDigit(b & 0xF, RADIX));
> buf.append(hex1);
> buf.append(hex2);
> }
> 
> And thus it gets URL encoded no ?
> 

Yes, I was mistaken. Sorry about that.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: UrlEncodedFormEntity and parameter value encoding

[Oleg Kalnichevski]

On Sun, 2016-09-18 at 14:08 +0200, Philippe Mouawad wrote:
> On Sunday, September 18, 2016, Oleg Kalnichevski <ol...@apache.org> wrote:
> 
> > On Sat, 2016-09-17 at 15:55 +0200, Philippe Mouawad wrote:
> > > Hello,
> > > We have a bug report at JMeter :
> > > https://bz.apache.org/bugzilla/show_bug.cgi?id=60120
> > >
> > > Where  a user post a form with a parameter having this value
> > > 'IqGo6EM1JEVZ+MSRJqUSo@qhjVMSFBTs'
> > >
> > > It appears that the '@' character is encoded.
> > >
> > > The form is submitted using application/x-www-form-urlencoded
> > >
> > > As per rfc:
> > > > then reserved characters are escaped as described in [RFC1738]
> > > <https://www.w3.org/TR/html401/references.html#ref-RFC1738>, section
> > 2.2:
> > >
> > >
> > > > The characters ";"  "/", "?", ":", "@", "=" and "&" are the characters
> > > which may be reserved for special meaning within a scheme. No other
> > > characters may be reserved within a scheme.
> > >
> > > So @ is reserved and as such HttpClient encodes it.
> > >
> > > But it is not clear for me if @ is reserved when place in URI or also
> > > reserved when being part of a form parameter value.
> > >
> > > In JMeter code, we use UrlEncodedFormEntity and I check that when
> > parameter
> > > is passed , its value has still @ unencoded.
> > >
> >
> > Why do you think it should be encoded?
> 
> 
> I don't think it should but it is by HtTpClient class.
> Maybe my last sentence was confusing, I recap, debugging the call, I see @
> passed unencoded to httpclient method which encodes it.
> 
> 

But it should not

https://github.com/apache/httpclient/blob/4.5.x/httpclient/src/main/java/org/apache/http/client/utils/URLEncodedUtils.java#L654

@ is not in the set of characters that need encoding

https://github.com/apache/httpclient/blob/4.5.x/httpclient/src/main/java/org/apache/http/client/utils/URLEncodedUtils.java#L454https://github.com/apache/httpclient/blob/4.5.x/httpclient/src/main/java/org/apache/http/client/utils/URLEncodedUtils.java#L457

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: UrlEncodedFormEntity and parameter value encoding

[Oleg Kalnichevski]

On Sat, 2016-09-17 at 15:55 +0200, Philippe Mouawad wrote:
> Hello,
> We have a bug report at JMeter :
> https://bz.apache.org/bugzilla/show_bug.cgi?id=60120
> 
> Where  a user post a form with a parameter having this value
> 'IqGo6EM1JEVZ+MSRJqUSo@qhjVMSFBTs'
> 
> It appears that the '@' character is encoded.
> 
> The form is submitted using application/x-www-form-urlencoded
> 
> As per rfc:
> > then reserved characters are escaped as described in [RFC1738]
> , section 2.2:
> 
> 
> > The characters ";"  "/", "?", ":", "@", "=" and "&" are the characters
> which may be reserved for special meaning within a scheme. No other
> characters may be reserved within a scheme.
> 
> So @ is reserved and as such HttpClient encodes it.
> 
> But it is not clear for me if @ is reserved when place in URI or also
> reserved when being part of a form parameter value.
> 
> In JMeter code, we use UrlEncodedFormEntity and I check that when parameter
> is passed , its value has still @ unencoded.
> 

Why do you think it should be encoded?

Oleg

> Thanks for clarification.
> Regards
> Philippe M.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Reading only the first X bytes of the response stream

[Oleg Kalnichevski]

On Thu, 2016-09-01 at 12:10 -0400, Dan Quaroni wrote:
> I'm writing a utility that will retrieve some metadata about swf (flash)
> files and I'd like to use httpclient to read the data but I want clarity on
> the necessity to read the entire stream.
> 
> The information I need is all in the first few bytes of the file, and the
> files could potentially be quite large so I don't want to read them in
> their entirety.  I'd like to just read what I need and then close the
> connection.
> 
> Here's a little test snippet, note that SWFInfo is closing the input stream
> on the response entity:
> 
> public static void main(String[] args) throws Exception {
> 
> CloseableHttpClient client = buildCloseableClient();
> HttpGet get = new HttpGet("http://localhost/Cat.swf;);
> for (int i = 0; i < 1; i++) {
> try (CloseableHttpResponse response = client.execute(get)) {
> SWFInfo info =
> SWFInfo.getInfo(response.getEntity().getContent());
> System.out.println(i + ": " + info.width + "," +
> info.height);
> }
> }
> }
> 
> 
> This test executes successfully, suggesting that it's OK but I want to
> double check that I won't be causing any types of resource usage problems.
> Note that we're using HttpClient in other parts of our application, but
> those are pooled connections that are constructed differently.
> 
> 

Hi Dan

Your code looks OK to me. As long as one closes CloseableHttpResponse
there should be no resource leaks.

Oleg

> Thanks,
> 
> Dan



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Closing HTTP connection of indefinite length

[Oleg Kalnichevski]

On August 30, 2016 10:26:29 PM GMT+02:00, Steven Schlansker 
<stevenschlans...@gmail.com> wrote:
>Thanks for the leads -- more questions inline.
>
>> On Aug 30, 2016, at 11:36 AM, Oleg Kalnichevski <ol...@apache.org>
>wrote:
>> 
>> On Mon, 2016-08-29 at 13:49 -0700, Steven Schlansker wrote:
>>> Hi httpclient-users,
>>> 
>>> I am using HttpClient 4.5.2 / HttpCore 4.4.5 to connect to a
>streaming endpoint.  The endpoint returns chunked records essentially
>forever.
>>> My application consumes these never ending records until an operator
>signals it to stop.  The code looks something like:
>>> 
>>> How can I abort this connection cleanly, without needing to read to
>the end of a
>>> never ending stream of data from the server?
>>> 
>> 
>> There are two options:
>> (1) Close the response without closing the content input stream
>
>I am looking at the HttpResponse interface (and its superinterface,
>HttpMessage) and not seeing
>anything like close().  The interfaces expose almost exclusively getter
>/ setter pairs.
>
>Did I miss something obvious here?

Look for ClosableHttpResponse in the HttpClient package.

Oleg
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Closing HTTP connection of indefinite length

[Oleg Kalnichevski]

On Mon, 2016-08-29 at 13:49 -0700, Steven Schlansker wrote:
> Hi httpclient-users,
> 
> I am using HttpClient 4.5.2 / HttpCore 4.4.5 to connect to a streaming 
> endpoint.  The endpoint returns chunked records essentially forever.
> My application consumes these never ending records until an operator signals 
> it to stop.  The code looks something like:
> 
> try (InputStream r = openHttpConnection(...)) {
>   while (running) {
> Record record = consumeRecord(r);
> processRecord(record);
>   }
> }
> 
> As you'd expect, when the operator signals stop, the ChunkedInputStream is 
> closed.
> However, the implementation of ChunkedInputStream.close is pretty painful in 
> this case:
> 
> if (!eof && state != CHUNK_INVALID) {
>   // read and discard the remainder of the message
>   final byte buff[] = new byte[BUFFER_SIZE];
>   while (read(buff) >= 0) {
>   }
> }
> 
> So my thread sits forever trying to close, discarding more and more data from
> the server.
> 
> How can I abort this connection cleanly, without needing to read to the end 
> of a
> never ending stream of data from the server?
> 

There are two options:
(1) Close the response without closing the content input stream
(2) Abort the request

In both cases the underlying connection will be shut down immediately
and returned to the connection pool.

Hope this helps

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Receiving error "java.io.IOException: Host name 'localhost' does not match the certificate subject provided by the peer"

[Oleg Kalnichevski]

On Tue, 2016-08-23 at 21:41 -0400, Sachin Nikumbh wrote:
> Hello,
> 
> I am working on a client application that is using
> both CloseableHttpAsyncClient and CloseableHttpClient with HTTPS. My
> application is sending a POST request with large amount of data and bunch
> of GET requests as well. I was initially using CloseableHttpAsyncClient for
> all the requests and everything was working fine with HTTPS. I recently
> changed my code to use CloseableHttpClient just for the POST request. Now,
> for the same client and server certificates (self signed), the SSL
> handshake for POST is failing with error:
> 
> Host name 'localhost' does not match the certificate subject provided by
> the peer (C=US, ST=MA, L=Natick, O=Mathworks, OU=MPS_QE, CN=localhost)
> 
> As can be seen, the CN of the server certificate is localhost. Following is
> the code that I am using to create CloseableHttpClient :
> 
> SSLConnectionSocketFactory sslFactory = new
> SSLConnectionSocketFactory(sslContext, hostnameVerifier);
> CloseableHttpClient httpClient = HttpClients.custom()
> .setUserAgent(ApacheHttpClient.class.getName())
> .setSSLSocketFactory(sslFactory)
> .build();
> 
> In the above code, sslContext is created using custom truststore and
> keystore using system properties
> like javax.net.ssl.keyStore, javax.net.ssl.trustStore, etc. The
> hostnameVerifier's verify method is returning false. I have a feeling that
> I am doing something wrong in the way I am setting up the
> CloseableHttpClient.
> 
> Could using CloseableHttpAsyncClient and CloseableHttpClient in the same
> application to use HTTPS cause any issues?
> 

I cannot think of any reason why it would Conceptually however using two
clients instead of one sounds somehow wrong to me.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: CloseableHttpClient and custom verification of SSL session

[Oleg Kalnichevski]

On August 21, 2016 10:50:47 PM GMT+02:00, Sachin Nikumbh  
wrote:
>Hi all,
>
>I am using CloseableHttpClient to support HTTPS. I need to do some
>client
>side verification after receiving the server certificate. This needs to
>happen immediately after the SSL handshake and before the actual data
>is
>exchanged. Can someone please direct me in the right direction?
>
>Thanks
>Sachin

Custom HostnameVerifier would be the most convenient plugin point.

Oleg
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Can a 201 (Created) response cause HTTP async client to prematurely send a FIN to the server?

[Oleg Kalnichevski]

On Fri, 2016-08-19 at 14:28 -0400, Sachin Nikumbh wrote:
> If the client is aborting the request after it receives the 201 response,
> is there any way to overwrite this behavior from the user's point of view?
> If I were to look into the source code to see where it is done, can someone
> please point me in the right direction in the sense of which package/class
> I should be looking at?
> 

https://github.com/apache/httpcore/blob/4.4.x/httpcore-nio/src/main/java/org/apache/http/nio/protocol/HttpAsyncRequestExecutor.java#L297

Oleg

> Thanks
> Sachin
> 
> On Fri, Aug 19, 2016 at 12:12 PM, David Motes  wrote:
> 
> > You are dealing with 2 different HTTP client implementations, the browser
> > and the Java HTTP client.
> >
> > >The section for 201 says it should be returned when "The request has been
> > fulfilled"
> >
> >  Sending a 201 back says the request has been fulfilled, there is nothing
> > more to do.
> > If you have fulfilled the request then what data are you sending in the
> > post?  It is obviously not needed if you can fulfill the request without
> > it.
> >
> >  I have not looked at the code but I would imagine when the Java client
> > gets the 201 response it thinks the request processing is complete so why
> > should I continue to send data. I will just FIN the connection and get out.
> >
> >  The browser code just works differently.
> >
> > This is not a problem with the Java client.
> >
> > On Fri, Aug 19, 2016 at 11:42 AM, Sachin Nikumbh 
> > wrote:
> >
> > > Hi David and Stefan,
> > >
> > > Thanks for your responses. When the client sends the POST request, we are
> > > sending the Content-Length to the correct value. Since the client is
> > using
> > > a duplex channel to write/read data, why would the server sending 201
> > cause
> > > any issues with client writing all the data? Shouldn't the client make
> > sure
> > > that it is writing all the data corresponding to Content-Length? Is the
> > > async client aborting request explicitly?
> > >
> > > I would like to add that the same server can successfully talk to a
> > browser
> > > without any issues with similar request payload. We also made sure that
> > we
> > > were successful in communicating with the server using a python client
> > > applications.
> > >
> > > We have verified using wireshark that when browser (javascript) or python
> > > sends the request, we receive the 201 while client is still in the middle
> > > of sending the request payload. After receiving 201, thew browser and
> > > python still continue with sending the data and successfully send all the
> > > data.
> > >
> > > I looked at the RFC : https://www.ietf.org/rfc/rfc2616.txt. The section
> > > for
> > > 201 says it should be returned when "The request has been fulfilled" and
> > we
> > > have fulfilled the request by creating our resource and by sending the
> > > resource uri as part of Location header.
> > >
> > >
> > > Thanks once again,
> > > Sachin
> > >
> > >
> > >
> > >
> > > On Fri, Aug 19, 2016 at 11:01 AM, David Motes 
> > > wrote:
> > >
> > > >  The response code should be sent back after the entire request has
> > been
> > > > processed. You cannot send a 201 created back after the headers and
> > then
> > > > discover something is wrong with the body and change the response code
> > to
> > > > some kind of error, it is too late.
> > > >
> > > >  Your server needs to be changed to send that response code upon
> > > completion
> > > > of the request.
> > > >
> > > > On Fri, Aug 19, 2016 at 10:47 AM, Sachin Nikumbh 
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I realized that in my previous post, I did not do a good job of
> > > > explaining
> > > > > the problem that I am facing. My sincere apologies for that.
> > > > >
> > > > > We have a custom server that my client Java application is
> > > communicating
> > > > > with using the async client. To be specific we are using an instance
> > of
> > > > > CloseableHttpAsyncClient from the client application. The client
> > sends
> > > a
> > > > > POST request with few kilo bytes of data. The server reads the
> > headers,
> > > > > sends a 201 back to the client acknowledging the receipt of request
> > and
> > > > > continues reading with the request body. What I see using wireshark
> > on
> > > > the
> > > > > client side is that the client receives 201 when it is still in the
> > > > middle
> > > > > of sending the data. But then the client sends a FIN even before it
> > has
> > > > > sent all the data. This results in server not receiving all the data.
> > > > >
> > > > > Now, if we remove the server 201 response, everything works fine. I.e
> > > > > client sends all the data to the server. We also don't see this
> > > behavior
> > > > if
> > > > > the client sends small amount of data < 3 kb.
> > > > >
> > > > > Is this a known issue? Are there any client side hooks that I can use
> > > to
> > > > 

Re: Trouble compiling httpcomponents-client-4.5.2 due to missing org.apache.http.annotation.Immutable and org.apache.http.annotation.NotThreadSafe classes

[Oleg Kalnichevski]

On Fri, 2016-08-19 at 14:26 -0400, Sachin Nikumbh wrote:
> Hi
> 
> I am trying to compile httpcomponents-client-4.5.2 and it's failing to find
> bunch of classes including org.apache.http.annotation.Immutable,
> org.apache.http.annotation.NotThreadSafe. Which package are these classes
> part of? I would like to include the source for them in my project.
> 
> Any help would be greatly appreciated.
> 
> Thanks
> Sachin

Sachin

You should be compiling HttpClient 4.5.2 with HttpCore 4.4.4. Newer
versions of HttpCore no longer ship with annotations originally based on
CC-BY licensed source. See for details:

https://issues.apache.org/jira/browse/HTTPCLIENT-1743

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Support for HTTP/2

[Oleg Kalnichevski]

On Mon, 2016-08-15 at 06:44 +, Refael Botbol wrote:
> Hi HTTPClient user list,
> 
> I was wondering if there is an ETA to support HTTP/2 ?
> 
> I've found this
> 
> thread from Jan 2016 which state it will be supported towards the end of
> the year (in alpha) - I'm wondering if this is still the ETA since more and
> more web-sites are moving to HTTP/2.
> 

ALPHA release with a partial HTTP/2 support by the end of the year is
still realistic. 

The problem is that HTTP/1.1 and HTTP/2 have completely different
message transport protocols with literally nothing in common but name.
If we want the HTTP/1.1 and HTTP/2 transports share the same protocol
handling code and have consistent message handling APIs (most likely we
do) I will likely have to completely rewrite the existing HTTP/1.1
message transport code. It is a daunting task. I am still unsure how to
approach it best and what impact it may have on the project time line.
All I can say I am working on it.

Oleg   



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HttpAsyncRequestProducer is not sending content fully with HTTPS

[Oleg Kalnichevski]

On Fri, 2016-08-12 at 13:02 +0900, Koji Kawamura wrote:
> Hello httpclient community,
> 
> I'm having an issue with HTTPS and HttpAsyncRequestProducer
> combination, so I would like to ask for some help.
> 
> I've been using httpclient HttpAsyncRequestProducer to make a
> non-blocking HTTP POST request.
> My use case consists of two threads, one is pushing data to a channel,
> and the other is reading streaming data from that channel, then
> producing a POST chunked content.
> 
> It's been working wonderfully with HTTP, but when I tested with HTTPS,
> noticed it only sends first few KB of data, even if I'm using the same
> client code.
> If the whole data size is less that about 8KB, it works, but bigger
> than that, further data is not sent.
> 
> Debugging around AbstractContentEncoder and SSLIOSession, but I
> couldn't find the exact cause of this behavior yet. But found that
> Line Buffer Size is being 8192, it seems this is the threshold of the
> behavior.
> 
> Wondering if calling ContentEncoder.write(ByteBuffer) multiple times
> causes this issue. It look like once about 8KB data is sent, further
> write() doesn't write anything, returning 0 even if the buffer has
> remaining data.
> 

Koji

This is how non-blocking I/O works. If the underlying I/O session is
unable to accept any more input the write operation returns immediately
with zero bytes written instead of blocking the thread. Once the I/O
session can accept more input it will trigger #produceContent event
again and the producer can produce more data. Please take a look at data
producers shipped with the library as reference.

Hope this helps

Oleg 

> To illustrate this issue, I wrote a simple test case which POST data
> via HTTP and HTTPS. The issue is reproducible.
> https://gist.github.com/ijokarumawak/6027ca64eb28fd143acd173d799ffd48
> 
> Has anyone encountered similar issue? I searched web and JIRA but
> couldn't find similar issue..
> Any advice or insight would be appreciated,
> 
> Thanks!
> Koji
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Building a parsed body cache atop the http caching client

[Oleg Kalnichevski]

On Fri, 2016-07-01 at 00:04 -0400, Donald Ball wrote:
> Hi folks. I'm using the caching http client to nice effect, thanks for the
> good work.
> 
> Since parsing json and other structured response bodies is not actually the
> fastest process in the world, I thought it would be a useful optimization
> to build a parsed body cache atop the http caching client. I've written one
> and it does work fine, but there is an annoying wrinkle: I don't know
> whether or not a given response is cacheable except by observing from the
> context's cache response if the response was in some way served from the
> cache. This results in having to perform two parses on a given body before
> it can be served from the parsed body cache. It's not the end of the world,
> but it's not optimal.
> 
> Is there some way that I have overlooked by which I can tell if a given
> response is cacheable?
> 
> I appreciate any thoughts. Thanks.

Have you looked at ResponseCachingPolicy? 

http://hc.apache.org/httpcomponents-client-4.5.x/httpclient-cache/xref/org/apache/http/impl/client/cache/ResponseCachingPolicy.html

Its #isResponseCacheable method is likely what you want.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents HttpAsyncClient 4.1.2 GA Released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.1.2 GA
release of HttpComponents HttpAsyncClient.

This is a maintenance release that fixes a number of minor issues
discovered since 4.1.1 and upgrades HttpCore and HttpClient
dependencies.

---
Download - 


Release notes -


HttpComponents site -


---
About Apache HttpAsyncClient

Although the java.net package provides basic functionality for accessing
resources via HTTP, it doesn't provide the full flexibility or
functionality needed by many applications. HttpAsyncClient seeks to fill
this void by providing an efficient, up-to-date, and feature-rich
package with an event-driven programming interface based on a
non-blocking I/O model.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Telling HC to make new connections all the time

[Oleg Kalnichevski]

On Wed, 2016-06-22 at 07:07 -0400, Benson Margulies wrote:
> On Wed, Jun 22, 2016 at 3:05 AM, Stefan Magnus Landrø
>  wrote:
> > Hi Benson,
> >
> > Have you considered implementing a custom
> > https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html
> > that handles DNS changes in a better way than the default implementation?
> > You could probably use http://www.xbill.org/dnsjava/ and lookup names
> > directly with the amazon NS?
> 
> Thanks for reminding me to ask about that. Does HC call the resolver
> as part of deciding whether to keep a connection or not? 

No, it does not, but one could make it do so by using a custom
ConnectionReuseStrategy.

Oleg

> The problem
> as we see it is this: the DNS mapping for an ELB changes, and a minute
> later the connections to it become useless but don't necessarily get
> properly closed with a FIN by the ELB. I've written some goofy code
> that notices when the address resolution changes for any of the ELB's
> we care about, and when it does it calls 'closeIdleConnections' with a
> time of 0. On the other hand, I don't think we're currently having any
> problems getting InetAddress.getByName to keep up with Amazon, but
> your suggestion would be safer.
> 
> 
> >
> > Stefan
> >
> > 2016-06-20 19:57 GMT+02:00 Benson Margulies :
> >
> >> We use the async http components library to connect various components
> >> in a system that is deployed, in some cases, on AWS. When it's
> >> deployed on AWS, there are ELBs involved.
> >>
> >> Like some other people, we've having various trauma dealing with the
> >> very dynamic environment of AWS ELBs, which includes constant DNS
> >> changes and some other issues. So far, we've been making our HC
> >> configuration more and more complex. I'm wondering if we should back
> >> off and make it simpler -- make a new connection for each request.
> >> Obviously, this will be somewhat slower and more expensive, but at the
> >> moment we might prefer a slower and more expensive approach that works
> >> to one that occasionally gets lost.
> >>
> >> I appreciate that even making new connections all the time won't
> >> compensate for all possible DNS problems -- if Java itself caches a
> >> stale name->address translation, making a new connection will fail
> >> just as effectively as anything else.
> >>
> >> Does anyone out there have advice to share (other than running away from
> >> ELB)?
> >>
> >> -
> >> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> >> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >>
> >>
> >
> >
> > --
> > BEKK Open
> > http://open.bekk.no
> >
> > TesTcl - a unit test framework for iRules
> > http://testcl.com
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Help/Advise debugging SocketException in BHttpConnectionBase.fillInputBuffer()

[Oleg Kalnichevski]

On Wed, 2016-06-22 at 13:00 +0200, Mark Rekveld wrote:
> Hi,
> 
> I would like to ask for your assistance to resolve an issue that we are
> facing that has the nature of being a race condition.
> 
> The problem is that a SocketException is thrown in the finally clause of
> BHttpConnectionBase.fillInputBuffer() [
> https://github.com/apache/httpcore/blob/4.3.x/httpcore/src/main/java/org/apache/http/impl/BHttpConnectionBase.java#L326
> ]
> 
> We use 4.3.6 of the HttpClient with 4.3.3 of HttpCore, and have a custom
> extension of org.apache.http.protocol.HttpRequestExecutor that sends the
> request entity asynchronously and immediately try to read the response from
> the sever.
> 

We can only adequately support two branches of HttpClient: stable
(currently 4.5.x) and dev (currently 5.0-alpha)

Please consider upgrading to HttpClient 4.5. There have been numerous
changes since 4.3.6.  

Oleg

> The exception occurs within this custom HttpRequestExecutor when the
> conn.isResponseAvailable(1000 * 5) call is made in this code snippet. See
> stack trace below
> 
> private void sendRequestEntity(final HttpEntityEnclosingRequest req) throws
> IOException {
>   Thread entitySender = new Thread(getClass().getSimpleName() + "-" +
> System.currentTimeMillis())
>   {
> @Override
> public void run() {
>   try {
> conn.sendRequestEntity(req);
> conn.flush();
> context.setAttribute(HttpCoreContext.HTTP_REQ_SENT, Boolean.TRUE);
>   }
>   catch (Throwable t) {
> error = t;
>   }
> }
>   };
>   entitySender.start();
> 
>   while (entitySender.isAlive() && !conn.isResponseAvailable(1000 * 5)) {
> /*
>  * Wait here until either we have fully uploaded the request entity or
> the server sends a response. If
>  * we were to return immediately and called doReceiveResponse() right
> away we would provoke socket
>  * timeouts.
>  */
>   }
> }
> 
> The full stack trace:
> 
> 2016-06-03 02:36:04,149+1000 ERROR [qtp761509273-12469 - PUT
> /rest/ci/scan/treasury-web-deployable] jenkins-server
> com.sonatype.insight.brain.hds.HdsClient - Socket is closed
> java.net.SocketException: Socket is closed
>  at java.net.Socket.setSoTimeout(Socket.java:1137) ~[na:1.8.0_91]
>  at
> sun.security.ssl.BaseSSLSocketImpl.setSoTimeout(BaseSSLSocketImpl.java:633)
> ~[na:1.8.0_91]
>  at sun.security.ssl.SSLSocketImpl.setSoTimeout(SSLSocketImpl.java:2526)
> ~[na:1.8.0_91]
>  at
> org.apache.http.impl.BHttpConnectionBase.fillInputBuffer(SourceFile:326)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.BHttpConnectionBase.awaitInput(SourceFile:334)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.http.impl.DefaultBHttpClientConnection.isResponseAvailable(SourceFile:131)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.http.impl.conn.CPoolProxy.isResponseAvailable(SourceFile:141)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.sendRequestEntity(SourceFile:201)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.doSendRequest(SourceFile:177)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.client.utils.HttpRequestExecutor$HttpExchange.perform(SourceFile:108)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> com.sonatype.insight.client.utils.HttpRequestExecutor.execute(SourceFile:53)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.execchain.MainClientExec.execute(SourceFile:254)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.execchain.ProtocolExec.execute(SourceFile:195)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.execchain.RetryExec.execute(SourceFile:86)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.execchain.RedirectExec.execute(SourceFile:108)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at
> org.apache.http.impl.client.InternalHttpClient.doExecute(SourceFile:184)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:82)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:106)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at org.apache.http.impl.client.CloseableHttpClient.execute(SourceFile:57)
> ~[nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:380)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.execute(SourceFile:329)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.getResponse(SourceFile:132)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at com.sonatype.insight.brain.hds.HdsClient.get(SourceFile:178)
> [nexus-iq-server-1.20.0-02.jar:1.20.0-02]
>  at 

Re: Telling HC to make new connections all the time

[Oleg Kalnichevski]

On Tue, 2016-06-21 at 08:28 -0400, Benson Margulies wrote:
> Oleg, If I set a TTL, do I also need an IdleConnectionMonitorThread (
> http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d5e652
> )? thanks, benson.
> 

TTL setting only ensures connections cannot get re-used beyond their TTL
but they can still be kept alive in the pool until the pool manager
discards them either while trying to find a valid connection or while
evicting expired ones. One might still need IdleConnectionMonitorThread
to ensure eviction of expired connections when the connection manager
becomes idle after a period of heavy activity. 

Oleg

> 
> On Tue, Jun 21, 2016 at 4:41 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> > On Mon, 2016-06-20 at 13:57 -0400, Benson Margulies wrote:
> >> We use the async http components library to connect various components
> >> in a system that is deployed, in some cases, on AWS. When it's
> >> deployed on AWS, there are ELBs involved.
> >>
> >> Like some other people, we've having various trauma dealing with the
> >> very dynamic environment of AWS ELBs, which includes constant DNS
> >> changes and some other issues. So far, we've been making our HC
> >> configuration more and more complex. I'm wondering if we should back
> >> off and make it simpler -- make a new connection for each request.
> >> Obviously, this will be somewhat slower and more expensive, but at the
> >> moment we might prefer a slower and more expensive approach that works
> >> to one that occasionally gets lost.
> >>
> >> I appreciate that even making new connections all the time won't
> >> compensate for all possible DNS problems -- if Java itself caches a
> >> stale name->address translation, making a new connection will fail
> >> just as effectively as anything else.
> >>
> >> Does anyone out there have advice to share (other than running away from 
> >> ELB)?
> >>
> >
> > Hi Benson
> >
> > I personally would consider disabling connection persistence extreme and
> > unnecessary. Consider setting the TTL (total time to live) of persistent
> > connections to some low value (say, 10 seconds or even less) instead.
> > This should improve distribution of connections across the cluster of
> > servers behind a load balancer.
> >
> > Hope it helps
> >
> > Oleg
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Telling HC to make new connections all the time

[Oleg Kalnichevski]

On Mon, 2016-06-20 at 13:57 -0400, Benson Margulies wrote:
> We use the async http components library to connect various components
> in a system that is deployed, in some cases, on AWS. When it's
> deployed on AWS, there are ELBs involved.
> 
> Like some other people, we've having various trauma dealing with the
> very dynamic environment of AWS ELBs, which includes constant DNS
> changes and some other issues. So far, we've been making our HC
> configuration more and more complex. I'm wondering if we should back
> off and make it simpler -- make a new connection for each request.
> Obviously, this will be somewhat slower and more expensive, but at the
> moment we might prefer a slower and more expensive approach that works
> to one that occasionally gets lost.
> 
> I appreciate that even making new connections all the time won't
> compensate for all possible DNS problems -- if Java itself caches a
> stale name->address translation, making a new connection will fail
> just as effectively as anything else.
> 
> Does anyone out there have advice to share (other than running away from ELB)?
> 

Hi Benson

I personally would consider disabling connection persistence extreme and
unnecessary. Consider setting the TTL (total time to live) of persistent
connections to some low value (say, 10 seconds or even less) instead.
This should improve distribution of connections across the cluster of
servers behind a load balancer.  

Hope it helps

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

[ANNOUNCEMENT] HttpComponents Core 4.4.5 GA released

[Oleg Kalnichevski]

The Apache HttpComponents project is pleased to announce 4.4.5 GA
release of HttpComponents Core. 

This is a maintenance release that fixes a number of issues discovered
since 4.4.4.

Please note that as of 4.4 HttpCore requires Java 1.6 or newer.

Download - 
Release notes -

HttpComponents site - 

About HttpComponents Core

HttpCore is a set of low level HTTP transport components that can be
used to build custom client and server side HTTP services with a minimal
footprint. HttpCore supports two I/O models: a blocking I/O model based
on the classic Java I/O and a non-blocking, event driven I/O model based
on Java NIO.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Keep-Alive

[Oleg Kalnichevski]

On Mon, 2016-06-13 at 11:23 -0700, Rallavagu wrote:
> 
> On 6/7/16 6:13 AM, Oleg Kalnichevski wrote:
> > On Mon, 2016-06-06 at 13:43 -0700, Rallavagu wrote:
> >> HttpClient 4.3.2, JDK 7
> >>
> >> I was doing something similar to what is talked in this link related to
> >> http commons client 3.x
> >>
> >> http://httpcomponents.10934.n7.nabble.com/quot-Keep-alive-quot-stale-connections-and-socket-reuse-td15315.html
> >>
> >> I am using PoolingHttpClientConnectionManager with following SocketConfig.
> >>
> >> SocketConfig socketConfig =
> >> SocketConfig.copy(SocketConfig.DEFAULT).setSoKeepAlive(true).setSoReuseAddress(true).build();
> >>
> >>  From debug logs, I see that connections are maintained in the pool.
> >> Here is the snippet from the log.
> >>
> >> [DEBUG] 06/06/2016 12:58:16
> >> org.apache.http.impl.conn.PoolingHttpClientConnectionManager Connection
> >> request: [route: {s}->https://:443][total kept alive: 1; route
> >> allocated: 1 of 50; total allocated: 1 of 100]
> >> [DEBUG] 06/06/2016 12:58:16
> >> org.apache.http.impl.conn.PoolingHttpClientConnectionManager Connection
> >> leased: [id: 2][route: {s}->https://:443][total kept alive: 0;
> >> route allocated: 1 of 50; total allocated: 1 of 100]
> >> [DEBUG] 06/06/2016 12:58:16
> >> org.apache.http.impl.conn.PoolingHttpClientConnectionManager Connection
> >> released: [id: 2][route: {s}->https://:443][total kept alive: 1;
> >> route allocated: 1 of 50; total allocated: 1 of 100]
> >>
> >> What I am noticing is that every time a connection is used (leased) from
> >> the pool, a connection is being established (socketConnect) to the
> >> server unless requested in quick successions where a connection is
> >> detected as "not stale". Snippet from "MainClientExec.java"
> >>
> >> if (config.isStaleConnectionCheckEnabled()) {
> >>  // validate connection
> >>  if (managedConn.isOpen()) {
> >>  this.log.debug("Stale connection check");
> >>  if (managedConn.isStale()) {
> >>  this.log.debug("Stale connection detected");
> >>  managedConn.close();
> >>  }
> >>  }
> >>  }
> >>
> >>
> >> Using AspectJ, I could track connection opening to the server.
> >>
> >> [INFO] 06/06/2016 12:58:00 com.test.aop.AbstractLogAspect
> >> SSLConnectionSocketFactory.java:232.connectSocket arguments [3,
> >> Socket[addr=/,port=443,localport=49439],
> >> https://:443, /:443, null,
> >> org.apache.http.client.protocol.HttpClientContext@137f7542] Execution
> >> time: 313
> >>
> >> The above call is not happening when "managedConn.isStale()" returns false.
> >>
> >> As you can see, I would like to avoid the "connectSocket" call as it is
> >> expensive. Am I right in my assumption that a "keep-alive" connection is
> >> supposed to work on an already established socket so it is being reused?
> >
> > Yes, you are.
> 
> Thanks. I could validate that keep-alive connection works on already 
> established socket.
> >
> >> Also,  I understand that we can't rely on isStale() method. Any
> >> comments/suggestions are welcome.
> >>
> >
> > Please consider upgrading to 4.5.2. Stale connection check logic has
> > been reworked in 4.5
> I have upgraded to 4.5.2 and found that "setValidateAfterInactivity" 
> method could be used to check for stale connections.
> 
> I have a background thread to detect and clear idle/expired connections 
> from PoolingHttpClientConnectionManager. I wonder how would 
> "setValidateAfterInactivity" impact that if at all.
> 

It would not (unless, of course, expiry deadline is less than inactivity
period).

> I understand that per http/1.1 spec a keep-alive request is assumed to 
> be "forever" in case server does not respond with keep-alive header. 
> But, connections are in fact not left open forever (in tomcat, it 
> defaults to 20 sec; connectionTimeout="2"). Doesn't it make sense to 
> set a response header so a client can manage keep-alive connection more 
> gracefully by sending "Connection: close" when they expire? In the case 
> of background thread to check expired connections, there won't be any as 
> server never responded with keep-alive header and connections are marked 
> to be kept alive forever. Is the only solution is to create a servlet 
> filter that sets the response header?
> 

Use custom ConnectionKeepAliveStrategy to set a finite keep-alive period
by default .

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: ManagedNHttpClientConnectionImpl Shutdown Vs. Close

[Oleg Kalnichevski]

On Fri, 2016-06-10 at 14:15 +, Poonam Chawla wrote:
> Hello Oleg, 
> 
> 
> I am using Apache HttpAsycnClient 4.1 and I noticed that the
> connections are not released on calling close.
> Here is what I am trying to do: (Please note that it is not a
> compileable code as I had to remove lines that are internal to my
> company). 
> 
> 
> - Create CloseableHttpAsyncClient
> 
> 
> CloseableHttpAsyncClient httpClientAsync;
> 
> 
> SSLIOSessionStrategy sslsf = new SSLIOSessionStrategy(sslContext,
> hostnameVerifier) {
> @Override
> protected void verifySession(HttpHost host,
> org.apache.http.nio.reactor.IOSession
> iosession,
> SSLSession sslsession) throws SSLException {
> verifySSLSession(iosession, sslsession,
> sslConfig);
> }
> };
> 
> 
> Registry sessionStrategyRegistry
> = RegistryBuilder
> . create()
> .register("http", NoopIOSessionStrategy.INSTANCE)
> .register("https", sslsf).build();
> 
> 
> IOReactorConfig ioReactorConfig = IOReactorConfig.custom()
> .setConnectTimeout((int)
> clientConfig.getTimeOutMs())
> .setSoTimeout((int) clientConfig.getTimeOutMs())
> .setTcpNoDelay(false).build();
> 
> 
> ConnectingIOReactor ioReactor = new
> DefaultConnectingIOReactor(
> ioReactorConfig);
> 
> 
> NHttpConnectionFactory
> connFactory = new ManagedNHttpClientConnectionFactory();
> PoolingNHttpClientConnectionManager connManager = new
> PoolingNHttpClientConnectionManager(
> ioReactor,connFactory, sessionStrategyRegistry);
> connManager.setDefaultMaxPerRoute(clientConfig
> .getMaxConnectionsPerAddress());
> 
> 
> httpClientAsync = HttpAsyncClients.custom()
> .setUserAgent(ApacheHttpClient.class.getName())
> .setConnectionManager(connManager)
> .setSSLContext(sslContext)
> .setSSLHostnameVerifier(hostnameVerifier)
> .setSSLStrategy(sslsf).build();
> httpClientAsync.start();
> 
> 
> 
> 
> - Execute HTTPPosts using the httpClientAsync 
> 
> 
> try{ 
> // underneath creates the CloseableHttpAsyncClient
> MWHttpClient client = new MWHttpClient();
> 
> //underneath calls httpClientAsync.execute(….) 
> Future<> f = client.invoke(….)
> f.get()
> 
> 
> //underneath calls httpClientAsync.execute(….)
> f1 = httpClientAsync.execute(…)
> 
> 
> catch{
> }finally{
> // spawn a new thread to close the httpClientAsync
> // If there are pending requests waiting for a response and the user
> calls close,
> // abort the pending requests using request.abort();
> // underneath calls httpClientAsync.close();
> 
> 
> client.close();
> }
> 
> 
> 
> 
> Here is a part of the entire log I am confused about:
> 
> 
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> GET 
> /~81b806db-4546-4a7d-b623-6f0848ff6b05/requests?since=2448=protobuf=97628818-0daf-405c-a14c-4891738587b7
>  HTTP/1.1
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >> Host:
> localhost:53097
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> Connection: Keep-Alive
> 2016/06/09 16:30:34:753 EDT [DEBUG] headers - http-outgoing-0 >>
> User-Agent: com.mathworks.mps.client.internal.ApacheHttpClient
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: Event
> set [w]
> 2016/06/09 16:30:34:753 EDT [DEBUG] MainClientExec - [exchange: 12]
> Request completed
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][rw:w]: 247
> bytes written
> 2016/06/09 16:30:34:753 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]: Event
> cleared [w]
> 2016/06/09 16:30:34:852 EDT [DEBUG] InternalHttpAsyncClient -
> [exchange: 12] Cancelled
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-0 127.0.0.1:54726<->127.0.0.1:53097[ACTIVE][r:w]:
> Shutdown
> 2016/06/09 16:30:34:853 EDT [DEBUG]
> PoolingNHttpClientConnectionManager - Connection manager is shutting
> down
> 2016/06/09 16:30:34:853 EDT [DEBUG] ManagedNHttpClientConnectionImpl -
> http-outgoing-1 127.0.0.1:54727<->127.0.0.1:53097[ACTIVE][r:r]: Close
> 2016/06/09 16:30:34:853 EDT [DEBUG] InternalHttpAsyncClient -
> [exchange: 12] connection aborted
> 2016/06/09 16:30:34:854 EDT [DEBUG] InternalIODispatch -
> http-outgoing-0 [CLOSED]: Disconnected
> 2016/06/09 16:30:34:855 EDT [DEBUG] InternalIODispatch -
> http-outgoing-1 [CLOSED]: Disconnected
> 2016/06/09 16:30:34:855 EDT [DEBUG]
> 

Re: How to retrieve server ip of the connected connections

[Oleg Kalnichevski]

On Thu, 2016-06-09 at 19:40 +, Vicky Singh wrote:
> 
> 
>  I am using PoolingHttpClientConnectionManager to create connections. I would 
> like to get what IP address my connections resolve to. Is there a API that I 
> can use.
> I need it for reporting and also to drop connections if I think connected IP 
> is not the best IP address. I found this 
> https://issues.apache.org/jira/browse/HTTPCLIENT-656  
> 
> Summary of my code,
> PoolingHttpClientConnectionManager connMgr = new 
> PoolingHttpClientConnectionManager(socketFactoryRegistry);// 
> build custom http client . ...Set hosts = 
> connMgr.getRoutes();for (HttpRoute r: hosts) {InetAddress 
> ip = r.getTargetHost().getAddress();break;}
> The above code gets an IP only if HttpHost was created using IP in 
> constructor . From documentation of  HttpHost.getAddress() Returns the inet 
> address if explicitly set by a constructor, null otherwise.
> 

There are different ways to approach the problem
(1) Have full control over hostname resolution by using a custom
DnsResolver
(2) Get hold of active connections from a custom subsclass of
HttpRequestExecutor 

Oleg
  



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Backward compatibility report

[Oleg Kalnichevski]

On Wed, 2016-06-08 at 16:05 +0300, Ponomarenko Andrey wrote:
> Hello,
> 
> Thank you for reporting this bug. Usually the tool checks super-classes and 
> doesn't report removal of overwritten methods. But in this case the 
> super-class is defined in the different jar (httpcore-4.2.5.jar) and 
> therefore it was not taken into account. I'll try to fix the report soon.
> 

Great! 

Are you by any chance going to build a Maven plugin for your report
generator? I would love to replace Clirr plugin with something better.

Oleg 

> 07.06.2016, 16:09, "Oleg Kalnichevski":
> > On Mon, 2016-06-06 at 18:09 +0300, Ponomarenko Andrey wrote:
> >>  Hello,
> >>
> >>  I've prepared backward compatibility reports for HttpClient and HttpCore 
> >> libraries:
> >>
> >>  http://abi-laboratory.pro/java/tracker/timeline/httpclient/
> >>  http://abi-laboratory.pro/java/tracker/timeline/httpcore/
> >>
> >>  I hope they will help users, maintainers and developers to maintain 
> >> backward compatibility. The reports are updated daily by the 
> >> japi-compliance-checker and japi-tracker tools: 
> >> https://github.com/lvc/japi-compliance-checker
> >>
> >>  Please let me know if there are some false positives in the report.
> >>
> >>  Thank you.
> >
> > Hi Andrey
> >
> > The reports look pretty awesome.
> >
> > However, if I am not mistaken removal of overwritten methods from a
> > subclass should not affect binary compatibility, should it?
> >
> > http://abi-laboratory.pro/java/tracker/compat_report/httpcore/4.2.5/4.3/55421/bin_compat_report.html#Removed
> >
> > High severity for these changes sounds a bit too much if you ask me.
> >
> > Oleg
> >
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SSL problem when using pooling connection manager

[Oleg Kalnichevski]

On Tue, 2016-06-07 at 15:45 +0200, Ivan Brencsics wrote:
> Hi Oleg,
> 
> Thanks for your answer, maybe I did not explain clearly our use case. We
> need to call several external systems over HTTP, and every system has
> different requirements. All need TLS, but some need client certificate,
> others not, some of them needs a certain client certificate that differs
> from others, some of them wants to exclude certain protocols and cipher
> suites. My idea is that we need as many SSLConnectionFactiories as many
> external systems. This is what I cant achieve now with a single HttpClient
> / PoolingHttpClientConnectionManager. The PoolingHttpClientConnectionManager
> can be given a single HTTPS connection factory, but I would need multiple
> different factories I suspect. Am I wrong with this?
> 

No need for multiple factories. A single factory should be perfectly
capable of setting different SSL contexts based on hostname /  DNS /
HttpContext attributes.

Oleg 

> Ivan
> 
> 2016-06-07 15:18 GMT+02:00 Oleg Kalnichevski <ol...@apache.org>:
> 
> > On Tue, 2016-06-07 at 11:06 +0200, Ivan Brencsics wrote:
> > > Hi,
> > >
> > > I have some trouble using SSL together
> > > with PoolingHttpClientConnectionManager.
> > >
> > > I create an HttpClient by setting both the SSLContext and
> > ConnectionManager
> > > and the SSLContext is ignored. I read that others had the same problem
> > and
> > > solved it by adding a Socket Factory Registry to the Connection Manager.
> > > However, I dont find this a good solution. I would like to create one and
> > > only one connection pool for my application, and then provide it to all
> > > HttpClient instances I create later on.
> >
> > What is the point doing so? You might as well have just one instance of
> > HttpClient which is basically recommended anyway.
> >
> > > When I instantiate the connection
> > > pool, I dont know exactly what connections I will create later with what
> > > SSL parameters. Your solution assumes that when instantiating the
> > > PoolingHttpClientConnectionManager, we are already aware of all the SSL
> > > settings that we will ever use. This is very often not the case.
> > >
> > > Why are the SSL settings so tightly coupled to
> > > PoolingHttpClientConnectionManager?
> >
> > Because the pool manager is responsible for keeping track of persistent
> > connection state.
> >
> > > And do I understand correctly that if I
> > > am using SSL, I should create separate PoolingHttpClientConnectionManager
> > > instances towards the different remote servers?
> > >
> >
> > No, you should not.
> >
> > Oleg
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >
> >



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SSL problem when using pooling connection manager

[Oleg Kalnichevski]

On Tue, 2016-06-07 at 11:06 +0200, Ivan Brencsics wrote:
> Hi,
> 
> I have some trouble using SSL together
> with PoolingHttpClientConnectionManager.
> 
> I create an HttpClient by setting both the SSLContext and ConnectionManager
> and the SSLContext is ignored. I read that others had the same problem and
> solved it by adding a Socket Factory Registry to the Connection Manager.
> However, I dont find this a good solution. I would like to create one and
> only one connection pool for my application, and then provide it to all
> HttpClient instances I create later on. 

What is the point doing so? You might as well have just one instance of
HttpClient which is basically recommended anyway.

> When I instantiate the connection
> pool, I dont know exactly what connections I will create later with what
> SSL parameters. Your solution assumes that when instantiating the
> PoolingHttpClientConnectionManager, we are already aware of all the SSL
> settings that we will ever use. This is very often not the case.
> 
> Why are the SSL settings so tightly coupled to
> PoolingHttpClientConnectionManager? 

Because the pool manager is responsible for keeping track of persistent
connection state. 

> And do I understand correctly that if I
> am using SSL, I should create separate PoolingHttpClientConnectionManager
> instances towards the different remote servers?
> 

No, you should not.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Keep-Alive

[Oleg Kalnichevski]

On Mon, 2016-06-06 at 13:43 -0700, Rallavagu wrote:
> HttpClient 4.3.2, JDK 7
> 
> I was doing something similar to what is talked in this link related to 
> http commons client 3.x
> 
> http://httpcomponents.10934.n7.nabble.com/quot-Keep-alive-quot-stale-connections-and-socket-reuse-td15315.html
> 
> I am using PoolingHttpClientConnectionManager with following SocketConfig.
> 
> SocketConfig socketConfig = 
> SocketConfig.copy(SocketConfig.DEFAULT).setSoKeepAlive(true).setSoReuseAddress(true).build();
> 
>  From debug logs, I see that connections are maintained in the pool. 
> Here is the snippet from the log.
> 
> [DEBUG] 06/06/2016 12:58:16 
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager Connection 
> request: [route: {s}->https://:443][total kept alive: 1; route 
> allocated: 1 of 50; total allocated: 1 of 100]
> [DEBUG] 06/06/2016 12:58:16 
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager Connection 
> leased: [id: 2][route: {s}->https://:443][total kept alive: 0; 
> route allocated: 1 of 50; total allocated: 1 of 100]
> [DEBUG] 06/06/2016 12:58:16 
> org.apache.http.impl.conn.PoolingHttpClientConnectionManager Connection 
> released: [id: 2][route: {s}->https://:443][total kept alive: 1; 
> route allocated: 1 of 50; total allocated: 1 of 100]
> 
> What I am noticing is that every time a connection is used (leased) from 
> the pool, a connection is being established (socketConnect) to the 
> server unless requested in quick successions where a connection is 
> detected as "not stale". Snippet from "MainClientExec.java"
> 
> if (config.isStaleConnectionCheckEnabled()) {
>  // validate connection
>  if (managedConn.isOpen()) {
>  this.log.debug("Stale connection check");
>  if (managedConn.isStale()) {
>  this.log.debug("Stale connection detected");
>  managedConn.close();
>  }
>  }
>  }
> 
> 
> Using AspectJ, I could track connection opening to the server.
> 
> [INFO] 06/06/2016 12:58:00 com.test.aop.AbstractLogAspect 
> SSLConnectionSocketFactory.java:232.connectSocket arguments [3, 
> Socket[addr=/,port=443,localport=49439], 
> https://:443, /:443, null, 
> org.apache.http.client.protocol.HttpClientContext@137f7542] Execution 
> time: 313
> 
> The above call is not happening when "managedConn.isStale()" returns false.
> 
> As you can see, I would like to avoid the "connectSocket" call as it is 
> expensive. Am I right in my assumption that a "keep-alive" connection is 
> supposed to work on an already established socket so it is being reused? 

Yes, you are.

> Also,  I understand that we can't rely on isStale() method. Any 
> comments/suggestions are welcome.
> 

Please consider upgrading to 4.5.2. Stale connection check logic has
been reworked in 4.5 

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Backward compatibility report

[Oleg Kalnichevski]

On Mon, 2016-06-06 at 18:09 +0300, Ponomarenko Andrey wrote:
> Hello,
> 
> I've prepared backward compatibility reports for HttpClient and HttpCore 
> libraries:
> 
> http://abi-laboratory.pro/java/tracker/timeline/httpclient/
> http://abi-laboratory.pro/java/tracker/timeline/httpcore/
> 
> I hope they will help users, maintainers and developers to maintain backward 
> compatibility. The reports are updated daily by the japi-compliance-checker 
> and japi-tracker tools: https://github.com/lvc/japi-compliance-checker
> 
> Please let me know if there are some false positives in the report.
> 
> Thank you.
> 

Hi Andrey

The reports look pretty awesome. 

However, if I am not mistaken removal of overwritten methods from a
subclass should not affect binary compatibility, should it?

http://abi-laboratory.pro/java/tracker/compat_report/httpcore/4.2.5/4.3/55421/bin_compat_report.html#Removed

High severity for these changes sounds a bit too much if you ask me.  

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Multipart Mixed

[Oleg Kalnichevski]

On Thu, 2016-06-02 at 12:52 +0200, Stefan Magnus Landrø wrote:
> Hi there,
> 
> We're struggeling with some limitations in the current multipart
> implementation in 4.5 (latest).
> 
> The MultipartEntityBuilder is actually building a MultipartFormEntity,
> whereas we actually want to be able to build a multipart/mixed entity which
> does have limitations with respect to filesize as defined in
> MultipartFormEntity.
> 
> We're considering improving the MultipartEntityBuilder in such a way that
> it can also produce MultipartMixedEntity (new type)
> 
> Oleg, any comments?
> 

Stefan

I am not sure I fully understand the issue but you are welcome to submit
a patch with changes you deem necessary.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: I/O dispatch worker terminated abnormally, what to do?

[Oleg Kalnichevski]

On Mon, 2016-05-30 at 21:37 +, John Price wrote:
> We're using the HttpAsyncClient to make calls from within our own server
> application. Periodically (once a week or less), our service stops being
> able to make requests and we see errors of the form:
> 
> java.lang.IllegalStateException: Request cannot be executed; I/O reactor
> status: STOPPED
> at org.apache.http.util.Asserts.check(Asserts.java:46)
> at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
> at
> org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
> ...
> 
> Digging further into the logs, it seems these begin after we see an
> exception that I've included at the bottom but basically seems to boil down
> to "I/O dispatch worker terminated abnormally" caused by "IOException:
> Connection reset by peer" in the depths of SSLIOSession.
> 
> I guess I've got two questions: first, is this a bug? I'm not sure how much
> additional info I can provide into the root cause of the IOException since
> it happens so infrequently, but I'll try to provide what I can if it's
> relevant.
> 
> Second, how should we be handling this issue? Some of the Apache
> documentation and older mailing list posts talk about "restarting the I/O
> Reactor" but I'm not clear what means in this context since I'm not
> interacting with it directly. Should I just be throwing away my instance of
> HttpAsyncClient and creating a new one?
> 
> We're using:
> httpclient 4.5.2
> httpasyncclient 4.1.1
> httpcore.nio4.4.4
> 
> John.
> 
> org.apache.http.nio.reactor.IOReactorException: I/O dispatch worker
> terminated abnormally
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:358)
> 
> at
> org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:191)
> 
> at
> org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
> 
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
> Caused by: rx.exceptions.OnErrorNotImplementedException: Connection reset
> by peer

The original 'java.io.IOException: Connection reset by peer' exception
was re-thrown as OnErrorNotImplementedException from inside the
FutureCallback#failed method, which killed the i/o dispatch thread and
caused the i/o reactor to shut down.  

This issue needs to be addressed either in your application code or in
the RX library.

Hope this helps

Oleg


> at rx.Observable$27.onError(Observable.java:7923)
> at rx.observers.SafeSubscriber._onError(SafeSubscriber.java:159)
> at rx.observers.SafeSubscriber.onError(SafeSubscriber.java:120)
> at
> rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
> at OurCode.SessionManager.lambda$null$10(SessionManager.java:174)
> at rx.Observable$10.onError(Observable.java:4407)
> at
> rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:65)
> at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.reportError(OperatorMerge.java:240)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.checkTerminate(OperatorMerge.java:776)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.emitLoop(OperatorMerge.java:537)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.emit(OperatorMerge.java:526)
> at
> rx.internal.operators.OperatorMerge$MergeSubscriber.onError(OperatorMerge.java:250)
> at rx.internal.operators.OperatorMap$1.onError(OperatorMap.java:48)
> at
> rx.internal.operators.OperatorDoOnEach$1.onError(OperatorDoOnEach.java:71)
> at rx.apache.http.ObservableHttp$2$1.failed(ObservableHttp.java:195)
> at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134)
> at
> org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:419)
> 
> at
> org.apache.http.nio.protocol.HttpAsyncRequestExecutor.exception(HttpAsyncRequestExecutor.java:152)
> 
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:76)
> 
> at
> org.apache.http.impl.nio.client.InternalIODispatch.onException(InternalIODispatch.java:39)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:127)
> 
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278)
> 
> at
> org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
> 
> at
> org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
> Caused 

Re: How to debug a hang during POST

[Oleg Kalnichevski]

On Tue, 2016-05-24 at 17:24 +0100, Tim Dudgeon wrote:
> Hi, I'm having problems with a POST operation that's being done along 
> these lines:
> 
> HttpPost post = new HtttpPost("http://.;);
> post.addHeader("name", "value"); // etc
> post.setEntity( ... );
> CloseableHttpResponse resp = httpClient.execute(post);
> ...
> 
> First time the operation works fine. But a second time it hangs at the 
> execute step.
> Other similar scenarios work fine repeatedly, but in on particular case 
> it hangs second time through.
> I was thinking it might be due to the response not being closed but I 
> can't find any cases where this could be the case.
> 
> Any suggestions what to look for and how to debug this?
> 
> Thanks
> Tim
> 
> 
> 

Tim,

Please post wire / context log of the session.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: using apache httpclient efficiently in multithreaded environment

[Oleg Kalnichevski]

On Fri, 2016-05-13 at 10:46 -0700, Check Peck wrote:
> You mean to say this is inefficient? If yes, then what should I do instead
> of below?
> 
> String responseBody = IOUtils.toString(entity.getContent(), StandardCharsets
> .UTF_8);
> 

You should avoid building intermediate representations of response
content. If your application layer ultimately expects JSON objects as
input you should be creating JSON objects directly from the underlying
content stream instead of building an intermediate string and then
parsing it to JSON.

Oleg 


> On Fri, May 13, 2016 at 4:03 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> 
> > On Thu, 2016-05-12 at 13:17 -0700, Check Peck wrote:
> > > I have a library which is being used by customer and they are passing
> > > DataRequest object which has userid, various timeouts and some other
> > fields
> > > in it. Now I use this DataRequest object to make a URL and then I make an
> > > HTTP call using Apache HttpClient and my service returns back a JSON
> > > response which I use it to make a DataResponse object and return this
> > > DataResponse object back to them.
> > >
> > > Below is my DataClient class used by customer by passing DataRequest
> > object
> > > to it.
> > >
> > > https://gist.github.com/TechGeeky/250be2d9cdef3fa5107a17058a265d4c
> > >
> > > And here is DataFetcherTask class:
> > >
> > > https://gist.github.com/TechGeeky/c1b21025e0f81d222b792dedac0a817d
> > >
> > > Customer within our company will use my library like this as shown below
> > by
> > > using my factory in their code base -
> > >
> > > // if they are calling getSyncData() method
> > > DataResponse response =
> > > DataClientFactory.getInstance().getSyncData(key);
> > >
> > > // and if they want to call getAsyncData() method
> > > Future response =
> > > DataClientFactory.getInstance().getAsyncData(key);
> > >
> > > I am implementing "sync call as async + waiting" since I want to throttle
> > > them with the number of threads otherwise they can bombard our service
> > > without any control. My library will be used by lot of customers within
> > our
> > > company and their applications won't ever shutdown, they will keep
> > running
> > > always. The only thing will happen is their machines will get restarted,
> > > that's all.
> > >
> > > Is this the right way to use Apache HttpClient in production in
> > > multithreaded environment? Or there is any better/efficient way?
> > >
> > > I have to use various timeout values present in my DataRequest class in
> > my
> > > Apache HttpClient calls so that's  why I am creating RequestConfig and
> > > using it in my call method. I have simplified the code so that idea gets
> > > clear what I am trying to do.
> >
> > Looks OK for the most part. However I would consider conversion of
> > response entity to a string rather inefficient.
> >
> > Oleg
> >
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >
> >



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: using apache httpclient efficiently in multithreaded environment

[Oleg Kalnichevski]

On Thu, 2016-05-12 at 13:17 -0700, Check Peck wrote:
> I have a library which is being used by customer and they are passing
> DataRequest object which has userid, various timeouts and some other fields
> in it. Now I use this DataRequest object to make a URL and then I make an
> HTTP call using Apache HttpClient and my service returns back a JSON
> response which I use it to make a DataResponse object and return this
> DataResponse object back to them.
> 
> Below is my DataClient class used by customer by passing DataRequest object
> to it.
> 
> https://gist.github.com/TechGeeky/250be2d9cdef3fa5107a17058a265d4c
> 
> And here is DataFetcherTask class:
> 
> https://gist.github.com/TechGeeky/c1b21025e0f81d222b792dedac0a817d
> 
> Customer within our company will use my library like this as shown below by
> using my factory in their code base -
> 
> // if they are calling getSyncData() method
> DataResponse response =
> DataClientFactory.getInstance().getSyncData(key);
> 
> // and if they want to call getAsyncData() method
> Future response =
> DataClientFactory.getInstance().getAsyncData(key);
> 
> I am implementing "sync call as async + waiting" since I want to throttle
> them with the number of threads otherwise they can bombard our service
> without any control. My library will be used by lot of customers within our
> company and their applications won't ever shutdown, they will keep running
> always. The only thing will happen is their machines will get restarted,
> that's all.
> 
> Is this the right way to use Apache HttpClient in production in
> multithreaded environment? Or there is any better/efficient way?
> 
> I have to use various timeout values present in my DataRequest class in my
> Apache HttpClient calls so that's  why I am creating RequestConfig and
> using it in my call method. I have simplified the code so that idea gets
> clear what I am trying to do.

Looks OK for the most part. However I would consider conversion of
response entity to a string rather inefficient.

Oleg 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Apache HttpClient TCP Keep-Alive (socket keep-alive)

[Oleg Kalnichevski]

On Thu, 2016-05-12 at 13:46 +0100, Baratali Izmailov wrote:
> > Please consider upgrading. I am not entirely sure if HC 4.2 supports TCP 
> > keepalive
> setting.
> 
> Unfortunately, to upgrade HC we need to upgrade Spring to the latest
> version which requires Java 8. But, we cannot force our clients to use Java
> 8 yet.

There is no need to upgrade Spring. You can upgrade HC dependency to
something more recent without upgrading Spring itself. 

In this case however you should pass a custom instance of HttpClient to
ClientHttpRequestFactory

https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/http/client/HttpComponentsClientHttpRequestFactory.html#HttpComponentsClientHttpRequestFactory-org.apache.http.client.HttpClient-

> However, I don't see SO_KEEPALIVE parameter in the lastest Apache HC
> javadocs in "The following parameters can be used to customize the behavior
> of this class:" section:
> https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/DefaultHttpClient.html
> 
> Could you please show me example how to set SO_KEEPALIVE parameter in new
> versions of Apache HC?
> 

---
SocketConfig socketConfig = SocketConfig.custom()
.setSoKeepAlive(true)
.build();
CloseableHttpClient client = HttpClientBuilder.create()
.setDefaultSocketConfig(socketConfig)
.build();
ClientHttpRequestFactory clientfactory = new 
HttpComponentsClientHttpRequestFactory(client); 
---

Hope this helps

Oleg


> Thanks,
> Baratali Izmailov.
> 
> On 12 May 2016 at 09:53, Oleg Kalnichevski <ol...@apache.org> wrote:
> 
> > On Thu, 2016-05-12 at 09:23 +0100, Baratali Izmailov wrote:
> > > Hello. Thanks for the quick response.
> > >
> > > > Is there any way you can turn this into 2 requests?
> > > For now we cannot split this into 2 HTTP requests, because we have to
> > > change client-server protocol communication and re-implement some parts
> > of
> > > our application, which will take much time.
> > > I understand that it is not that effective to keep connection open too
> > > long, but it's only simple solution I see for now.
> > >
> > > > What version of HttpClient are you using? It looks like something
> > fairly
> > > old.
> > > It is 4.2.2. Actually I use Spring 3.2.1 httpInvoker which uses
> > > DefaultHttpClient.
> > >
> >
> > Please consider upgrading. I am not entirely sure if HC 4.2 supports TCP
> > keepalive setting.
> >
> > Oleg
> >
> > > > Using TCP keepalive wont help you with default OS settings, it would
> > not
> > > start to send them in the first 20 idle minutes.
> > > Can I configure Apache httpd server to do this (set keep-alive timeouts)?
> > >
> > > Thanks,
> > > Baratali.
> >
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >
> >



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Apache HttpClient TCP Keep-Alive (socket keep-alive)

[Oleg Kalnichevski]

On Thu, 2016-05-12 at 09:23 +0100, Baratali Izmailov wrote:
> Hello. Thanks for the quick response.
> 
> > Is there any way you can turn this into 2 requests?
> For now we cannot split this into 2 HTTP requests, because we have to
> change client-server protocol communication and re-implement some parts of
> our application, which will take much time.
> I understand that it is not that effective to keep connection open too
> long, but it's only simple solution I see for now.
> 
> > What version of HttpClient are you using? It looks like something fairly
> old.
> It is 4.2.2. Actually I use Spring 3.2.1 httpInvoker which uses
> DefaultHttpClient.
> 

Please consider upgrading. I am not entirely sure if HC 4.2 supports TCP
keepalive setting.

Oleg

> > Using TCP keepalive wont help you with default OS settings, it would not
> start to send them in the first 20 idle minutes.
> Can I configure Apache httpd server to do this (set keep-alive timeouts)?
> 
> Thanks,
> Baratali.



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Apache HttpClient TCP Keep-Alive (socket keep-alive)

[Oleg Kalnichevski]

On Wed, 2016-05-11 at 15:52 +0100, Baratali Izmailov wrote:
> Hello,
> 
> I have http request that takes too much time to be processed by the server
> (about 5 minutes). Because connection becomes idle for 5 minutes, proxy
> server shutdowns the connection. I'm trying to use TCP Keep-Alive in Apache
> DefaultHttpClient to make connection be alive for a long time (Not confuse
> TCP Keep-Alive with HTTP Keep-Alive that simply doesn't closes connection
> after response is sent).
> 
> Apache http core has following parameter SO_KEEPALIVE:
> http://hc.apache.org/httpcomponents-core-ga/httpcore/apidocs/org/apache/http/params/CoreConnectionPNames.html#SO_KEEPALIVE.
> However, due to DefaultHttpClient javadocs I can't customize client's
> behavior with that parameter:
> https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/impl/client/DefaultHttpClient.html
> .
> 
> Do you know how to make DefaultHttpClient use TCP Keep-Alive strategy?
> 
> 
> Thanks,
> 
> Baratali.

Baratali

What version of HttpClient are you using? It looks like something fairly
old. 

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HttpClient with a web service: Consistent data center switch-over

[Oleg Kalnichevski]

On Fri, 2016-05-06 at 07:17 -0700, Aaron Curley wrote:
> On 5/6/2016 5:15 AM, Oleg Kalnichevski wrote:
> > On Thu, 2016-05-05 at 07:12 -0700, Aaron Curley wrote:
> >> On 5/3/2016 9:49 AM, Oleg Kalnichevski wrote:
> >>> On Tue, 2016-05-03 at 09:15 -0700, Aaron Curley wrote:
> >>>> On 5/3/2016 8:22 AM, Oleg Kalnichevski wrote:
> >>>>> On Mon, 2016-05-02 at 21:28 -0700, Aaron Curley wrote:
> >>>>>> Hello all,
> >>>>>>
> >>>>>> I'm using HttpClient version 4.5.1 to make high-volume calls to an 
> >>>>>> internal highly-available web service that is hosted at multiple 
> >>>>>> geographic locations.  On average, the web service calls need to be 
> >>>>>> executed rather quickly; therefore, we are presently making heavy use 
> >>>>>> of HttpClient's connection pooling & reuse behavior (via 
> >>>>>> PoolingHttpClientConnectionManager) to alleviate the overhead (TCP and 
> >>>>>> TLS) incurred when setting up a new connection.  Our DNS records 
> >>>>>> direct clients of the service to the (geographically) nearest data 
> >>>>>> center. Should a data center experience problems, we update our DNS 
> >>>>>> records to redirect our clients to our remaining "good" site(s).
> >>>>>>
> >>>>>> In times of an outage, HttpClient appears to have automatically 
> >>>>>> failed-over to our alternate sites (IPs) pretty timely and 
> >>>>>> consistently; however, upon service restoration at the primary 
> >>>>>> data-center for a particular client node, HttpClient does not appear 
> >>>>>> to reliably switch back to calling our primary site.  (In such 
> >>>>>> instances, I have been able to confirm that our DNS records ARE 
> >>>>>> switching back in a timely manner.  The TTLs are set to an 
> >>>>>> appropriately short value.)
> >>>>>>
> >>>>>> My best guess is that the lack of consistent "switch back" to the 
> >>>>>> primary site is caused by HttpClient's connection pooling & reuse 
> >>>>>> behavior.  Because of our relatively high (and consistent) request 
> >>>>>> volume, creation of NEW connections is likely a rarity once a 
> >>>>>> particular client node is operating for a while; instead, as 
> >>>>>> previously noted, I would generally expect that connections in the 
> >>>>>> pool be re-used whenever possible.  Any re-used connections will still 
> >>>>>> be established with the alternate site(s), therefore, the client nodes 
> >>>>>> communicating with alternate sites would generally never (or only VERY 
> >>>>>> gradually) switch back to communicating with the primary site.  This 
> >>>>>> lines up with what I have observed; "switching back" seems to only 
> >>>>>> happen once request throughput drops sufficiently to allow most of our 
> >>>>>> client nodes' pooled connections to "time out" and be closed due to 
> >>>>>> inactivity (e.g. during overnight hours).
> >>>>>>
> >>>>>>
> >>>>>> I believe a reasonably "standard" way to solve this problem would be 
> >>>>>> to configure a maximum 'lifetime' of a connection in the connection 
> >>>>>> pool (e.g. 1 hour).  This lifetime would be enforced regardless of 
> >>>>>> whether or not the connection is idle or can otherwise be re-used.  On 
> >>>>>> first glance, the HttpClientBuilder.setConnectionTimeToLive() method 
> >>>>>> seemed ideal for this, but upon further investigation and review of 
> >>>>>> the HttpClient code base, this method appears to configure the maximum 
> >>>>>> TTL without introducing any element of randomness into each 
> >>>>>> connection's TTL.  As a result, I'm concerned that if I enable the 
> >>>>>> built-in TTL feature, my clients are likely to experience regular 
> >>>>>> performance "spikes" at the configured TTL interval.  (This would be 
> >>>>>> caused when most/all of the pooled connections expire simultaneously, 
> >>>>>> since they wer

Re: NoHttpResponseException received at startup

[Oleg Kalnichevski]

On Thu, 2016-05-05 at 14:46 +, Pete Keyes wrote:
> Update for the elapsed time for these calls that fail strictly at startup.
>  The stair-step 5/25/500... was strictly due to a pause we insert between
> retries.  All 4 tries fail within 5ms.
> 
> Given the configuration settings for CloseableHttpClient a failure within
> 5ms seems unlikely.  The trip to the F5 VIP and back would barely complete
> within 5ms.
> 
> 
> Any thoughts on how to stabilize this a bit at startup?  Or glean more
> information about why apache-hc would fail with NoHttpResponseException so
> quickly.  

The best course of action would be to capture IP packets with a traffic
analyzer such as Wireshark and to try to figure out why TCP connections
get reset.

Oleg 



> Stack trace for the error:
> org.apache.http.NoHttpResponseException:
> crm-asp-prod.starbucks.net:443 failed to respond
> 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe
> sponseParser.java:143)
> 
> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpRe
> sponseParser.java:57)
> 
> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.j
> ava:261)
> 
> org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(Def
> aultBHttpClientConnection.java:165)
> 
> org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:
> 167)
> 
> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestE
> xecutor.java:272)
> 
> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.ja
> va:124)
> 
> org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:2
> 71)
> 
> org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
> org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
> 
> org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
> 
> org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient
> .java:184)
> 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient
> .java:82)
> 
> 
> 
> ‹ 
> Pete Keyes
> 
> 
> 
> 
> 
> 
> On 5/4/16, 11:13 AM, "Pete Keyes"  wrote:
> 
> >We have an odd issue wherein at Tomcat(TomEE) startup apache-hc execution
> >requests throw a NoHttpResponseException within less than 1 second.
> >After a few minutes this behavior completely stops and isn¹t seen again
> >until the next time we stop and restart Tomcat(TomEE).
> >
> >Basic apache-hc configuration information:
> >  - single static CloseableHttpClient instance initialized at container
> >startup;
> >  - all requests managed via PoolingHttpClientConnectionManager
> >  - connection timeout 6000ms
> >  - read timeout 6000ms
> >  - we use 4 retries
> >  - the typical timing for each
> >HttpClient.execute(HttpUriRequest,HttpContext) during this startup period
> >is that they fail very fast:
> >  try-1: ~  5ms
> >  try-2: ~250ms
> >  try-3: ~500ms
> >  try-5: ~700ms
> >We measure the elapsed milli-seconds with respect to ³execute()².  It
> >seems very odd that execute() fails so fast given the apache-hc settings.
> > It isn¹t every execution request that fails.  But, a very high
> >percentage of them do exhibit this behavior during the initial ~5 minutes
> >after Tomcat(TomEE) begins dispatching inbound requests.
> >
> >The inbound HTTP request volume inbound is ~70 RPS(requests per second).
> >
> >After ~5 minutes this type of error never repeats.  We have logging in
> >place that verifies all the apache-hc components are fully instantiated
> >prior to any request being processed.
> >
> >Version information:
> >  apache-hc: 4.5.2
> >  java : 1.8r60
> >  tomcat   : 7.0.68
> >  tomee: 1.7.4
> >
> >
> >Any ideas at all given the minimal information that I have to provide are
> >greatly appreciated.
> >
> >‹
> >Pete
> >
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HttpClient with a web service: Consistent data center switch-over

[Oleg Kalnichevski]

On Thu, 2016-05-05 at 07:12 -0700, Aaron Curley wrote:
> On 5/3/2016 9:49 AM, Oleg Kalnichevski wrote:
> > On Tue, 2016-05-03 at 09:15 -0700, Aaron Curley wrote:
> >> On 5/3/2016 8:22 AM, Oleg Kalnichevski wrote:
> >>> On Mon, 2016-05-02 at 21:28 -0700, Aaron Curley wrote:
> >>>> Hello all,
> >>>>
> >>>> I'm using HttpClient version 4.5.1 to make high-volume calls to an 
> >>>> internal highly-available web service that is hosted at multiple 
> >>>> geographic locations.  On average, the web service calls need to be 
> >>>> executed rather quickly; therefore, we are presently making heavy use of 
> >>>> HttpClient's connection pooling & reuse behavior (via 
> >>>> PoolingHttpClientConnectionManager) to alleviate the overhead (TCP and 
> >>>> TLS) incurred when setting up a new connection.  Our DNS records direct 
> >>>> clients of the service to the (geographically) nearest data center. 
> >>>> Should a data center experience problems, we update our DNS records to 
> >>>> redirect our clients to our remaining "good" site(s).
> >>>>
> >>>> In times of an outage, HttpClient appears to have automatically 
> >>>> failed-over to our alternate sites (IPs) pretty timely and consistently; 
> >>>> however, upon service restoration at the primary data-center for a 
> >>>> particular client node, HttpClient does not appear to reliably switch 
> >>>> back to calling our primary site.  (In such instances, I have been able 
> >>>> to confirm that our DNS records ARE switching back in a timely manner.  
> >>>> The TTLs are set to an appropriately short value.)
> >>>>
> >>>> My best guess is that the lack of consistent "switch back" to the 
> >>>> primary site is caused by HttpClient's connection pooling & reuse 
> >>>> behavior.  Because of our relatively high (and consistent) request 
> >>>> volume, creation of NEW connections is likely a rarity once a particular 
> >>>> client node is operating for a while; instead, as previously noted, I 
> >>>> would generally expect that connections in the pool be re-used whenever 
> >>>> possible.  Any re-used connections will still be established with the 
> >>>> alternate site(s), therefore, the client nodes communicating with 
> >>>> alternate sites would generally never (or only VERY gradually) switch 
> >>>> back to communicating with the primary site.  This lines up with what I 
> >>>> have observed; "switching back" seems to only happen once request 
> >>>> throughput drops sufficiently to allow most of our client nodes' pooled 
> >>>> connections to "time out" and be closed due to inactivity (e.g. during 
> >>>> overnight hours).
> >>>>
> >>>>
> >>>> I believe a reasonably "standard" way to solve this problem would be to 
> >>>> configure a maximum 'lifetime' of a connection in the connection pool 
> >>>> (e.g. 1 hour).  This lifetime would be enforced regardless of whether or 
> >>>> not the connection is idle or can otherwise be re-used.  On first 
> >>>> glance, the HttpClientBuilder.setConnectionTimeToLive() method seemed 
> >>>> ideal for this, but upon further investigation and review of the 
> >>>> HttpClient code base, this method appears to configure the maximum TTL 
> >>>> without introducing any element of randomness into each connection's 
> >>>> TTL.  As a result, I'm concerned that if I enable the built-in TTL 
> >>>> feature, my clients are likely to experience regular performance 
> >>>> "spikes" at the configured TTL interval.  (This would be caused when 
> >>>> most/all of the pooled connections expire simultaneously, since they 
> >>>> were mostly all created at once, at application start-up.)
> >>>>
> >>>>
> >>>> Instead of using the built-in TTL limit setting, I considered overriding 
> >>>> the time to live using the available callbacks 
> >>>> (ConnectionKeepAliveStrategy, ConnectionReuseStrategy).  Unfortunately, 
> >>>> this approach appears to be infeasible because the parameters to those 
> >>>> callbacks do not have access to the underlying connection object; there 
> >>>> is no "key&q

Re: HttpClient with a web service: Consistent data center switch-over

[Oleg Kalnichevski]

On Tue, 2016-05-03 at 09:15 -0700, Aaron Curley wrote:
> On 5/3/2016 8:22 AM, Oleg Kalnichevski wrote:
> > On Mon, 2016-05-02 at 21:28 -0700, Aaron Curley wrote:
> >> Hello all,
> >>
> >> I'm using HttpClient version 4.5.1 to make high-volume calls to an 
> >> internal highly-available web service that is hosted at multiple 
> >> geographic locations.  On average, the web service calls need to be 
> >> executed rather quickly; therefore, we are presently making heavy use of 
> >> HttpClient's connection pooling & reuse behavior (via 
> >> PoolingHttpClientConnectionManager) to alleviate the overhead (TCP and 
> >> TLS) incurred when setting up a new connection.  Our DNS records direct 
> >> clients of the service to the (geographically) nearest data center. Should 
> >> a data center experience problems, we update our DNS records to redirect 
> >> our clients to our remaining "good" site(s).
> >>
> >> In times of an outage, HttpClient appears to have automatically 
> >> failed-over to our alternate sites (IPs) pretty timely and consistently; 
> >> however, upon service restoration at the primary data-center for a 
> >> particular client node, HttpClient does not appear to reliably switch back 
> >> to calling our primary site.  (In such instances, I have been able to 
> >> confirm that our DNS records ARE switching back in a timely manner.  The 
> >> TTLs are set to an appropriately short value.)
> >>
> >> My best guess is that the lack of consistent "switch back" to the primary 
> >> site is caused by HttpClient's connection pooling & reuse behavior.  
> >> Because of our relatively high (and consistent) request volume, creation 
> >> of NEW connections is likely a rarity once a particular client node is 
> >> operating for a while; instead, as previously noted, I would generally 
> >> expect that connections in the pool be re-used whenever possible.  Any 
> >> re-used connections will still be established with the alternate site(s), 
> >> therefore, the client nodes communicating with alternate sites would 
> >> generally never (or only VERY gradually) switch back to communicating with 
> >> the primary site.  This lines up with what I have observed; "switching 
> >> back" seems to only happen once request throughput drops sufficiently to 
> >> allow most of our client nodes' pooled connections to "time out" and be 
> >> closed due to inactivity (e.g. during overnight hours).
> >>
> >>
> >> I believe a reasonably "standard" way to solve this problem would be to 
> >> configure a maximum 'lifetime' of a connection in the connection pool 
> >> (e.g. 1 hour).  This lifetime would be enforced regardless of whether or 
> >> not the connection is idle or can otherwise be re-used.  On first glance, 
> >> the HttpClientBuilder.setConnectionTimeToLive() method seemed ideal for 
> >> this, but upon further investigation and review of the HttpClient code 
> >> base, this method appears to configure the maximum TTL without introducing 
> >> any element of randomness into each connection's TTL.  As a result, I'm 
> >> concerned that if I enable the built-in TTL feature, my clients are likely 
> >> to experience regular performance "spikes" at the configured TTL interval. 
> >>  (This would be caused when most/all of the pooled connections expire 
> >> simultaneously, since they were mostly all created at once, at application 
> >> start-up.)
> >>
> >>
> >> Instead of using the built-in TTL limit setting, I considered overriding 
> >> the time to live using the available callbacks 
> >> (ConnectionKeepAliveStrategy, ConnectionReuseStrategy).  Unfortunately, 
> >> this approach appears to be infeasible because the parameters to those 
> >> callbacks do not have access to the underlying connection object; there is 
> >> no "key" that can be used to look up a connection's lifetime (i.e. in a 
> >> ConcurrentHashMap) such that a decision could be made about whether to 
> >> close or retain the connection.  I also took a look at the various places 
> >> that I could try to override the default connection pooling behavior (e.g. 
> >> MainClientExec, HttpClientConnectionManager). HttpClientConnectionManager 
> >> appears to be the best bet (specifically, 
> >> HttpClientConnectionManager.releaseConnection() would have to be 
> >> overridden), but this would requi

Re: HttpClient with a web service: Consistent data center switch-over

[Oleg Kalnichevski]

On Mon, 2016-05-02 at 21:28 -0700, Aaron Curley wrote:
> Hello all,
> 
> I'm using HttpClient version 4.5.1 to make high-volume calls to an internal 
> highly-available web service that is hosted at multiple geographic locations. 
>  On average, the web service calls need to be executed rather quickly; 
> therefore, we are presently making heavy use of HttpClient's connection 
> pooling & reuse behavior (via PoolingHttpClientConnectionManager) to 
> alleviate the overhead (TCP and TLS) incurred when setting up a new 
> connection.  Our DNS records direct clients of the service to the 
> (geographically) nearest data center. Should a data center experience 
> problems, we update our DNS records to redirect our clients to our remaining 
> "good" site(s).
> 
> In times of an outage, HttpClient appears to have automatically failed-over 
> to our alternate sites (IPs) pretty timely and consistently; however, upon 
> service restoration at the primary data-center for a particular client node, 
> HttpClient does not appear to reliably switch back to calling our primary 
> site.  (In such instances, I have been able to confirm that our DNS records 
> ARE switching back in a timely manner.  The TTLs are set to an appropriately 
> short value.)
> 
> My best guess is that the lack of consistent "switch back" to the primary 
> site is caused by HttpClient's connection pooling & reuse behavior.  Because 
> of our relatively high (and consistent) request volume, creation of NEW 
> connections is likely a rarity once a particular client node is operating for 
> a while; instead, as previously noted, I would generally expect that 
> connections in the pool be re-used whenever possible.  Any re-used 
> connections will still be established with the alternate site(s), therefore, 
> the client nodes communicating with alternate sites would generally never (or 
> only VERY gradually) switch back to communicating with the primary site.  
> This lines up with what I have observed; "switching back" seems to only 
> happen once request throughput drops sufficiently to allow most of our client 
> nodes' pooled connections to "time out" and be closed due to inactivity (e.g. 
> during overnight hours).
> 
> 
> I believe a reasonably "standard" way to solve this problem would be to 
> configure a maximum 'lifetime' of a connection in the connection pool (e.g. 1 
> hour).  This lifetime would be enforced regardless of whether or not the 
> connection is idle or can otherwise be re-used.  On first glance, the 
> HttpClientBuilder.setConnectionTimeToLive() method seemed ideal for this, but 
> upon further investigation and review of the HttpClient code base, this 
> method appears to configure the maximum TTL without introducing any element 
> of randomness into each connection's TTL.  As a result, I'm concerned that if 
> I enable the built-in TTL feature, my clients are likely to experience 
> regular performance "spikes" at the configured TTL interval.  (This would be 
> caused when most/all of the pooled connections expire simultaneously, since 
> they were mostly all created at once, at application start-up.)
> 
> 
> Instead of using the built-in TTL limit setting, I considered overriding the 
> time to live using the available callbacks (ConnectionKeepAliveStrategy, 
> ConnectionReuseStrategy).  Unfortunately, this approach appears to be 
> infeasible because the parameters to those callbacks do not have access to 
> the underlying connection object; there is no "key" that can be used to look 
> up a connection's lifetime (i.e. in a ConcurrentHashMap) such that a decision 
> could be made about whether to close or retain the connection.  I also took a 
> look at the various places that I could try to override the default 
> connection pooling behavior (e.g. MainClientExec, 
> HttpClientConnectionManager). HttpClientConnectionManager appears to be the 
> best bet (specifically, HttpClientConnectionManager.releaseConnection() would 
> have to be overridden), but this would require duplication of the existing 
> releaseConnection() code with slight modifications in the overriding class. 
> This seems brittle.
> 
> 
> Can anyone think of a way that I could implement this (cleanly) with 
> HttpClient 4.x?  Maybe I missed something?
> 
> If not, I would be happy to open a JIRA for a possible HttpClient enhancement 
> to enable such a feature.  If people are open to this idea, I was generally 
> thinking that adding a more generic callback might be the best approach 
> (since my use case may not be other people's use cases), but I could also be 
> convinced to make the enhancement request specifically support a connection 
> expiration "window" for the TTL feature instead of a specific "hard-limit".  
> Any thoughts on this?
> 
> 
> Thanks in advance (and sorry for the long email)!
> 

Hi Aaron

PoolingHttpClientConnectionManager does not pro-actively evict expired
connections by default. I think it is unlikely that connections with a

Re: ResponseContentEncoding behaviour in HttpClient 4.5.2 and 5.0

[Oleg Kalnichevski]

On Sun, 2016-05-01 at 17:41 +0200, Philippe Mouawad wrote:
> Hi,
> Because during load testing you want:
> - to simulate browser behaviour, in a lot of case compression will be
> enabled
> - user will want to check server returns compressed response and also have
> information about response sizes compressed which is the real traffic
> taking place
> - but then user will want to check data in the responses using assertion,
> to do that we need responses to be uncompressed
> 

Right. There is nothing that should stop one from disabling automatic
content decompression (and thus response message modification) and
manually decompressing response entity as appropriate.

Oleg   

> Regards
> 
> On Sunday, May 1, 2016, Oleg Kalnichevski <ol...@apache.org> wrote:
> 
> > On Sun, 2016-05-01 at 17:22 +0200, Philippe Mouawad wrote:
> > > Hi Oleg,
> > > I understand, but in our particular case users want to have information
> > > about original response headers.
> > >
> >
> > Why do not just disable content decompression if it is not needed?
> >
> > Oleg
> >
> > > Does this way of implementing it look ok to you:
> > >
> > >- https://bz.apache.org/bugzilla/attachment.cgi?id=33817=diff
> > >
> > > Or can it break something in HttpClient ?
> > >
> > > Thanks
> > > Regards
> > >
> > > On Sun, May 1, 2016 at 5:18 PM, Oleg Kalnichevski <ol...@apache.org
> > <javascript:;>> wrote:
> > >
> > > > On Sun, 2016-05-01 at 16:08 +0200, Philippe Mouawad wrote:
> > > > > Hello,
> > > > > We have a regression report in JMeter 3.0 due to what seems to be a
> > new
> > > > > behaviour of HttpClient 4.5.2, introduced  on Feb 25, 2014 by:
> > > > >
> > > > >-
> > > > >
> > > >
> > https://github.com/apache/httpclient/commit/5d11a3e751fe0c02a7a4539d3436b06e0be35876#diff-c54e3439558bee75dd7e2953280a7e08
> > > > >
> > > > >
> > > > > As per following code:
> > > > >
> > > > > -
> > > >
> > https://github.com/apache/httpclient/blob/4.5.x/httpclient/src/main/java/org/apache/http/client/protocol/ResponseContentEncoding.java#L142
> > > > >
> > > > >
> > > > > When uncompressing HttpClient removes 3 headers:
> > > > > - Content-Length
> > > > > - Content-Encoding
> > > > > - Content-MD5
> > > > >
> > > >
> > > > This behavior was introduced in 4.2 (4 years ago). See HTTPCLIENT-1164.
> > > > https://issues.apache.org/jira/browse/HTTPCLIENT-1164
> > > >
> > > > > So in JMeter 3.0, we lose these 3 headers compared to 2.13:
> > > > > - https://bz.apache.org/bugzilla/show_bug.cgi?id=59401
> > > > >
> > > > > Is there a reason for removing them ?
> > > > >
> > > >
> > > > Automatic decompression invalidates these headers. Decompressed content
> > > > stream no longer has the same length, encoding and MD5 checksum as
> > > > declared in the original response message.
> > > >
> > > > Oleg
> > > >
> > > >
> > > > -
> > > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > <javascript:;>
> > > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> > <javascript:;>
> > > >
> > > >
> > >
> > >
> >
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > <javascript:;>
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> > <javascript:;>
> >
> >
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: ResponseContentEncoding behaviour in HttpClient 4.5.2 and 5.0

[Oleg Kalnichevski]

On Sun, 2016-05-01 at 17:22 +0200, Philippe Mouawad wrote:
> Hi Oleg,
> I understand, but in our particular case users want to have information
> about original response headers.
> 

Why do not just disable content decompression if it is not needed?

Oleg

> Does this way of implementing it look ok to you:
> 
>- https://bz.apache.org/bugzilla/attachment.cgi?id=33817=diff
> 
> Or can it break something in HttpClient ?
> 
> Thanks
> Regards
> 
> On Sun, May 1, 2016 at 5:18 PM, Oleg Kalnichevski <ol...@apache.org> wrote:
> 
> > On Sun, 2016-05-01 at 16:08 +0200, Philippe Mouawad wrote:
> > > Hello,
> > > We have a regression report in JMeter 3.0 due to what seems to be a new
> > > behaviour of HttpClient 4.5.2, introduced  on Feb 25, 2014 by:
> > >
> > >-
> > >
> > https://github.com/apache/httpclient/commit/5d11a3e751fe0c02a7a4539d3436b06e0be35876#diff-c54e3439558bee75dd7e2953280a7e08
> > >
> > >
> > > As per following code:
> > >
> > > -
> > https://github.com/apache/httpclient/blob/4.5.x/httpclient/src/main/java/org/apache/http/client/protocol/ResponseContentEncoding.java#L142
> > >
> > >
> > > When uncompressing HttpClient removes 3 headers:
> > > - Content-Length
> > > - Content-Encoding
> > > - Content-MD5
> > >
> >
> > This behavior was introduced in 4.2 (4 years ago). See HTTPCLIENT-1164.
> > https://issues.apache.org/jira/browse/HTTPCLIENT-1164
> >
> > > So in JMeter 3.0, we lose these 3 headers compared to 2.13:
> > > - https://bz.apache.org/bugzilla/show_bug.cgi?id=59401
> > >
> > > Is there a reason for removing them ?
> > >
> >
> > Automatic decompression invalidates these headers. Decompressed content
> > stream no longer has the same length, encoding and MD5 checksum as
> > declared in the original response message.
> >
> > Oleg
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >
> >
> 
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: ResponseContentEncoding behaviour in HttpClient 4.5.2 and 5.0

[Oleg Kalnichevski]

On Sun, 2016-05-01 at 16:08 +0200, Philippe Mouawad wrote:
> Hello,
> We have a regression report in JMeter 3.0 due to what seems to be a new
> behaviour of HttpClient 4.5.2, introduced  on Feb 25, 2014 by:
> 
>-
>
> https://github.com/apache/httpclient/commit/5d11a3e751fe0c02a7a4539d3436b06e0be35876#diff-c54e3439558bee75dd7e2953280a7e08
> 
> 
> As per following code:
> 
> - 
> https://github.com/apache/httpclient/blob/4.5.x/httpclient/src/main/java/org/apache/http/client/protocol/ResponseContentEncoding.java#L142
> 
> 
> When uncompressing HttpClient removes 3 headers:
> - Content-Length
> - Content-Encoding
> - Content-MD5
> 

This behavior was introduced in 4.2 (4 years ago). See HTTPCLIENT-1164. 
https://issues.apache.org/jira/browse/HTTPCLIENT-1164

> So in JMeter 3.0, we lose these 3 headers compared to 2.13:
> - https://bz.apache.org/bugzilla/show_bug.cgi?id=59401
> 
> Is there a reason for removing them ?
> 

Automatic decompression invalidates these headers. Decompressed content
stream no longer has the same length, encoding and MD5 checksum as
declared in the original response message.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Question about ThreadSafeClientConnManager change maxTotal on fly

[Oleg Kalnichevski]

On Fri, 2016-04-29 at 14:51 +0200, NGUYEN-VIET Quang wrote:
> Hello,
> 
> We use httpclient 4.1. ( we plan to upgrade soon)
> 
> Juste a simple question about how connection pool behave  when we change
> maxTotal on fly, without releasing connetions in use. ( with
> releaseConnection() for instance )
> 
> If i increase the maxTotal size, I guess that it's not a problem, it simply
> allocate more possible connection in pool.
> 
> But how about when we decrease the pool's size and all connexions are
> pending or occupied, do they become some kind of zombi connection and we
> lost control on these connections.
> 

No, they are not. Those connections will still be kept in the pool and
re-used until they expire. The connection manager however is not going
to allocate new connections until the total number of connections
becomes less or equal to maxTotal.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Pooling Connection Manager Issue

[Oleg Kalnichevski]

On Tue, 2016-04-26 at 15:37 +, Gordon Ross wrote:
> > On 26 Apr 2016, at 15:55, Oleg Kalnichevski <ol...@apache.org> wrote:
> > 
> > Could you please make the entire log available?
> 
> What logs do you want? Currently, I’ve got set:
> 
> Error: org.apache.http.wire
> Debug: org.apache.http.impl.conn, org.apache.http.impl.client, 
> org.apache.http.client
> 
> Would you want any more?
> 

All but org.apache.http.wire

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Pooling Connection Manager Issue

[Oleg Kalnichevski]

On Tue, 2016-04-26 at 11:27 +, Gordon Ross wrote:
> I’m using HTTP Client 4.3.5 with Java 1.7 on Ubuntu Linux.
> 
> I’m having problems where the pooling connection manager suddenly claims to 
> have no routes to a host.
> 
> Usually, in my logs I see:
> 
> 2016-04-26 11:13:08,588 [TP-Processor5] DEBUG 
> conn.PoolingHttpClientConnectionManager - Connection request: [route: 
> {}->http://example.com:80][total kept alive: 1; route allocated: 1 of 50; 
> total allocated: 1 of 200]
> 2016-04-26 11:13:08,588 [TP-Processor5] DEBUG 
> conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: 
> {}->http://example.com:80][total kept alive: 0; route allocated: 1 of 50; 
> total allocated: 1 of 200]
> 2016-04-26 11:13:08,588 [TP-Processor5] DEBUG 
> conn.DefaultManagedHttpClientConnection - http-outgoing-1: Close connection
> 
> But occasionally, I see:
> 
> 2016-04-26 10:54:05,287 [TP-Processor3] DEBUG 
> conn.PoolingHttpClientConnectionManager - Connection request: [route: 
> {}->http://example.com:80][total kept alive: 0; route allocated: 50 of 50; 
> total allocated: 50 of 200]
> 
> i.e. suddenly the routes allocated has jumped from 1 of 50 to 50 of 50.
> 
> What’s going on? If I had some form of leak, surely I’d see the routes 
> allocated slowly increasing? (2 of 50, 3 of 50, etc..)
> 
> In my application, I’m closing the HttpResponse InputStream to indicate I’ve 
> finished with the connection. Should I be doing anything else/more?
> 

No, you should not. 

Could you please make the entire log available?

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: httpasyncclient BasicAsyncRequestProducer prevents HttpRequestInterceptor from replacing HttpEntity of the request

[Oleg Kalnichevski]

On Wed, 2016-04-20 at 15:03 +, Sven Zethelius wrote:
> I am trying to use a HttpRequestInterceptor to adapt a request sent to a 
> org.apache.http.impl.nio.client.CloseableHttpAsyncClient.  I want to do 
> request body compression (e.g. Content-Encoding: gzip for HttpRequest, not 
> Accept-Encoding) following the example here: 
> https://hc.apache.org/httpcomponents-client-4.2.x/httpclient/examples/org/apache/http/examples/client/ClientGZipContentCompression.java
>  .  The example is for response compression, but it looked like I could do 
> the same thing to do request compression, and I need to handle response 
> compression too.
> 
> After much debugging I realized the HttpRequestInterceptor would allow me to 
> change the headers, but not the HttpEntity that was sent.  The entity sent is 
> always the original one passed into the AsyncHttpClient.  On the server, I 
> see the "Content-Type: gzip" and a "Content-Length" header for the gzipped 
> length (Content-Length was set by the internal interceptors), but the bytes 
> sent are the original values from the NByteArrayEntity, which leads to 
> exceptions since Content-Length doesn't match actual length after GZIP.
> 
> I tracked it down to the constructor for 
> org.apache.http.nio.protocol.BasicAsyncRequestProducer.  The 
> BasicAsyncRequestProducer is created before the interceptors run.  In the 
> producer constructor, the entity from the request is stored as the producer.  
> This occurs before the HttpRequestInterceptor has a chance to manipulate the 
> entity.
> 
> My idea for a patch would be to delay load the producer field until after the 
> interceptors have run.  Looking at the class, that would have to be in 
> produceContent, before it checks for producer==null. Since the request is 
> also a member, this would allow the producer to be set based on what's in the 
> request when we actually produce the output, rather than what was in it when 
> the whole processing chain started.
> 
> Is patching the BasicAsyncRequestProducer the only way to do this or is there 
> a simpler way that I am missing?
> 
> 

Hi Sven

Building a custom request producer is currently the only way to
implement content compression. 

HttpAsyncClient presently does not support transparent content
compression. This feature is tentatively planned for 5.0.

Oleg  



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: HttpClientBuilder does not use system properties for proxy

[Oleg Kalnichevski]

On Fri, 2016-04-15 at 11:59 +0200, Philip Kahle wrote:
> Hi,
> 
> I am instantiating a HttpClientBuilder like this and tried to make it
> use the Java system properties for a proxy server via useSystemProperties():
> 
> protected static HttpClientBuilder builder;
> ...
> PoolingHttpClientConnectionManager cm = new
> PoolingHttpClientConnectionManager();
> builder =
> HttpClients.custom().setConnectionManager(cm).useSystemProperties();
> ...
> CloseableHttpClient httpClient = builder.build();
> 
> However it does not set the proxy (all hostname, port, username and
> password have been set at this point).
> 
> I don't see how to get a SystemDefaultHttpClient at this point, which to
> my understanding should be used at this point, and I did not find the
> correct solution besides passing all the data manually, e.g. like this:
> 
> NTCredentials ntCreds = new NTCredentials(user, password,
> localMachineName, domainName );
> CredentialsProvider credsProvider = new BasicCredentialsProvider();
> setCredentials( new AuthScope(host,port), ntCreds);
> setDefaultCredentialsProvider(credsProvider);
> setProxyAuthenticationStrategy(new ProxyAuthenticationStrategy());
> 
> However, here localMachineName and domainName have to be guessed somehow...
> 
> What is the correct solution to set a proxy in this setting?
> 

Do not set fully initialized connection manager. It makes all connection
management settings have no effect.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Proxy Authentication Strategy

[Oleg Kalnichevski]

On Thu, 2016-04-07 at 17:54 +, Mark A. Claassen wrote:
> Apache HTTPClient 4.4.1
> 
> I was doing some more tests with proxy authentication and I ran into 
> something I didn't expect.  AuthenticationStrategyImpl goes though the list 
> of schemes ( SPNEGO, NTLM, ...) to find the ones that have "authentications". 
>  However, this goes through all of them and does not stop after it gets one.
> 
> I created a CredentialsProvider that prompts the user for a username and 
> password for certain schemes.  I was surprised to find out that this made my 
> password dialog appeared too often.  I configured my proxy server to allow 
> for NTLM and DIGEST authentication schemes.  Then using my HttpClient, I 
> thought that if NTLM was higher in priority, and there were credentials for 
> that, it would not continue to ask for credentials for the DIGEST scheme.  
> However, it always goes through the whole list.  Is this supposed to be this 
> way? 

It is. AuthenticationStrategyImpl does so to identify all auth schemes
that can potentially be used to respond to the authentication challenges
sent by the server. Naturally it can only do so if there are credentials
matching the actual authentication scope.  

>  
> 
> Does this make sense with how proxy servers work?  In my case, configuring 
> the proxy server to allow NTML and DIGEST authentication means that it will 
> allow access if *either* is specified, and does not require both.
> 
> It would be nice if AuthenticationStrategyImpl would stop searching once it 
> found something, however there does not seem to be an easy way to handle 
> this.  This makes me think I might be doing something wrong.  I thought a 
> custom CredentialProvider would be the correct place to code in a password 
> dialog.  But maybe there is a better "hook" I can use to bring up the dialog, 
> and then just put the information in the CredentialsProvider.  Is there a 
> better place to put this code?
> 
> It seems the only way to handle this is to re-write 
> AuthenticationStrategyImpl, adding a single line so it will stop after the 
> first non-null credentials are returned.
> 

There should be nothing stopping you from returning the same creds
multiple times. You just need to make your implementation of
CredentialsProvider a little smarter.

Oleg



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: Why no setServiceUnavailableRetryHandler in the AsyncClientBuilder?

[Oleg Kalnichevski]

On Tue, 2016-03-29 at 14:12 -0400, Benson Margulies wrote:
> Please see subject. Is there some other way around it?
> 

Hi Benson

HttpAsyncClient presently can not automatically retry requests. One has
to re-submit failed requests manually.

Oleg


-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org