HTTPClient 4.3-beta2: Failure to re-use connections with Mutual SSL Auth

[Erik Pilz]

We're migrating an old application from Java's HttpsUrlConnection to
HttpClient. After running a performance test we found a high cost around
connection acquisition. A bit of debugging later led me to find that we're
not getting the full benefit of pooled connections. What I eventually found
is that I have to create a HttpClientContext and populate the userToken
with some object and use HttpClient#execute(method, context). This wasn't
obvious to me; is this intentional or did I just find a workaround to a bug?

Here's the background of how I got here.

Quick facts:

   - We're using HttpClient 4.3-beta2
   - PoolingHttpClientConnectionManager is configured with a max of 9
   connections and a 30 second idle
   - We're using Mutual SSL Auth with the server
   - Functionally everything performs as expected -- we can successfully
   send and receive messages
   - Our code simply uses httpClient.execute(httpPost) to make the call

What I've found is that eventually the code
reaches org.apache.http.pool.RouteSpecificPool#getFree(Object state) to get
a connection, but that method always returns null. Here's the code for ease
of reference:

    public E getFree(final Object state) {
        if (!this.available.isEmpty()) {
            if (state != null) {
                final Iterator<E> it = this.available.iterator();
                while (it.hasNext()) {
                    final E entry = it.next();
                    if (state.equals(entry.getState())) {
                        it.remove();
                        this.leased.add(entry);
                        return entry;
                    }
                }
            }
            final Iterator<E> it = this.available.iterator();
            while (it.hasNext()) {
                final E entry = it.next();
                if (entry.getState() == null) {
                    it.remove();
                    this.leased.add(entry);
                    return entry;
                }
            }
        }
        return null;
    }

The "state" parameter is always null when this method is called. The
members of the available pool, however, have their state set to the
principal (the cert) of the SSLSession. Consequently this method always
returns null and connections in the pool aren't re-used.

Working backwards I found that the state parameter originates from the
userToken in MainClientExec#execute(
            final HttpRoute route,
            final HttpRequestWrapper request,
            final HttpClientContext context,
            final HttpExecutionAware execAware)

Here's the relevant snippet:

        Object userToken = context.getUserToken();
        final ConnectionRequest connRequest =
connManager.requestConnection(route, userToken);

Calling HttpClient#execute(method) will create a context whose userToken is
null. To work around the behavior it's instead necessary to create an
HttpClientContext and populate it with a userToken of your own (it can be
any object); this will then be used as the state in the available
connections instead of the principal from the SSLSession; you then use
HttpClient#execute(method, context) in place of HttpClient#execute(method).

Thanks,
Erik