Hi! I'm using httpclient to consume a rest service. Everything was (is)
working like a charm but we had an episode last week:
- After a short outage in our network (blaming this outage is an assumption
I can't confirm) we got 16 threads stuck in our app server on:
"[STUCK] ExecuteThread: '45' for queue: 'weblogic.kernel.Default
(self-tuning)'" #29841 daemon prio=1 os_prio=0 tid=0x00007f07ac1e1800
rc="DOMAIN" ac=2 nid=0x8c6d runnable [0x00007f077c1de000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at
java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at
java.net.SocketInputStream.read(SocketInputStream.java:170)
at
java.net.SocketInputStream.read(SocketInputStream.java:141)
at
sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
- locked <0x0000000093d38470> (a java.lang.Object)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
- locked <0x0000000093d38530> (a java.lang.Object)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394)
at
org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.upgrade(DefaultHttpClientConnectionOperator.java:185)
at
org.apache.http.impl.conn.PoolingHttpClientConnectionManager.upgrade(PoolingHttpClientConnectionManager.java:369)
at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:415)
- 16 was the value we used for defaultMaxPerRoute
(org.apache.http.impl.conn.PoolingHttpClientConnectionManager) so it makes
sense that we got 16
threads stuck.
- All subsequent threads trying to get a connection got an exception (after
X seconds with X equal to our value for connectionRequestTimeout) as
expected.
- I think I've narrowed the problem down to this part of the code
(MainClientExec class)
if (!managedConn.isOpen()) {
this.log.debug("Opening connection " + route);
try {
establishRoute(proxyAuthState, managedConn,
route, request, context);
} catch (final TunnelRefusedException ex) {
if (this.log.isDebugEnabled()) {
this.log.debug(ex.getMessage());
}
response = ex.getResponse();
break;
}
}
final int timeout = config.getSocketTimeout();
if (timeout >= 0) {
managedConn.setSocketTimeout(timeout);
}
It seems like socket timeout is set after establishRoute so the SSL
handshake isn't affected by this setting.
I am right?
Could the connection timeout be passed down to this.connManager.upgrade
(inside establishRoute) in order to use
it as socket timeout while establishing the SSL connection?
Versions:
httpclient-4.5.jar
httpcore-4.4.1.jar
httpcore-nio-4.4.4.jar
Thank you very much!
Best regards
