Have you considered using the PoolingClientConnectionManager? I see the BasicHttpClientConnectionManager in the stacktrace.
It wouldn't solve the root issue, but if your application is making requests to the same servers over and over again you might be able to save some handshaking and latency by reusing HTTP connections. On Oct 5, 2013, at 5:39 AM, Sebastiano Vigna <[email protected]> wrote: > On 25 Sep 2013, at 2:00 PM, Oleg Kalnichevski <[email protected]> wrote: > >> I do not have any suggestions other than taking note of problematic >> hosts, monitoring them closely and probably applying more aggressive >> timeout parameters to those hosts by dynamically reducing socket timeout >> if throughput drops below a certain limit. > > > We did it, and it solved part of the problems we're having. > > The fact is that there are parts of the stack we cannot control. This threads > are stuck since ~12h (we have about 30 of the same kinds over 6000 overall > fetching threads on 3 machines): > > "FetchingThread-151" prio=10 tid=0x00007f0cc4e8f000 nid=0x256b runnable > [0x00007f082dfdd000] > java.lang.Thread.State: RUNNABLE > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:152) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at sun.security.ssl.InputRecord.readFully(InputRecord.java:442) > at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554) > at sun.security.ssl.InputRecord.read(InputRecord.java:509) > at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927) > - locked <0x0000000576497a70> (a java.lang.Object) > at > sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312) > - locked <0x0000000576497a30> (a java.lang.Object) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339) > at > sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323) > at > org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:262) > at > org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118) > at > org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:318) > at > org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357) > at > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218) > at > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194) > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85) > at > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) > at > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185) > at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308) > at > it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235) > > > > "FetchingThread-1856" prio=10 tid=0x00007f4a7db9f000 nid=0x26ef runnable > [0x00007f41b8b0e000] > java.lang.Thread.State: RUNNABLE > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:152) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at > org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136) > at > org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152) > at > org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) > at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) > at > org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161) > at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) > at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) > at > org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:253) > at > org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194) > at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85) > at > org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) > at > org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214) > at > org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185) > at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308) > at > it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235) > > Suggestions? Should we give an interrupt after a certain amount of time has > elapsed? > > Ciao, > > seba > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
