On 25 Sep 2013, at 2:00 PM, Oleg Kalnichevski <[email protected]> wrote:
> I do not have any suggestions other than taking note of problematic
> hosts, monitoring them closely and probably applying more aggressive
> timeout parameters to those hosts by dynamically reducing socket timeout
> if throughput drops below a certain limit.
We did it, and it solved part of the problems we're having.
The fact is that there are parts of the stack we cannot control. This threads
are stuck since ~12h (we have about 30 of the same kinds over 6000 overall
fetching threads on 3 machines):
"FetchingThread-151" prio=10 tid=0x00007f0cc4e8f000 nid=0x256b runnable
[0x00007f082dfdd000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:442)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:554)
at sun.security.ssl.InputRecord.read(InputRecord.java:509)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:927)
- locked <0x0000000576497a70> (a java.lang.Object)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
- locked <0x0000000576497a30> (a java.lang.Object)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1339)
at
sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1323)
at
org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:262)
at
org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:118)
at
org.apache.http.impl.conn.BasicHttpClientConnectionManager.connect(BasicHttpClientConnectionManager.java:318)
at
org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:357)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:218)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
at
it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
"FetchingThread-1856" prio=10 tid=0x00007f4a7db9f000 nid=0x26ef runnable
[0x00007f41b8b0e000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at
org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:136)
at
org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:152)
at
org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:270)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260)
at
org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:161)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123)
at
org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:253)
at
org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:194)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:85)
at
org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at
org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:186)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:214)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:185)
at it.unimi.di.law.bubing.util.FetchData.fetch(FetchData.java:308)
at
it.unimi.di.law.bubing.frontier.FetchingThread.run(FetchingThread.java:235)
Suggestions? Should we give an interrupt after a certain amount of time has
elapsed?
Ciao,
seba
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
