On Tue, 2013-01-29 at 14:59 -0500, Mark Claassen wrote: > CLUE! > > I opened up the cipher suites on Tomecat like I mentioned and then made my > own socketfactory (limiting the cipher suites) as I saw in an example. Now > when I run the program it dies at the same place every time...on the 101st > connection. This has to be significant in some way...although I don't know > how. > > I about doubled the size of the message I send, and it makes no difference. > > I put the iteration count in the HttpHeader so I could track it in the Tomcat > access log. Tomcat shows that it receives this message (message 100) and > that it thinks it succeeded (status == 200) > > The SocketTimeoutException I noticed before is everywhere in the logs. This > is from the StaleConnectionCheck and is likely irrelevant. > >
Tomcat by default closes persistent connections once it has been used to serve 100 requests. See 'maxKeepAliveRequests' parameter: http://tomcat.apache.org/tomcat-7.0-doc/config/http.html Try setting this value to -1 and see if that makes the problem go away. Oleg > > > -----Original Message----- > From: Mark Claassen [mailto:[email protected]] > Sent: Tuesday, January 29, 2013 2:01 PM > To: 'HttpClient User Discussion' > Subject: RE: Connection Reset errors > > I saw some things on the web with this error, but those seemed to be errors > on startup because particular protocols where not enabled. I changed my > Tomcat configuration to use SSL instead of TLS, and removed the cipherSuites > attribute. Similar result. However I saw the below in the debug output. > > What is the socket timeout doing there, and why is there a > SocketTimeoutException when this is all occurring in the same second. > > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire > FINE: << " <p>If you page is not redirected, click <a > href="JViewer/">here</a>[\n]" > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire > Thread-00, handling exception: java.net.SocketTimeoutException: Read timed > out Thread-00, setSoTimeout(45000) called > FINE: << " </BODY>[\n]" > Thread-00, setSoTimeout(45000) called > > [snip] > > Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]: length = > 176 > > [snip] > > [Raw read]: length = 5 > 0000: 17 03 01 03 0A ..... > Thread-00, handling exception: java.net.SocketException: Connection reset %% > Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND TLSv1 > ALERT: fatal, description = unexpected_message Padded plaintext before > ENCRYPTION: len = 18 > 0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B > 0010: C8 08 .. > Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception sending > alert: java.net.SocketException: Connection reset by peer: socket write error > Thread-00, called closeSocket() Thread-00, called close() Thread-00:Jan 29, > 2013 1:48:32 PM org.apache.http.impl.conn.PoolingClientConnectionManager > releaseConnection > > -----Original Message----- > From: Mark Claassen [mailto:[email protected]] > Sent: Tuesday, January 29, 2013 12:39 PM > To: 'HttpClient User Discussion' > Subject: RE: Connection Reset errors > > Don't know if this will help either...but here is hoping! > > -----Original Message----- > From: Oleg Kalnichevski [mailto:[email protected]] > Sent: Tuesday, January 29, 2013 9:03 AM > To: HttpClient User Discussion > Subject: Re: Connection Reset errors > > On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote: > > I ran my test program using 4.01 and it seems to fail even faster. > > (Switching a few things around to get it to compile, or course. Using > > ThreadSafeClientConnManager.) > > > > This is odd, since the user said she didn't experience the problem until > > the upgrade which gave her the newer version. Must also be something else > > going on. > > > > Still, the Java.net stuff is completely error free. (I can't say for > > sure if she would notice automatic retries, if they occurred.) > > > > Is there any way I can reduce the occurrences of the error? Something I > > could set for specific users that are experiencing this problem? > > > > Mark, > > This log unfortunately does not help. Interesting bit, though, it that a > connection reset happens while trying to read a head of a new message. > So, clearly the stale connection check did not work. In the former days (Java > 1.2 and possibly 1.3) state connection checking was simply useless with SSL > encrypted connections. Apparently it can still be ineffective even with > modern JREs. > > So, running your application with SSL debugging turned on should be more > informative: > > http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug.html > > Oleg > > > ------------------------------------ > > Stack trace and debug using 4.0.1 > > > > FINE: Stale connection check > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.client.protocol.RequestAddCookies process > > FINE: CookieSpec selected: best-match > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.client.DefaultRequestDirector execute > > FINE: Attempt 1 to execute request > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013 12:08:45 > > PM org.apache.http.impl.conn.Wire wire > > FINE: >> "POST / HTTP/1.1[EOL]" > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire > > FINE: >> "Content-Type: binary[EOL]" > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire > > FINE: >> "Content-Length: 34[EOL]" > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire > > FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]" > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire > > FINE: >> "Connection: Keep-Alive[EOL]" > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire > > FINE: >> "[EOL]" > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: >> POST / HTTP/1.1 > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: >> Content-Type: binary > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: >> Content-Length: 34 > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013 > > 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection > > sendRequestHeader > > FINE: >> Connection: Keep-Alive > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire wire > > FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX" > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.DefaultClientConnection close > > FINE: Connection closed > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.client.DefaultRequestDirector execute > > FINE: Closing the connection. > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.DefaultClientConnection close > > FINE: Connection closed > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.DefaultClientConnection shutdown > > FINE: Connection shut down > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager > > releaseConnection > > FINE: Released connection is not reusable. > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry > > FINE: Releasing connection > > [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null] > > Thread-00:Jan 28, 2013 12:08:45 PM > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread > > FINE: Notifying no-one, there are no waiting threads > > java.net.SocketException: Connection reset > > at java.net.SocketInputStream.read(SocketInputStream.java:168) > > at > > com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) > > at > > com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405) > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360) > > at > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798) > > at > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755) > > at > > com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75) > > at > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130) > > at > > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127) > > at > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233) > > at > > org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100) > > at > > org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98) > > at > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210) > > at > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271) > > at > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227) > > at > > org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209) > > at > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292) > > at > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126) > > at > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483) > > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641) > > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576) > > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554) > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191) > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157) > > at java.lang.Thread.run(Thread.java:662) > > > > -----Original Message----- > > From: Mark Claassen [mailto:[email protected]] > > Sent: Friday, January 25, 2013 12:30 PM > > To: 'HttpClient User Discussion' > > Subject: RE: Connection Reset errors > > > > I was able to put a break point in DefaultClientConnection.close(). It > > sometimes gets called when a connection is releasedManagedConnection is > > called. When the Connection Reset happens, it gets called 3 times in quick > > succession. However, looks like this is just normal processing for the > > failed connection. Whatever error is the root cause seems to have already > > occurred. > > > > The Connection Reset calls do not seem to be correlated to the normal > > releasedManagedConnection calls that work. > > > > First call to close() > > ---------------------- > > DefaultClientConnection.close:168 > > ManagedClientConnectionImpl.close:127 > > HttpRequestExecutor.closeConnection:144 > > HttpRequestExecutor.execute:131 > > DefaultRequestDirector.tryExecute:717 > > DefaultRequestDirector.execute:522 > > AbstractHttpClient.execute:906 > > AbstractHttpClient.execute:805 > > AbstractHttpClient.execute:784 > > TestNetwork$Worker.connect:187 > > TestNetwork$Worker.run:153 > > Thread.run:662 > > > > Second call to close() > > ---------------------- > > DefaultClientConnection.close:168 > > ManagedClientConnectionImpl.close:127 > > DefaultRequestDirector.tryExecute:723 > > DefaultRequestDirector.execute:522 > > AbstractHttpClient.execute:906 > > AbstractHttpClient.execute:805 > > AbstractHttpClient.execute:784 > > TestNetwork$Worker.connect:187 > > TestNetwork$Worker.run:153 > > Thread.run:662 > > > > Third call to close() > > ---------------------- > > DefaultClientConnection.close:168 > > HttpPoolEntry.close:89 > > AbstractConnPool.release:323 > > PoolingClientConnectionManager.releaseConnection:278 > > ManagedClientConnectionImpl.abortConnection:463 > > DefaultRequestDirector.abortConnection:1157 > > DefaultRequestDirector.execute:621 > > AbstractHttpClient.execute:906 > > AbstractHttpClient.execute:805 > > AbstractHttpClient.execute:784 > > TestNetwork$Worker.connect:187 > > TestNetwork$Worker.run:153 > > Thread.run:662 > > > > > > -----Original Message----- > > From: Mark Claassen [mailto:[email protected]] > > Sent: Friday, January 25, 2013 11:41 AM > > To: 'HttpClient User Discussion' > > Subject: RE: Connection Reset errors > > > > I set it back to SSL for now, took out the wait() in my loop, set MAX to 1 > > (pool size). So, I had just 1 thread making the request over and over > > again without pausing. > > > > Below is the last part of what I got. > > > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.PoolingClientConnectionManager > > releaseConnection > > FINE: Connection [id: 0][route: > > {s}->https://x86test.dev.donnell.com:5502] can be kept alive > > indefinitely Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.PoolingClientConnectionManager > > releaseConnection > > FINE: Connection released: [id: 0][route: > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 > > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager > > closeExpiredConnections > > FINE: Closing expired connections > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.PoolingClientConnectionManager > > requestConnection > > FINE: Connection request: [route: > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; route > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 > > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager > > leaseConnection > > FINE: Connection leased: [id: 0][route: > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 > > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector execute > > FINE: Stale connection check > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.client.protocol.RequestAddCookies process > > FINE: CookieSpec selected: best-match > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.client.protocol.RequestAuthCache process > > FINE: Auth cache not set in the context Thread-00:Jan 25, 2013 > > 11:38:01 AM > > org.apache.http.client.protocol.RequestTargetAuthentication process > > FINE: Target auth state: UNCHALLENGED > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.client.protocol.RequestProxyAuthentication process > > FINE: Proxy auth state: UNCHALLENGED > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.client.DefaultRequestDirector tryExecute > > FINE: Attempt 1 to execute request > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013 11:38:01 > > AM org.apache.http.impl.conn.Wire wire > > FINE: >> "POST / HTTP/1.1[\r][\n]" > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire > > FINE: >> "Content-Type: binary[\r][\n]" > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire > > FINE: >> "Content-Length: 34[\r][\n]" > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire > > FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]" > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire > > FINE: >> "Connection: Keep-Alive[\r][\n]" > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire > > FINE: >> "[\r][\n]" > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: >> POST / HTTP/1.1 > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: >> Content-Type: binary > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: >> Content-Length: 34 > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013 > > 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection > > sendRequestHeader > > FINE: >> Connection: Keep-Alive > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire wire > > FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX" > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.DefaultClientConnection close > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.client.DefaultRequestDirector tryExecute > > FINE: Closing the connection. > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.DefaultClientConnection close > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.DefaultClientConnection shutdown > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.DefaultClientConnection close > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed > > Thread-00:Jan 25, 2013 11:38:01 AM > > org.apache.http.impl.conn.PoolingClientConnectionManager > > releaseConnection > > FINE: Connection released: [id: 0][route: > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; route > > allocated: 0 of 1; total allocated: 0 of 1] > > java.net.SocketException: Connection reset > > at java.net.SocketInputStream.read(SocketInputStream.java:168) > > at > > com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) > > at > > com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405) > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360) > > at > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863) > > at > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820) > > at > > com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75) > > at > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) > > at > > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) > > at > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) > > at > > org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115) > > at > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) > > at > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61) > > at > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) > > at > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) > > at > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) > > at > > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) > > at > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) > > at > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) > > at > > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717) > > at > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522) > > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) > > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) > > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187) > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153) > > at java.lang.Thread.run(Thread.java:662) > > Java Result: 1 > > BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message----- > > From: Mark Claassen [mailto:[email protected]] > > Sent: Friday, January 25, 2013 11:25 AM > > To: 'HttpClient User Discussion' > > Subject: RE: Connection Reset errors > > > > I am running the test using a non-SSL connection and the Connection Reset > > is not happening. I am going to continue to run the test, but none of the > > SSL tests ran this long without failing. > > > > -----Original Message----- > > From: Mark Claassen [mailto:[email protected]] > > Sent: Friday, January 25, 2013 11:07 AM > > To: 'HttpClient User Discussion' > > Subject: RE: Connection Reset errors > > > > Before the upgrade, she was using HttpClient 4.01. > > --- > > Not sure if it is kosher to send an attachment to the list, but here is a > > test program. This errors for me after just a minute or two. > > > > The website it is connecting to is just a Tomcat server running the > > default ROOT webapp. It is delivering the default HTML page we have > > set up on that. I would imagine that it is ignoring the POST and > > treating it like a GET > > > > In production, where she is having the problem, it is using the default > > HttpsURLConnection socket factory. However, for testing, I am using this > > one to bypass the normal certificate checks and stuff. It happens to both > > of us, so I don't think the error has anything to do with the socket > > factory. The message that we send in production are also a lot longer, so > > I don't think it has anything to do with message size either. > > > > -----Original Message----- > > From: Oleg Kalnichevski [mailto:[email protected]] > > Sent: Friday, January 25, 2013 7:46 AM > > To: HttpClient User Discussion > > Subject: Re: Connection Reset errors > > > > Mark > > > > You do not need to do anything special other than closing the content input > > stream of the response entity (if it is present) to ensure proper resource > > deallocation. What is important that it has to be done no matter if you > > need the content or not. I do not think the problem you are dealing with > > has anything to do with resource management. > > > > There are probably several things you might want to try out > > > > (1) if you can confirm that this issue is HttpClient version specific and > > tell me what version of HttpClient exhibited it first (say, after upgrade > > from 4.0.3 to 4.1) I could analyze the differences and try to figure out > > what could potentially be the cause of connection resets. > > > > (2) This is entirely possible this is somehow SSL related and connection > > gets reset by the SSL stack. You might want to run your app with SSL debug > > enabled and see if there is anything in the SSL logs that could help > > explain connection being reset. > > > > (3) Otherwise there is no way around getting very close and personal with > > Wireshark. > > > > Oleg > > > > > > > > On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote: > > > Something else I thought of is this: We have some transactions where the > > > response is not too important. For instance, if we send a pulse to the > > > server to let it know we are still here. We don't really care what the > > > response is, if we didn't get an error then we did our part. If we did > > > get and error, we will try again soon anyway, so no big deal. > > > > > > Is there something that needs to be done on a connection that is closed > > > before reading all the data? I have never experienced reading stale > > > data, so this seems a bit unlikely to me. However, if there is implicit > > > cleanup that is being done, could this be, in some way, causing my > > > problems? > > > > > > -----Original Message----- > > > From: Mark Claassen [mailto:[email protected]] > > > Sent: Thursday, January 24, 2013 10:29 AM > > > To: 'HttpClient User Discussion' > > > Subject: RE: Connection Reset errors > > > > > > I am just not convinced this is a network thing. It happens too often > > > for her, and never with the java.net connection. She has a newer > > > machine, maybe threading issues are just more apparent on her hardware. > > > > > > So, maybe the problem is somewhere in my code. Can you tell me what are > > > the methods that might release a connection back to the pool? I am > > > wondering if there is a way I could be releasing the connection back to > > > the pool, it gets picked up by something else, and then the original > > > thread tries to close it again or something. > > > > > > The docs say the closing the InputStream will do it, but will anything > > > else do it implicitly? > > > -- > > > > > > Another avenue I am investigating is trying to force the issue by making > > > the connections work harder. > > > I am trying to force the issue doing a lot of requests all at once. > > > I reduced my MAX down to 1 to force contention. Everything seems to > > > work pretty well. However, I did get the Connection Reset error > > > twice on my machine! The logs were fairly regular looking since > > > there is only one connection at a time. However, when this > > > happened, it looks like it is sending 2 request at once. This > > > shouldn't be happening because there is only one connection. > > > (Granted, it is hard to tell if the logging is indicating a > > > threading issue, or if the "Connection Reset" error caused the > > > logging abnormality.) > > > > > > I don't need to block around the "httpClient.execute(request);" do I? > > > > > > I am using HttpClient 4.2.3 (with HttpCore 4.2.3) > > > -- > > > > > > I do this to establish the connection: > > > > > > This is run once in the setup > > > ----------------------------- > > > connectionManager = new > > > PoolingClientConnectionManager(schemeRegistry); > > > connectionManager.setDefaultMaxPerRoute(MAX); > > > connectionManager.setMaxTotal(MAX); > > > > > > httpClient = new DefaultHttpClient(connectionManager, params); > > > > > > > > > > > > This is called when I want to make a request > > > -------------------------------------------- > > > response = httpClient.execute(request); > > > > > > > > > > > > -----Original Message----- > > > From: Mark Claassen [mailto:[email protected]] > > > Sent: Friday, January 18, 2013 5:06 PM > > > To: 'HttpClient User Discussion' > > > Subject: RE: Connection Reset errors > > > > > > Well, the user went all day without any errors while using the Java.net > > > connector. > > > > > > The error using the HttpClient connector has been happening with > > > HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 > > > (and now is continuing under 4.2.3). I will have to check on the upgrade > > > path and see if she had been using a 4.0.x version before this, or if > > > they jumped from a version that used a 3.x version. > > > > > > Mark > > > > > > > > > > > > -----Original Message----- > > > From: Mark Claassen [mailto:[email protected]] > > > Sent: Friday, January 18, 2013 1:47 PM > > > To: 'HttpClient User Discussion' > > > Subject: RE: Connection Reset errors > > > > > > Oh, I forgot to add that all my requests are POST requests. > > > > > > -----Original Message----- > > > From: Mark Claassen [mailto:[email protected]] > > > Sent: Friday, January 18, 2013 1:02 PM > > > To: 'HttpClient User Discussion' > > > Subject: RE: Connection Reset errors > > > > > > While she is testing the java.net stuff, I decided to look into the > > > possibility of retrying the request. Looking at this more closely, I see > > > that the error is on the reading of the data, and the write seems to > > > succeed. This gives me something to check on, since I should be able to > > > see whether or not the servlet actually got her request or not. > > > > > > However, is this a common manifestation of the Connection Reset error? > > > The write succeeds (or fails silently) and then it is on the read the > > > problem can be detected? Or does this mean that the socket was fine > > > during the write operation, but then was somehow severed before the read? > > > > > > Mark > > > > > > -----Original Message----- > > > From: Mark Claassen [mailto:[email protected]] > > > Sent: Friday, January 18, 2013 12:06 PM > > > To: 'HttpClient User Discussion' > > > Subject: RE: Connection Reset errors > > > > > > Looks like the stale connection check is enabled by default anyway, so I > > > must have been using it all along. > > > > > > > It may well be that HttpURLConnection silently retries failed > > > > requests > > > Is this a common thing to do, like for browsers? Granted, she is in our > > > app more than most others, but I don't think she experiences issues with > > > other applications. > > > > > > She is not getting a Connection Refused or some other unable to read > > > exception, but a Connection Reset. The stale connection check passed, so > > > HttpClient thinks this is a good connection, right? Or are there limits > > > to how reliable a stale connection check can be? > > > > > > public boolean isStale() { > > > if (!isOpen()) { > > > return true; > > > } > > > if (isEof()) { > > > return true; > > > } > > > try { > > > this.inbuffer.isDataAvailable(1); > > > return isEof(); > > > } catch (SocketTimeoutException ex) { > > > return false; > > > } catch (IOException ex) { > > > return true; > > > } > > > } > > > > > > > > > -----Original Message----- > > > From: Oleg Kalnichevski [mailto:[email protected]] > > > Sent: Friday, January 18, 2013 11:24 AM > > > To: HttpClient User Discussion > > > Subject: Re: Connection Reset errors > > > > > > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote: > > > > Before I switched to using HttpClient years ago, I used > > > > HttpsURLConnection. Not sure how things were going to turn out with > > > > HttpClient, I kept that code in place and made it so I could use either > > > > API. I had her switch back to using the java.net stuff the other day > > > > and the problem appeared to go away. I have configured her system to > > > > use the java.net connector again today. Before I make any conclusions > > > > I need some more data from her on each connector. However, the > > > > previous data seems to indicate that it is only an issue when using > > > > HttpClient (originally 4.1.3, now 4.2.3). > > > > > > > > > > It may well be that HttpURLConnection silently retries failed requests > > > without your knowing about it. It certainly did that when I last touched > > > it (which admittedly was a looooooong time ago), and that was precisely > > > the reason I had to stop using it and to look for a better alternative. > > > > > > > > > > Some other things of note: > > > > - She is using JRE 6 Update 13. (There is a long story behind > > > > this.) > > > > - The application is launched via Webstart > > > > - The SocketFactory is created with: fact = new > > > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory(),v > > > > er > > > > if > > > > ier); > > > > > > > > > > I do not thing any of these should matter. > > > > > > Oleg > > > > > > > -----Original Message----- > > > > From: Oleg Kalnichevski [mailto:[email protected]] > > > > Sent: Friday, January 18, 2013 6:17 AM > > > > To: HttpClient User Discussion > > > > Subject: Re: Connection Reset errors > > > > > > > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote: > > > > > Thanks for the explanation. > > > > > > pro-actively evicting expired connections and connections that > > > > > > have been idle > > > > > Seems like just doing the stale check would more reliable, and > > > > > speed does not seem to be a factory (especially with 4.2.3) > > > > > > > > > > I got in configured and distributed to this user and she is > > > > > still having the same problem. Any other ideas? I double > > > > > checked my code, and it looks like everything is correct. (She > > > > > is not using a > > > > > proxy.) > > > > > > > > > > > > > This can also be genuine network stability issues. Essentially, > > > > connection resets are inevitable and robust HTTP services have to be > > > > prepared to deal with them (most likely by re-trying idempotent methods > > > > automatically and reporting the problem back to the caller otherwise). > > > > > > > > Oleg > > > > > > > > > > > > > > > > > > Method that makes the request (from multiple threads) > > > > > { > > > > > response = httpClient.execute(request); > > > > > } > > > > > Bulk of configuration method (MAX is 10, > > > > > CONNECTION_ESTABLISH_TIMEOUT = 10) > > > > > { > > > > > HttpParams params = new BasicHttpParams(); > > > > > > > > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new > > > > > TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS)); > > > > > > > > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, > > > > > CONNECTION_ESTABLISH_TIMEOUT * 1000); > > > > > > > > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, > > > > > DEFAULT_HTTP_SOCKET_BUFFER_SIZE); > > > > > > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, > > > > > Boolean.TRUE); > > > > > configureProxy(params,protocol); > > > > > > > > > > connectionManager = new > > > > > PoolingClientConnectionManager(schemeRegistry); > > > > > connectionManager.setDefaultMaxPerRoute(MAX); > > > > > connectionManager.setMaxTotal(MAX); > > > > > > > > > > httpClient = new > > > > > DefaultHttpClient(connectionManager, params); > > > > > } > > > > > private void configureProxy(HttpParams params, String scheme) { > > > > > Proxy proxy = getProxy(); > > > > > if (proxy.type() != Proxy.Type.DIRECT) { > > > > > InetSocketAddress address = (InetSocketAddress) > > > > > proxy.address(); > > > > > //SSL or not, we set the proxy up as HTTP > > > > > HttpHost httpProxy = new > > > > > HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP); > > > > > > > > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy); > > > > > LogManager.log("Setting proxy in HostConfig to > > > > > " + address, Level.INFO); > > > > > if (!scheme.equals(SCHEME_HTTP)) { > > > > > //If the scheme is HTTPS, we need to > > > > > register an HTTP scheme for the proxy. > > > > > schemeRegistry.register(new > > > > > Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory())); > > > > > } > > > > > } > > > > > > > > > > } > > > > > > > > > > -----Original Message----- > > > > > From: Oleg Kalnichevski [mailto:[email protected]] > > > > > Sent: Thursday, January 17, 2013 7:21 AM > > > > > To: HttpClient User Discussion > > > > > Subject: Re: Connection Reset errors > > > > > > > > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote: > > > > > > > > > > Basically 'connection reset' errors are a direct result of a general > > > > > limitation of the blocking I/O model: when not blocked in an I/O > > > > > operation there is no way for the socket to get notified of the > > > > > opposite endpoint terminating the connection. While kept alive in the > > > > > pool blocking HTTP connections can get stale. However, the only way > > > > > to find it out is to try to attempt an I/O operation on such > > > > > connection. > > > > > > > > > > > I noticed that there is a stale connection check: > > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CH > > > > > > EC > > > > > > K, > > > > > > Boolean.TRUE); > > > > > > > > > > > > Is this something that I should be using? I am connecting to the > > > > > > same source over and over again. > > > > > > > > > > > > Mark > > > > > > > > > > > > > > > > Yes, this is what you should be using. Another alternative would be > > > > > pro-actively evicting expired connections and connections that have > > > > > been idle longer than a given period of time. > > > > > > > > > > Oleg > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Mark Claassen [mailto:[email protected]] > > > > > > Sent: Wednesday, January 16, 2013 3:07 PM > > > > > > To: 'HttpClient User Discussion' > > > > > > Subject: Connection Reset errors > > > > > > > > > > > > I have a user getting a lot of Connection Reset errors. I did > > > > > > not think this had do to with HttpClient, so much as other > > > > > > factors on her machine and the server. However, I did a bit of > > > > > > searching and see that people have commented on this error with > > > > > > HttpClient before. I just upgraded to HttpClient 4.2.3 to see if > > > > > > that would help at all, and it didn't. > > > > > > > > > > > > This user is fine for a while, and then will get the > > > > > > Connection Reset error. She then just retries the request, and it > > > > > > works. Any tips on how to resolve this would be greatly > > > > > > appreciated. > > > > > > > > > > > > Thanks, > > > > > > Mark > > > > > > > > > > > > We are using Tomcat 7.0.27, and the Connector is configured like > > > > > > this: > > > > > > <Connector > > > > > > port="5502" > > > > > > > > > > > > protocol="org.apache.coyote.http11.Http11NioProtocol" > > > > > > server="Unknown" > > > > > > connectionLinger="0" > > > > > > socket.soTimeout="300000" > > > > > > SSLEnabled="true" > > > > > > maxThreads="150" > > > > > > scheme="https" > > > > > > secure="true" > > > > > > clientAuth="false" > > > > > > keystoreFile="-----" > > > > > > keystoreType="PKCS12" > > > > > > keystorePass="-----" > > > > > > sslProtocol="TLS" > > > > > > > > > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_SHA > > > > > > ,T LS _D HE _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES > > > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3D > > > > > > ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/> > > > > > > > > > > > > > > > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException) > > > > > > Message (Connection reset) > > > > > > at java.net.SocketInputStream.read(Unknown Source) > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown > > > > > > Source) > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) > > > > > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown > > > > > > Source) > > > > > > at > > > > > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown > > > > > > Source) > > > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown > > > > > > Source) > > > > > > at > > > > > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) > > > > > > at > > > > > > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) > > > > > > at > > > > > > org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212) > > > > > > at > > > > > > org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177) > > > > > > at > > > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream. > > > > > > ja > > > > > > va:138) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > -- > > > > > > -- > > > > > > -- > > > > > > - To unsubscribe, e-mail: > > > > > > [email protected] > > > > > > For additional commands, e-mail: > > > > > > [email protected] > > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > -- > > > > > > -- > > > > > > -- > > > > > > - To unsubscribe, e-mail: > > > > > > [email protected] > > > > > > For additional commands, e-mail: > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------- > > > > > -- > > > > > -- > > > > > - To unsubscribe, e-mail: > > > > > [email protected] > > > > > For additional commands, e-mail: > > > > > [email protected] > > > > > > > > > > > > > > > ---------------------------------------------------------------- > > > > > -- > > > > > -- > > > > > - To unsubscribe, e-mail: > > > > > [email protected] > > > > > For additional commands, e-mail: > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: > > > > [email protected] > > > > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: > > > > [email protected] > > > > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
