On Tue, 2013-01-29 at 15:27 -0500, Mark Claassen wrote: > Wow, that certainly helped my test case! Now to try it for real. > > This leaves me with 2 big questions: > 1) Is there a reason for Tomcat to behave this way?
In the early days of HTTP/1.1 many HTTP agents had issues with persistent connection support. This feature was most likely intended to help broken clients recover by forcing them to open a new connection after a certain number of requests. > 2) Why is this not WAY more common? > > I think the answer to #2 might be (at least partially) that we have our > connectionLinger down to 0 to reduce the number of sockets in TIME_WAIT. So, > when Tomcat closed the connection, if the client was a bit too slow, the > socket would close before the client could finish reading all the data. > That explains it. By setting SO_LINGER to zero you are basically instructing Tomcat to terminate connections abnormally by sending a RST (connection reset) command instead of a normal TCP termination procedure. It looks like all this misery was self inflicted. Oleg > I had thought I had tried to change the linger in an earlier test, but maybe > I forgot to save the change or something. > > In any event, she is trying this and we will see how it goes. > > -----Original Message----- > From: Mark Claassen [mailto:[email protected]] > Sent: Tuesday, January 29, 2013 3:12 PM > To: 'HttpClient User Discussion' > Subject: RE: Connection Reset errors > > Thanks! I will check it out! > > -----Original Message----- > From: Oleg Kalnichevski [mailto:[email protected]] > Sent: Tuesday, January 29, 2013 3:09 PM > To: HttpClient User Discussion > Subject: Re: Connection Reset errors > > On Tue, 2013-01-29 at 14:59 -0500, Mark Claassen wrote: > > CLUE! > > > > I opened up the cipher suites on Tomecat like I mentioned and then made my > > own socketfactory (limiting the cipher suites) as I saw in an example. Now > > when I run the program it dies at the same place every time...on the 101st > > connection. This has to be significant in some way...although I don't know > > how. > > > > I about doubled the size of the message I send, and it makes no difference. > > > > I put the iteration count in the HttpHeader so I could track it in the > > Tomcat access log. Tomcat shows that it receives this message > > (message 100) and that it thinks it succeeded (status == 200) > > > > The SocketTimeoutException I noticed before is everywhere in the logs. > > This is from the StaleConnectionCheck and is likely irrelevant. > > > > > > > Tomcat by default closes persistent connections once it has been used to > serve 100 requests. > > See 'maxKeepAliveRequests' parameter: > > http://tomcat.apache.org/tomcat-7.0-doc/config/http.html > > Try setting this value to -1 and see if that makes the problem go away. > > Oleg > > > > > > > -----Original Message----- > > From: Mark Claassen [mailto:[email protected]] > > Sent: Tuesday, January 29, 2013 2:01 PM > > To: 'HttpClient User Discussion' > > Subject: RE: Connection Reset errors > > > > I saw some things on the web with this error, but those seemed to be errors > > on startup because particular protocols where not enabled. I changed my > > Tomcat configuration to use SSL instead of TLS, and removed the > > cipherSuites attribute. Similar result. However I saw the below in the > > debug output. > > > > What is the socket timeout doing there, and why is there a > > SocketTimeoutException when this is all occurring in the same second. > > > > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire > > FINE: << " <p>If you page is not redirected, click > > <a href="JViewer/">here</a>[\n]" > > Thread-00:Jan 29, 2013 1:48:32 PM org.apache.http.impl.conn.Wire wire > > Thread-00, handling exception: java.net.SocketTimeoutException: Read timed > > out Thread-00, setSoTimeout(45000) called > > FINE: << " </BODY>[\n]" > > Thread-00, setSoTimeout(45000) called > > > > [snip] > > > > Thread-00, WRITE: TLSv1 Application Data, length = 171 [Raw write]: > > length = 176 > > > > [snip] > > > > [Raw read]: length = 5 > > 0000: 17 03 01 03 0A ..... > > Thread-00, handling exception: java.net.SocketException: Connection reset > > %% Invalidated: [Session-1, SSL_RSA_WITH_RC4_128_MD5] Thread-00, SEND > > TLSv1 ALERT: fatal, description = unexpected_message Padded plaintext > > before ENCRYPTION: len = 18 > > 0000: 02 0A C7 6D 68 F3 71 E6 E2 D7 E2 55 CD A4 B1 42 ...mh.q....U...B > > 0010: C8 08 .. > > Thread-00, WRITE: TLSv1 Alert, length = 18 Thread-00, Exception > > sending alert: java.net.SocketException: Connection reset by peer: > > socket write error Thread-00, called closeSocket() Thread-00, called > > close() Thread-00:Jan 29, 2013 1:48:32 PM > > org.apache.http.impl.conn.PoolingClientConnectionManager > > releaseConnection > > > > -----Original Message----- > > From: Mark Claassen [mailto:[email protected]] > > Sent: Tuesday, January 29, 2013 12:39 PM > > To: 'HttpClient User Discussion' > > Subject: RE: Connection Reset errors > > > > Don't know if this will help either...but here is hoping! > > > > -----Original Message----- > > From: Oleg Kalnichevski [mailto:[email protected]] > > Sent: Tuesday, January 29, 2013 9:03 AM > > To: HttpClient User Discussion > > Subject: Re: Connection Reset errors > > > > On Mon, 2013-01-28 at 13:42 -0500, Mark Claassen wrote: > > > I ran my test program using 4.01 and it seems to fail even faster. > > > (Switching a few things around to get it to compile, or course. > > > Using > > > ThreadSafeClientConnManager.) > > > > > > This is odd, since the user said she didn't experience the problem until > > > the upgrade which gave her the newer version. Must also be something > > > else going on. > > > > > > Still, the Java.net stuff is completely error free. (I can't say > > > for sure if she would notice automatic retries, if they occurred.) > > > > > > Is there any way I can reduce the occurrences of the error? Something I > > > could set for specific users that are experiencing this problem? > > > > > > > Mark, > > > > This log unfortunately does not help. Interesting bit, though, it that a > > connection reset happens while trying to read a head of a new message. > > So, clearly the stale connection check did not work. In the former days > > (Java 1.2 and possibly 1.3) state connection checking was simply useless > > with SSL encrypted connections. Apparently it can still be ineffective even > > with modern JREs. > > > > So, running your application with SSL debugging turned on should be > > more > > informative: > > > > http://docs.oracle.com/javase/1.5.0/docs/guide/security/jsse/ReadDebug > > .html > > > > Oleg > > > > > ------------------------------------ > > > Stack trace and debug using 4.0.1 > > > > > > FINE: Stale connection check > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.client.protocol.RequestAddCookies process > > > FINE: CookieSpec selected: best-match Thread-00:Jan 28, 2013 > > > 12:08:45 PM org.apache.http.impl.client.DefaultRequestDirector > > > execute > > > FINE: Attempt 1 to execute request > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 28, 2013 > > > 12:08:45 PM org.apache.http.impl.conn.Wire wire > > > FINE: >> "POST / HTTP/1.1[EOL]" > > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "Content-Type: binary[EOL]" > > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "Content-Length: 34[EOL]" > > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "Host: x86test.dev.donnell.com:5502[EOL]" > > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "Connection: Keep-Alive[EOL]" > > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "[EOL]" > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: >> POST / HTTP/1.1 > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: >> Content-Type: binary > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: >> Content-Length: 34 > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 28, 2013 > > > 12:08:45 PM org.apache.http.impl.conn.DefaultClientConnection > > > sendRequestHeader > > > FINE: >> Connection: Keep-Alive > > > Thread-00:Jan 28, 2013 12:08:45 PM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "0000000300 XXXXXXXXXXXXXXXXXXXXXXX" > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.DefaultClientConnection close > > > FINE: Connection closed > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.client.DefaultRequestDirector execute > > > FINE: Closing the connection. > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.DefaultClientConnection close > > > FINE: Connection closed > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.DefaultClientConnection shutdown > > > FINE: Connection shut down > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager > > > releaseConnection > > > FINE: Released connection is not reusable. > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute freeEntry > > > FINE: Releasing connection > > > [HttpRoute[{s}->https://x86test.dev.donnell.com:5502]][null] > > > Thread-00:Jan 28, 2013 12:08:45 PM > > > org.apache.http.impl.conn.tsccm.ConnPoolByRoute notifyWaitingThread > > > FINE: Notifying no-one, there are no waiting threads > > > java.net.SocketException: Connection reset > > > at java.net.SocketInputStream.read(SocketInputStream.java:168) > > > at > > > com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) > > > at > > > com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405) > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360) > > > at > > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:798) > > > at > > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:755) > > > at > > > com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75) > > > at > > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:130) > > > at > > > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:127) > > > at > > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:233) > > > at > > > org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:100) > > > at > > > org.apache.http.impl.conn.DefaultResponseParser.parseHead(DefaultResponseParser.java:98) > > > at > > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:210) > > > at > > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:271) > > > at > > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:227) > > > at > > > org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:209) > > > at > > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:292) > > > at > > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:126) > > > at > > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:483) > > > at > > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641) > > > at > > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:576) > > > at > > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:554) > > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:191) > > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:157) > > > at java.lang.Thread.run(Thread.java:662) > > > > > > -----Original Message----- > > > From: Mark Claassen [mailto:[email protected]] > > > Sent: Friday, January 25, 2013 12:30 PM > > > To: 'HttpClient User Discussion' > > > Subject: RE: Connection Reset errors > > > > > > I was able to put a break point in DefaultClientConnection.close(). It > > > sometimes gets called when a connection is releasedManagedConnection is > > > called. When the Connection Reset happens, it gets called 3 times in > > > quick succession. However, looks like this is just normal processing for > > > the failed connection. Whatever error is the root cause seems to have > > > already occurred. > > > > > > The Connection Reset calls do not seem to be correlated to the normal > > > releasedManagedConnection calls that work. > > > > > > First call to close() > > > ---------------------- > > > DefaultClientConnection.close:168 > > > ManagedClientConnectionImpl.close:127 > > > HttpRequestExecutor.closeConnection:144 > > > HttpRequestExecutor.execute:131 > > > DefaultRequestDirector.tryExecute:717 > > > DefaultRequestDirector.execute:522 > > > AbstractHttpClient.execute:906 > > > AbstractHttpClient.execute:805 > > > AbstractHttpClient.execute:784 > > > TestNetwork$Worker.connect:187 > > > TestNetwork$Worker.run:153 > > > Thread.run:662 > > > > > > Second call to close() > > > ---------------------- > > > DefaultClientConnection.close:168 > > > ManagedClientConnectionImpl.close:127 > > > DefaultRequestDirector.tryExecute:723 > > > DefaultRequestDirector.execute:522 > > > AbstractHttpClient.execute:906 > > > AbstractHttpClient.execute:805 > > > AbstractHttpClient.execute:784 > > > TestNetwork$Worker.connect:187 > > > TestNetwork$Worker.run:153 > > > Thread.run:662 > > > > > > Third call to close() > > > ---------------------- > > > DefaultClientConnection.close:168 > > > HttpPoolEntry.close:89 > > > AbstractConnPool.release:323 > > > PoolingClientConnectionManager.releaseConnection:278 > > > ManagedClientConnectionImpl.abortConnection:463 > > > DefaultRequestDirector.abortConnection:1157 > > > DefaultRequestDirector.execute:621 > > > AbstractHttpClient.execute:906 > > > AbstractHttpClient.execute:805 > > > AbstractHttpClient.execute:784 > > > TestNetwork$Worker.connect:187 > > > TestNetwork$Worker.run:153 > > > Thread.run:662 > > > > > > > > > -----Original Message----- > > > From: Mark Claassen [mailto:[email protected]] > > > Sent: Friday, January 25, 2013 11:41 AM > > > To: 'HttpClient User Discussion' > > > Subject: RE: Connection Reset errors > > > > > > I set it back to SSL for now, took out the wait() in my loop, set MAX to > > > 1 (pool size). So, I had just 1 thread making the request over and over > > > again without pausing. > > > > > > Below is the last part of what I got. > > > > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.PoolingClientConnectionManager > > > releaseConnection > > > FINE: Connection [id: 0][route: > > > {s}->https://x86test.dev.donnell.com:5502] can be kept alive > > > indefinitely Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.PoolingClientConnectionManager > > > releaseConnection > > > FINE: Connection released: [id: 0][route: > > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; > > > route > > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 > > > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager > > > closeExpiredConnections > > > FINE: Closing expired connections > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.PoolingClientConnectionManager > > > requestConnection > > > FINE: Connection request: [route: > > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 1; > > > route > > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 > > > 11:38:01 AM org.apache.http.impl.conn.PoolingClientConnectionManager > > > leaseConnection > > > FINE: Connection leased: [id: 0][route: > > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; > > > route > > > allocated: 1 of 1; total allocated: 1 of 1] Thread-00:Jan 25, 2013 > > > 11:38:01 AM org.apache.http.impl.client.DefaultRequestDirector > > > execute > > > FINE: Stale connection check > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.client.protocol.RequestAddCookies process > > > FINE: CookieSpec selected: best-match Thread-00:Jan 25, 2013 > > > 11:38:01 AM org.apache.http.client.protocol.RequestAuthCache process > > > FINE: Auth cache not set in the context Thread-00:Jan 25, 2013 > > > 11:38:01 AM > > > org.apache.http.client.protocol.RequestTargetAuthentication process > > > FINE: Target auth state: UNCHALLENGED Thread-00:Jan 25, 2013 > > > 11:38:01 AM > > > org.apache.http.client.protocol.RequestProxyAuthentication process > > > FINE: Proxy auth state: UNCHALLENGED Thread-00:Jan 25, 2013 11:38:01 > > > AM org.apache.http.impl.client.DefaultRequestDirector tryExecute > > > FINE: Attempt 1 to execute request > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: Sending request: POST / HTTP/1.1 Thread-00:Jan 25, 2013 > > > 11:38:01 AM org.apache.http.impl.conn.Wire wire > > > FINE: >> "POST / HTTP/1.1[\r][\n]" > > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "Content-Type: binary[\r][\n]" > > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "Content-Length: 34[\r][\n]" > > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "Host: x86test.dev.donnell.com:5502[\r][\n]" > > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "Connection: Keep-Alive[\r][\n]" > > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "[\r][\n]" > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: >> POST / HTTP/1.1 > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: >> Content-Type: binary > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: >> Content-Length: 34 > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.DefaultClientConnection sendRequestHeader > > > FINE: >> Host: x86test.dev.donnell.com:5502 Thread-00:Jan 25, 2013 > > > 11:38:01 AM org.apache.http.impl.conn.DefaultClientConnection > > > sendRequestHeader > > > FINE: >> Connection: Keep-Alive > > > Thread-00:Jan 25, 2013 11:38:01 AM org.apache.http.impl.conn.Wire > > > wire > > > FINE: >> "0000000100 XXXXXXXXXXXXXXXXXXXXXXX" > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.DefaultClientConnection close > > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.client.DefaultRequestDirector tryExecute > > > FINE: Closing the connection. > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.DefaultClientConnection close > > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.DefaultClientConnection shutdown > > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 shut down > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.DefaultClientConnection close > > > FINE: Connection 0.0.0.0:61602<->192.9.201.75:5502 closed > > > Thread-00:Jan 25, 2013 11:38:01 AM > > > org.apache.http.impl.conn.PoolingClientConnectionManager > > > releaseConnection > > > FINE: Connection released: [id: 0][route: > > > {s}->https://x86test.dev.donnell.com:5502][total kept alive: 0; > > > route > > > allocated: 0 of 1; total allocated: 0 of 1] > > > java.net.SocketException: Connection reset > > > at java.net.SocketInputStream.read(SocketInputStream.java:168) > > > at > > > com.sun.net.ssl.internal.ssl.InputRecord.readFully(InputRecord.java:293) > > > at > > > com.sun.net.ssl.internal.ssl.InputRecord.readV3Record(InputRecord.java:405) > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(InputRecord.java:360) > > > at > > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:863) > > > at > > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820) > > > at > > > com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75) > > > at > > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) > > > at > > > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) > > > at > > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) > > > at > > > org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115) > > > at > > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) > > > at > > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61) > > > at > > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) > > > at > > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) > > > at > > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) > > > at > > > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) > > > at > > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) > > > at > > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) > > > at > > > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:717) > > > at > > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:522) > > > at > > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) > > > at > > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) > > > at > > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) > > > at dsi.app.TestNetwork$Worker.connect(TestNetwork.java:187) > > > at dsi.app.TestNetwork$Worker.run(TestNetwork.java:153) > > > at java.lang.Thread.run(Thread.java:662) > > > Java Result: 1 > > > BUILD SUCCESSFUL (total time: 26 seconds) -----Original Message----- > > > From: Mark Claassen [mailto:[email protected]] > > > Sent: Friday, January 25, 2013 11:25 AM > > > To: 'HttpClient User Discussion' > > > Subject: RE: Connection Reset errors > > > > > > I am running the test using a non-SSL connection and the Connection Reset > > > is not happening. I am going to continue to run the test, but none of > > > the SSL tests ran this long without failing. > > > > > > -----Original Message----- > > > From: Mark Claassen [mailto:[email protected]] > > > Sent: Friday, January 25, 2013 11:07 AM > > > To: 'HttpClient User Discussion' > > > Subject: RE: Connection Reset errors > > > > > > Before the upgrade, she was using HttpClient 4.01. > > > --- > > > Not sure if it is kosher to send an attachment to the list, but here is a > > > test program. This errors for me after just a minute or two. > > > > > > The website it is connecting to is just a Tomcat server running the > > > default ROOT webapp. It is delivering the default HTML page we have > > > set up on that. I would imagine that it is ignoring the POST and > > > treating it like a GET > > > > > > In production, where she is having the problem, it is using the default > > > HttpsURLConnection socket factory. However, for testing, I am using this > > > one to bypass the normal certificate checks and stuff. It happens to > > > both of us, so I don't think the error has anything to do with the socket > > > factory. The message that we send in production are also a lot longer, > > > so I don't think it has anything to do with message size either. > > > > > > -----Original Message----- > > > From: Oleg Kalnichevski [mailto:[email protected]] > > > Sent: Friday, January 25, 2013 7:46 AM > > > To: HttpClient User Discussion > > > Subject: Re: Connection Reset errors > > > > > > Mark > > > > > > You do not need to do anything special other than closing the content > > > input stream of the response entity (if it is present) to ensure proper > > > resource deallocation. What is important that it has to be done no matter > > > if you need the content or not. I do not think the problem you are > > > dealing with has anything to do with resource management. > > > > > > There are probably several things you might want to try out > > > > > > (1) if you can confirm that this issue is HttpClient version specific and > > > tell me what version of HttpClient exhibited it first (say, after upgrade > > > from 4.0.3 to 4.1) I could analyze the differences and try to figure out > > > what could potentially be the cause of connection resets. > > > > > > (2) This is entirely possible this is somehow SSL related and connection > > > gets reset by the SSL stack. You might want to run your app with SSL > > > debug enabled and see if there is anything in the SSL logs that could > > > help explain connection being reset. > > > > > > (3) Otherwise there is no way around getting very close and personal with > > > Wireshark. > > > > > > Oleg > > > > > > > > > > > > On Thu, 2013-01-24 at 10:51 -0500, Mark Claassen wrote: > > > > Something else I thought of is this: We have some transactions where > > > > the response is not too important. For instance, if we send a pulse to > > > > the server to let it know we are still here. We don't really care what > > > > the response is, if we didn't get an error then we did our part. If we > > > > did get and error, we will try again soon anyway, so no big deal. > > > > > > > > Is there something that needs to be done on a connection that is closed > > > > before reading all the data? I have never experienced reading stale > > > > data, so this seems a bit unlikely to me. However, if there is > > > > implicit cleanup that is being done, could this be, in some way, > > > > causing my problems? > > > > > > > > -----Original Message----- > > > > From: Mark Claassen [mailto:[email protected]] > > > > Sent: Thursday, January 24, 2013 10:29 AM > > > > To: 'HttpClient User Discussion' > > > > Subject: RE: Connection Reset errors > > > > > > > > I am just not convinced this is a network thing. It happens too often > > > > for her, and never with the java.net connection. She has a newer > > > > machine, maybe threading issues are just more apparent on her hardware. > > > > > > > > So, maybe the problem is somewhere in my code. Can you tell me what > > > > are the methods that might release a connection back to the pool? I am > > > > wondering if there is a way I could be releasing the connection back to > > > > the pool, it gets picked up by something else, and then the original > > > > thread tries to close it again or something. > > > > > > > > The docs say the closing the InputStream will do it, but will anything > > > > else do it implicitly? > > > > -- > > > > > > > > Another avenue I am investigating is trying to force the issue by > > > > making the connections work harder. > > > > I am trying to force the issue doing a lot of requests all at once. > > > > I reduced my MAX down to 1 to force contention. Everything seems > > > > to work pretty well. However, I did get the Connection Reset > > > > error twice on my machine! The logs were fairly regular looking > > > > since there is only one connection at a time. However, when this > > > > happened, it looks like it is sending 2 request at once. This > > > > shouldn't be happening because there is only one connection. > > > > (Granted, it is hard to tell if the logging is indicating a > > > > threading issue, or if the "Connection Reset" error caused the > > > > logging abnormality.) > > > > > > > > I don't need to block around the "httpClient.execute(request);" do I? > > > > > > > > I am using HttpClient 4.2.3 (with HttpCore 4.2.3) > > > > -- > > > > > > > > I do this to establish the connection: > > > > > > > > This is run once in the setup > > > > ----------------------------- > > > > connectionManager = new > > > > PoolingClientConnectionManager(schemeRegistry); > > > > connectionManager.setDefaultMaxPerRoute(MAX); > > > > connectionManager.setMaxTotal(MAX); > > > > > > > > httpClient = new DefaultHttpClient(connectionManager, params); > > > > > > > > > > > > > > > > This is called when I want to make a request > > > > -------------------------------------------- > > > > response = httpClient.execute(request); > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Mark Claassen [mailto:[email protected]] > > > > Sent: Friday, January 18, 2013 5:06 PM > > > > To: 'HttpClient User Discussion' > > > > Subject: RE: Connection Reset errors > > > > > > > > Well, the user went all day without any errors while using the Java.net > > > > connector. > > > > > > > > The error using the HttpClient connector has been happening with > > > > HttpClient 4.1 and higher, and did not happen until we upgrade to 4.1.3 > > > > (and now is continuing under 4.2.3). I will have to check on the > > > > upgrade path and see if she had been using a 4.0.x version before this, > > > > or if they jumped from a version that used a 3.x version. > > > > > > > > Mark > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Mark Claassen [mailto:[email protected]] > > > > Sent: Friday, January 18, 2013 1:47 PM > > > > To: 'HttpClient User Discussion' > > > > Subject: RE: Connection Reset errors > > > > > > > > Oh, I forgot to add that all my requests are POST requests. > > > > > > > > -----Original Message----- > > > > From: Mark Claassen [mailto:[email protected]] > > > > Sent: Friday, January 18, 2013 1:02 PM > > > > To: 'HttpClient User Discussion' > > > > Subject: RE: Connection Reset errors > > > > > > > > While she is testing the java.net stuff, I decided to look into the > > > > possibility of retrying the request. Looking at this more closely, I > > > > see that the error is on the reading of the data, and the write seems > > > > to succeed. This gives me something to check on, since I should be > > > > able to see whether or not the servlet actually got her request or not. > > > > > > > > However, is this a common manifestation of the Connection Reset error? > > > > The write succeeds (or fails silently) and then it is on the read the > > > > problem can be detected? Or does this mean that the socket was fine > > > > during the write operation, but then was somehow severed before the > > > > read? > > > > > > > > Mark > > > > > > > > -----Original Message----- > > > > From: Mark Claassen [mailto:[email protected]] > > > > Sent: Friday, January 18, 2013 12:06 PM > > > > To: 'HttpClient User Discussion' > > > > Subject: RE: Connection Reset errors > > > > > > > > Looks like the stale connection check is enabled by default anyway, so > > > > I must have been using it all along. > > > > > > > > > It may well be that HttpURLConnection silently retries failed > > > > > requests > > > > Is this a common thing to do, like for browsers? Granted, she is in > > > > our app more than most others, but I don't think she experiences issues > > > > with other applications. > > > > > > > > She is not getting a Connection Refused or some other unable to read > > > > exception, but a Connection Reset. The stale connection check passed, > > > > so HttpClient thinks this is a good connection, right? Or are there > > > > limits to how reliable a stale connection check can be? > > > > > > > > public boolean isStale() { > > > > if (!isOpen()) { > > > > return true; > > > > } > > > > if (isEof()) { > > > > return true; > > > > } > > > > try { > > > > this.inbuffer.isDataAvailable(1); > > > > return isEof(); > > > > } catch (SocketTimeoutException ex) { > > > > return false; > > > > } catch (IOException ex) { > > > > return true; > > > > } > > > > } > > > > > > > > > > > > -----Original Message----- > > > > From: Oleg Kalnichevski [mailto:[email protected]] > > > > Sent: Friday, January 18, 2013 11:24 AM > > > > To: HttpClient User Discussion > > > > Subject: Re: Connection Reset errors > > > > > > > > On Fri, 2013-01-18 at 10:17 -0500, Mark Claassen wrote: > > > > > Before I switched to using HttpClient years ago, I used > > > > > HttpsURLConnection. Not sure how things were going to turn out with > > > > > HttpClient, I kept that code in place and made it so I could use > > > > > either API. I had her switch back to using the java.net stuff the > > > > > other day and the problem appeared to go away. I have configured her > > > > > system to use the java.net connector again today. Before I make any > > > > > conclusions I need some more data from her on each connector. > > > > > However, the previous data seems to indicate that it is only an issue > > > > > when using HttpClient (originally 4.1.3, now 4.2.3). > > > > > > > > > > > > > It may well be that HttpURLConnection silently retries failed requests > > > > without your knowing about it. It certainly did that when I last > > > > touched it (which admittedly was a looooooong time ago), and that was > > > > precisely the reason I had to stop using it and to look for a better > > > > alternative. > > > > > > > > > > > > > Some other things of note: > > > > > - She is using JRE 6 Update 13. (There is a long story behind > > > > > this.) > > > > > - The application is launched via Webstart > > > > > - The SocketFactory is created with: fact = new > > > > > SSLSocketFactory(HttpsURLConnection.getDefaultSSLSocketFactory() > > > > > ,v > > > > > er > > > > > if > > > > > ier); > > > > > > > > > > > > > I do not thing any of these should matter. > > > > > > > > Oleg > > > > > > > > > -----Original Message----- > > > > > From: Oleg Kalnichevski [mailto:[email protected]] > > > > > Sent: Friday, January 18, 2013 6:17 AM > > > > > To: HttpClient User Discussion > > > > > Subject: Re: Connection Reset errors > > > > > > > > > > On Thu, 2013-01-17 at 16:41 -0500, Mark Claassen wrote: > > > > > > Thanks for the explanation. > > > > > > > pro-actively evicting expired connections and connections > > > > > > > that have been idle > > > > > > Seems like just doing the stale check would more reliable, and > > > > > > speed does not seem to be a factory (especially with 4.2.3) > > > > > > > > > > > > I got in configured and distributed to this user and she is > > > > > > still having the same problem. Any other ideas? I double > > > > > > checked my code, and it looks like everything is correct. > > > > > > (She is not using a > > > > > > proxy.) > > > > > > > > > > > > > > > > This can also be genuine network stability issues. Essentially, > > > > > connection resets are inevitable and robust HTTP services have to be > > > > > prepared to deal with them (most likely by re-trying idempotent > > > > > methods automatically and reporting the problem back to the caller > > > > > otherwise). > > > > > > > > > > Oleg > > > > > > > > > > > > > > > > > > > > > > Method that makes the request (from multiple threads) > > > > > > { > > > > > > response = httpClient.execute(request); > > > > > > } > > > > > > Bulk of configuration method (MAX is 10, > > > > > > CONNECTION_ESTABLISH_TIMEOUT = 10) > > > > > > { > > > > > > HttpParams params = new BasicHttpParams(); > > > > > > > > > > > > params.setIntParameter(AllClientPNames.SO_TIMEOUT, (int) new > > > > > > TimeSpan(45, TimeUnit.SECONDS).convertTo(TimeUnit.MILLISECONDS)); > > > > > > > > > > > > params.setIntParameter(AllClientPNames.CONNECTION_TIMEOUT, > > > > > > CONNECTION_ESTABLISH_TIMEOUT * 1000); > > > > > > > > > > > > params.setIntParameter(AllClientPNames.SOCKET_BUFFER_SIZE, > > > > > > DEFAULT_HTTP_SOCKET_BUFFER_SIZE); > > > > > > > > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_CHECK, > > > > > > Boolean.TRUE); > > > > > > configureProxy(params,protocol); > > > > > > > > > > > > connectionManager = new > > > > > > PoolingClientConnectionManager(schemeRegistry); > > > > > > connectionManager.setDefaultMaxPerRoute(MAX); > > > > > > connectionManager.setMaxTotal(MAX); > > > > > > > > > > > > httpClient = new > > > > > > DefaultHttpClient(connectionManager, params); > > > > > > } > > > > > > private void configureProxy(HttpParams params, String scheme) { > > > > > > Proxy proxy = getProxy(); > > > > > > if (proxy.type() != Proxy.Type.DIRECT) { > > > > > > InetSocketAddress address = (InetSocketAddress) > > > > > > proxy.address(); > > > > > > //SSL or not, we set the proxy up as HTTP > > > > > > HttpHost httpProxy = new > > > > > > HttpHost(address.getHostName(), address.getPort(), SCHEME_HTTP); > > > > > > > > > > > > params.setParameter(AllClientPNames.DEFAULT_PROXY, httpProxy); > > > > > > LogManager.log("Setting proxy in HostConfig to > > > > > > " + address, Level.INFO); > > > > > > if (!scheme.equals(SCHEME_HTTP)) { > > > > > > //If the scheme is HTTPS, we need to > > > > > > register an HTTP scheme for the proxy. > > > > > > schemeRegistry.register(new > > > > > > Scheme(SCHEME_HTTP,address.getPort(),PlainSocketFactory.getSocketFactory())); > > > > > > } > > > > > > } > > > > > > > > > > > > } > > > > > > > > > > > > -----Original Message----- > > > > > > From: Oleg Kalnichevski [mailto:[email protected]] > > > > > > Sent: Thursday, January 17, 2013 7:21 AM > > > > > > To: HttpClient User Discussion > > > > > > Subject: Re: Connection Reset errors > > > > > > > > > > > > On Wed, 2013-01-16 at 15:22 -0500, Mark Claassen wrote: > > > > > > > > > > > > Basically 'connection reset' errors are a direct result of a > > > > > > general limitation of the blocking I/O model: when not blocked in > > > > > > an I/O operation there is no way for the socket to get notified of > > > > > > the opposite endpoint terminating the connection. While kept alive > > > > > > in the pool blocking HTTP connections can get stale. However, the > > > > > > only way to find it out is to try to attempt an I/O operation on > > > > > > such connection. > > > > > > > > > > > > > I noticed that there is a stale connection check: > > > > > > > params.setBooleanParameter(AllClientPNames.STALE_CONNECTION_ > > > > > > > CH > > > > > > > EC > > > > > > > K, > > > > > > > Boolean.TRUE); > > > > > > > > > > > > > > Is this something that I should be using? I am connecting to the > > > > > > > same source over and over again. > > > > > > > > > > > > > > Mark > > > > > > > > > > > > > > > > > > > Yes, this is what you should be using. Another alternative would be > > > > > > pro-actively evicting expired connections and connections that have > > > > > > been idle longer than a given period of time. > > > > > > > > > > > > Oleg > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Mark Claassen [mailto:[email protected]] > > > > > > > Sent: Wednesday, January 16, 2013 3:07 PM > > > > > > > To: 'HttpClient User Discussion' > > > > > > > Subject: Connection Reset errors > > > > > > > > > > > > > > I have a user getting a lot of Connection Reset errors. I > > > > > > > did not think this had do to with HttpClient, so much as > > > > > > > other factors on her machine and the server. However, I did a > > > > > > > bit of searching and see that people have commented on this error > > > > > > > with HttpClient before. I just upgraded to HttpClient 4.2.3 to > > > > > > > see if that would help at all, and it didn't. > > > > > > > > > > > > > > This user is fine for a while, and then will get the > > > > > > > Connection Reset error. She then just retries the request, and > > > > > > > it works. Any tips on how to resolve this would be greatly > > > > > > > appreciated. > > > > > > > > > > > > > > Thanks, > > > > > > > Mark > > > > > > > > > > > > > > We are using Tomcat 7.0.27, and the Connector is configured like > > > > > > > this: > > > > > > > <Connector > > > > > > > port="5502" > > > > > > > > > > > > > > protocol="org.apache.coyote.http11.Http11NioProtocol" > > > > > > > server="Unknown" > > > > > > > connectionLinger="0" > > > > > > > socket.soTimeout="300000" > > > > > > > SSLEnabled="true" > > > > > > > maxThreads="150" > > > > > > > scheme="https" > > > > > > > secure="true" > > > > > > > clientAuth="false" > > > > > > > keystoreFile="-----" > > > > > > > keystoreType="PKCS12" > > > > > > > keystorePass="-----" > > > > > > > sslProtocol="TLS" > > > > > > > > > > > > > > ciphers="SSL_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_AES_128_CBC_S > > > > > > > HA ,T LS _D HE > > > > > > > _RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES > > > > > > > _128_CBC_SHA,SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_ > > > > > > > 3D ES _E DE _C BC_SHA,SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"/> > > > > > > > > > > > > > > > > > > > > > ---- (1) ---- Throwable - Class (class java.net.SocketException) > > > > > > > Message (Connection reset) > > > > > > > at java.net.SocketInputStream.read(Unknown Source) > > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown > > > > > > > Source) > > > > > > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown > > > > > > > Source) > > > > > > > at > > > > > > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown > > > > > > > Source) > > > > > > > at > > > > > > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown > > > > > > > Source) > > > > > > > at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown > > > > > > > Source) > > > > > > > at > > > > > > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) > > > > > > > at > > > > > > > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) > > > > > > > at > > > > > > > org.apache.http.impl.io.AbstractSessionInputBuffer.read(AbstractSessionInputBuffer.java:212) > > > > > > > at > > > > > > > org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:177) > > > > > > > at > > > > > > > org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream. > > > > > > > ja > > > > > > > va:138) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------ > > > > > > > -- > > > > > > > -- > > > > > > > -- > > > > > > > -- > > > > > > > - To unsubscribe, e-mail: > > > > > > > [email protected] > > > > > > > For additional commands, e-mail: > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------ > > > > > > > -- > > > > > > > -- > > > > > > > -- > > > > > > > -- > > > > > > > - To unsubscribe, e-mail: > > > > > > > [email protected] > > > > > > > For additional commands, e-mail: > > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > -- > > > > > > -- > > > > > > -- > > > > > > - To unsubscribe, e-mail: > > > > > > [email protected] > > > > > > For additional commands, e-mail: > > > > > > [email protected] > > > > > > > > > > > > > > > > > > -------------------------------------------------------------- > > > > > > -- > > > > > > -- > > > > > > -- > > > > > > - To unsubscribe, e-mail: > > > > > > [email protected] > > > > > > For additional commands, e-mail: > > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > > > > > > ---------------------------------------------------------------- > > > > > -- > > > > > -- > > > > > - To unsubscribe, e-mail: > > > > > [email protected] > > > > > For additional commands, e-mail: > > > > > [email protected] > > > > > > > > > > > > > > > ---------------------------------------------------------------- > > > > > -- > > > > > -- > > > > > - To unsubscribe, e-mail: > > > > > [email protected] > > > > > For additional commands, e-mail: > > > > > [email protected] > > > > > > > > > > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: > > > > [email protected] > > > > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: > > > > [email protected] > > > > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: > > > > [email protected] > > > > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: > > > > [email protected] > > > > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: > > > > [email protected] > > > > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: > > > > [email protected] > > > > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > > ------------------------------------------------------------------ > > > > -- > > > > - To unsubscribe, e-mail: > > > > [email protected] > > > > For additional commands, e-mail: > > > > [email protected] > > > > > > > > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > -------------------------------------------------------------------- > > > - To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
