On Thu, Jul 30, 2009 at 10:47:06AM -0700, Gerry Woods wrote: > Oleg, > Thanks very much for the reply. I mentioned the SO_TIMEOUT setting just to > explain the condition that led to the exception and seems to have ended up > leaving the socket in a bad state. I understand too that CLOSE_WAIT is due > to the server having closed the socket but httpclient has not. We do run the > idle connection timeout thread, and unfortunately in this case it doesn't > seem to help. > > The focus of my attention has been the HttpConnection.closeSocketAndStreams() > method which is where the socket seems to end up being disposed of both by > the IdleConnectionTimeoutThread and all other logic that closes the > connection. In 3.0.1 and 3.1, this simply calls close() on the streams and > sockets. In 4.x however, the SocketHttpClientConnection.close() method does > perform a shutdownOutput()/shutdownInput(). Some snippets of information > I've found seem to indicate that on some socket stacks close() is not enough > to actually close a socket and make the CLOSE_WAIT->LAST_ACK transition, and > that a shutdown is necessary. This seems to jive with our experience here. I > was assuming that the change in behavior from 3.x to 4.x was the result of > lessons learned and that maybe this had a bearing on our scenario. If not, > we can run some experiments and try to isolate it further. Eventually of > course we would like to migrate to the newer, cleaner, APIs. But in the > meantime if anything occurs to you or anyone else about this condition, > please let me know. > Thanks again. > Gerry >
Gerry, The Socket#shutdown*() methods have become available since Java 1.3 only. HttpClient 3.x cannot make use of those methods due to java 1.2.2 compatibility requirement. http://java.sun.com/j2se/1.4.2/docs/api/java/net/Socket.html#shutdownInput%28%29 http://java.sun.com/j2se/1.4.2/docs/api/java/net/Socket.html#shutdownOutput%28%29 I am still not convinced Socket#close() does not fully close the underlying TCP connection. I think shutdown methods are needed only if one wants to put a socket to a half-closed state (read-only / write-only). Oleg > -----Original Message----- > From: Oleg Kalnichevski [mailto:[email protected]] > Sent: Thursday, July 30, 2009 3:08 AM > To: HttpClient User Discussion > Subject: Re: CLOSE_WAIT on Linux > > On Wed, Jul 29, 2009 at 09:36:20PM -0700, Gerry Woods wrote: > > I haven't posted here before so let me start by thanking you guys for the > > great work. > > > > We are using 3.0.1 in some pretty heavy production environments. A recent > > issue arose with a customer who had trouble during SSL handshakes. The > > negotiation was timing out: > > > > [120009] java.net.SocketTimeoutException: Read timed out > > at java.net.SocketInputStream.socketRead0(Native Method) > > at java.net.SocketInputStream.read(Unknown Source) > > at com.sun.net.ssl.internal.ssl.InputRecord.a(Unknown > > Source) > > at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown > > Source) > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.a(Unknown > > Source) > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.j(Unknown > > Source) > > at com.sun.net.ssl.internal.ssl.SSLSocketImpl.b(Unknown > > Source) > > at > > com.sun.net.ssl.internal.ssl.SSLSocketImpl.startHandshake(Unknown Source) > > ... > > > > We're setting SO_TIMEOUT to 2 minutes. The customer reports that the > > machine that initiated the connection ended up with a lot of sockets in > > CLOSE_WAIT. From what I've been able to glean from some quick searches, it > > sounds like on some socket stacks, a shutdownOutput is required, and that > > close() is not enough to nudge the socket from CLOSE_WAIT to LAST_ACK. I > > notice that in the 4.x codebase, shutdownOutput is being used, but in the > > 3.x code it is not. Can anyone confirm that this is indeed the case for > > Linux? If so, is there a known workaround other than upgrading to 4.x? > > > > Thanks for any help you can offer, and thanks again for all the countless > > hours (and dollars!) your efforts have saved us. > > > > > > Gerry, > > As far as I know SO_TIMEOUT parameter and CLOSE_WAIT socket state are > completely unrelated. Usually sockets end up in the CLOSE_WAIT state when a > persistent connection is closed by the remote server while still kept alive in > the connection pool on the client. This is general problem with the blocking > I/O model and both HttpClient 3.x and 4.0 are affected by it. However, > HttpClient 4.0 is much better equipped to deal with the issue. In both cases > one needs to implement some sort of connection eviction policy to make sure > that persistent connections that have been idle for too long get closed and > evicted from the pool. > > http://hc.apache.org/httpcomponents-client/tutorial/html/ch02.html#d4e638 > > Hope this helps > > Oleg > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
