On Tue, Apr 14, 2009 at 11:39:01PM +0100, Sam Crawford wrote: > Oleg, > > Thanks for the quick reply, as always. > > I'm afraid I haven't been able to get a wire log of the extreme scenario I > highlighted earlier (I have a tcpdump capture of it from last night, but I > doubt that's of much use to you). > > I have however got a wire log of the issue as it's manifesting itself this > evening. Inspection of the tcpdump traces reveal that this particular > webserver is sending FINs for idle TCP connections after 30 seconds, whereas > many of the other servers we're dealing with are all much higher (in the > order of 5 minutes or so). This server does not reply with a Keep-Alive > header, despite Connection: Keep-Alive being sent as a request header (I > appreciate it's under no obligation to obey the client's wishes). > > I've attached the wire log anyway, but I think for the moment I'm going to > have to reduce my IdleConnectionEvictor to run at least every 30 seconds (it > runs every two minutes at present). > > A couple of closing questions if I may: > > 1) Is there any way with the ClientConnectionManager to perform > closeIdleConnections only for a specific HttpRoute? If the answer is > "override the closeIdleConnections method and implement it" then that's fine > by me :-)
A better approach could be implementing a custom ConnectionKeepAliveStrategy and setting a lower keep-alive timeout for known naughty servers. > 2) You said the Stale Connection check is (in most cases) "evil". Is there > any docs detailing exactly what it does and what we will lose if we disable > it? (I believe it's enabled by default). The tiny speed hit is not a massive > issue for us at the moment (but could be later). > Stale connection checking simply cannot be 100% reliable as there is always a window of time between a successful stale check and the request execution when the connection can go stale on unsuspecting HttpClient. A well designed application has to have a recovery code for such situations anyways, which kind of makes stale connection checking pretty much pointless. The performance is not that tiny. 30 milliseconds for small payloads that take only 3-5 milliseconds to execute is a lot. Oleg > Thanks again, > > Sam > > > > 2009/4/14 Oleg Kalnichevski <[email protected]> > > > Sam Crawford wrote: > > > >> Afternoon all, > >> A few months back we had an issue with handling half closed TCP > >> connections > >> with HttpClient, and at the time I was advised to include something akin > >> to > >> the IdleConnectionEvictor - which we did and it's working very nicely in > >> nearly all scenarios. > >> > >> However, in the past few days we've encountered a few WebLogic based hosts > >> that aren't playing fair. > >> > >> The following is one (extreme) example of the issue we're encountering: > >> > >> Time (ms) TCP action > >> 0.0000 Client > Server [SYN] > >> 0.5634 Server > Client [SYN,ACK] > >> 1.2092 Client > Server [ACK] <-- TCP session established > >> 312.5276 Server > Client [FIN,ACK] > >> 313.1309 Client > Server [ACK] > >> 401.5089 Client > Server [HTTP POST /blah] > >> 403.2986 Server > Client [RST] > >> > >> In the above example, the server closes its side of the connection only > >> 300ms after establishment (by sending the FIN). (As an aside I'm curious > >> as > >> to why HttpClient is taking 400ms after the TCP connection has been > >> established to send the request - any suggestions are also much > >> appreciated, > >> but this doesn't happen often). > >> > >> > > This does not sound right. The stale connection check may cause a 20 to 30 > > millisecond delay (and generally should be avoided) but this is a bit too > > much. Can you produce a wire / context log of the session? > > > > > > But the above is an extreme example. We see other cases where the WebLogic > >> server is closing the connection of a keep-alive connection around 10-15 > >> seconds after the last request. > >> > > > > Does the server send a 'Keep-alive' header with the response? > > > > > > Our IdleConnectionEvictor doesn't run that > > > >> often, so we end up with unusable connections. We could just run > >> IdleConnectionEvictor more often, but that's not really desirable. > >> > >> I'm going to be digging into the WebLogic side of things this afternoon > >> (to > >> see if there's any limits we can modify there), but it does seem as though > >> there should be a nice way for HttpClient to detect such cases. I've got > >> stale connection checking enabled already by the way. > >> > >> > > Stale connection checking is (in most cases) evil and should be avoided. > > > > I'm interested in any feedback/ideas here! I can include a wire capture as > >> an example if it would be helpful. > >> > >> > > A wire / context log that correlates with the TCP dump would be great. > > > > Oleg > > > > Thanks again, > >> > >> Sam > >> > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
