Perfect, thanks. I think the custom ConnectionKeepAliveStrategy is the way forward, as you suggest.
I'll have a tinker with disabling stale connection checking too and see how it affects performance. Thanks again, Sam 2009/4/15 Oleg Kalnichevski <[email protected]> > On Tue, Apr 14, 2009 at 11:39:01PM +0100, Sam Crawford wrote: > > Oleg, > > > > Thanks for the quick reply, as always. > > > > I'm afraid I haven't been able to get a wire log of the extreme scenario > I > > highlighted earlier (I have a tcpdump capture of it from last night, but > I > > doubt that's of much use to you). > > > > I have however got a wire log of the issue as it's manifesting itself > this > > evening. Inspection of the tcpdump traces reveal that this particular > > webserver is sending FINs for idle TCP connections after 30 seconds, > whereas > > many of the other servers we're dealing with are all much higher (in the > > order of 5 minutes or so). This server does not reply with a Keep-Alive > > header, despite Connection: Keep-Alive being sent as a request header (I > > appreciate it's under no obligation to obey the client's wishes). > > > > I've attached the wire log anyway, but I think for the moment I'm going > to > > have to reduce my IdleConnectionEvictor to run at least every 30 seconds > (it > > runs every two minutes at present). > > > > A couple of closing questions if I may: > > > > 1) Is there any way with the ClientConnectionManager to perform > > closeIdleConnections only for a specific HttpRoute? If the answer is > > "override the closeIdleConnections method and implement it" then that's > fine > > by me :-) > > A better approach could be implementing a custom > ConnectionKeepAliveStrategy and setting a lower keep-alive timeout for > known naughty servers. > > > > 2) You said the Stale Connection check is (in most cases) "evil". Is > there > > any docs detailing exactly what it does and what we will lose if we > disable > > it? (I believe it's enabled by default). The tiny speed hit is not a > massive > > issue for us at the moment (but could be later). > > > > Stale connection checking simply cannot be 100% reliable as there is > always a window of time between a successful stale check and the request > execution when the connection can go stale on unsuspecting HttpClient. > A well designed application has to have a recovery code for such > situations anyways, which kind of makes stale connection checking pretty > much pointless. > > The performance is not that tiny. 30 milliseconds for small payloads > that take only 3-5 milliseconds to execute is a lot. > > Oleg > > > Thanks again, > > > > Sam > > > > > > > > 2009/4/14 Oleg Kalnichevski <[email protected]> > > > > > Sam Crawford wrote: > > > > > >> Afternoon all, > > >> A few months back we had an issue with handling half closed TCP > > >> connections > > >> with HttpClient, and at the time I was advised to include something > akin > > >> to > > >> the IdleConnectionEvictor - which we did and it's working very nicely > in > > >> nearly all scenarios. > > >> > > >> However, in the past few days we've encountered a few WebLogic based > hosts > > >> that aren't playing fair. > > >> > > >> The following is one (extreme) example of the issue we're > encountering: > > >> > > >> Time (ms) TCP action > > >> 0.0000 Client > Server [SYN] > > >> 0.5634 Server > Client [SYN,ACK] > > >> 1.2092 Client > Server [ACK] <-- TCP session > established > > >> 312.5276 Server > Client [FIN,ACK] > > >> 313.1309 Client > Server [ACK] > > >> 401.5089 Client > Server [HTTP POST /blah] > > >> 403.2986 Server > Client [RST] > > >> > > >> In the above example, the server closes its side of the connection > only > > >> 300ms after establishment (by sending the FIN). (As an aside I'm > curious > > >> as > > >> to why HttpClient is taking 400ms after the TCP connection has been > > >> established to send the request - any suggestions are also much > > >> appreciated, > > >> but this doesn't happen often). > > >> > > >> > > > This does not sound right. The stale connection check may cause a 20 to > 30 > > > millisecond delay (and generally should be avoided) but this is a bit > too > > > much. Can you produce a wire / context log of the session? > > > > > > > > > But the above is an extreme example. We see other cases where the > WebLogic > > >> server is closing the connection of a keep-alive connection around > 10-15 > > >> seconds after the last request. > > >> > > > > > > Does the server send a 'Keep-alive' header with the response? > > > > > > > > > Our IdleConnectionEvictor doesn't run that > > > > > >> often, so we end up with unusable connections. We could just run > > >> IdleConnectionEvictor more often, but that's not really desirable. > > >> > > >> I'm going to be digging into the WebLogic side of things this > afternoon > > >> (to > > >> see if there's any limits we can modify there), but it does seem as > though > > >> there should be a nice way for HttpClient to detect such cases. I've > got > > >> stale connection checking enabled already by the way. > > >> > > >> > > > Stale connection checking is (in most cases) evil and should be > avoided. > > > > > > I'm interested in any feedback/ideas here! I can include a wire > capture as > > >> an example if it would be helpful. > > >> > > >> > > > A wire / context log that correlates with the TCP dump would be great. > > > > > > Oleg > > > > > > Thanks again, > > >> > > >> Sam > > >> > > >> > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
