For anyone else's reference who might be following this thread... I've extended the DefaultConnectionKeepAliveStrategy to allow a default timeout to be specified (which is used if the Keep-Alive header doesn't exist).
I've then removed the connectionManager.closeIdleConnections(30000, TimeUnit.SECONDS) line in my IdleConnectionEvictor and now rely upon the connectionManager.closeExpiredConnections() method to handle the closure of connections. Seems to be working well so far. Thanks, Sam 2009/4/15 Sam Crawford <[email protected]> > Perfect, thanks. I think the custom ConnectionKeepAliveStrategy is the way > forward, as you suggest. > > I'll have a tinker with disabling stale connection checking too and see how > it affects performance. > > Thanks again, > > Sam > > > 2009/4/15 Oleg Kalnichevski <[email protected]> > > On Tue, Apr 14, 2009 at 11:39:01PM +0100, Sam Crawford wrote: >> > Oleg, >> > >> > Thanks for the quick reply, as always. >> > >> > I'm afraid I haven't been able to get a wire log of the extreme scenario >> I >> > highlighted earlier (I have a tcpdump capture of it from last night, but >> I >> > doubt that's of much use to you). >> > >> > I have however got a wire log of the issue as it's manifesting itself >> this >> > evening. Inspection of the tcpdump traces reveal that this particular >> > webserver is sending FINs for idle TCP connections after 30 seconds, >> whereas >> > many of the other servers we're dealing with are all much higher (in the >> > order of 5 minutes or so). This server does not reply with a Keep-Alive >> > header, despite Connection: Keep-Alive being sent as a request header (I >> > appreciate it's under no obligation to obey the client's wishes). >> > >> > I've attached the wire log anyway, but I think for the moment I'm going >> to >> > have to reduce my IdleConnectionEvictor to run at least every 30 seconds >> (it >> > runs every two minutes at present). >> > >> > A couple of closing questions if I may: >> > >> > 1) Is there any way with the ClientConnectionManager to perform >> > closeIdleConnections only for a specific HttpRoute? If the answer is >> > "override the closeIdleConnections method and implement it" then that's >> fine >> > by me :-) >> >> A better approach could be implementing a custom >> ConnectionKeepAliveStrategy and setting a lower keep-alive timeout for >> known naughty servers. >> >> >> > 2) You said the Stale Connection check is (in most cases) "evil". Is >> there >> > any docs detailing exactly what it does and what we will lose if we >> disable >> > it? (I believe it's enabled by default). The tiny speed hit is not a >> massive >> > issue for us at the moment (but could be later). >> > >> >> Stale connection checking simply cannot be 100% reliable as there is >> always a window of time between a successful stale check and the request >> execution when the connection can go stale on unsuspecting HttpClient. >> A well designed application has to have a recovery code for such >> situations anyways, which kind of makes stale connection checking pretty >> much pointless. >> >> The performance is not that tiny. 30 milliseconds for small payloads >> that take only 3-5 milliseconds to execute is a lot. >> >> Oleg >> >> > Thanks again, >> > >> > Sam >> > >> > >> > >> > 2009/4/14 Oleg Kalnichevski <[email protected]> >> > >> > > Sam Crawford wrote: >> > > >> > >> Afternoon all, >> > >> A few months back we had an issue with handling half closed TCP >> > >> connections >> > >> with HttpClient, and at the time I was advised to include something >> akin >> > >> to >> > >> the IdleConnectionEvictor - which we did and it's working very nicely >> in >> > >> nearly all scenarios. >> > >> >> > >> However, in the past few days we've encountered a few WebLogic based >> hosts >> > >> that aren't playing fair. >> > >> >> > >> The following is one (extreme) example of the issue we're >> encountering: >> > >> >> > >> Time (ms) TCP action >> > >> 0.0000 Client > Server [SYN] >> > >> 0.5634 Server > Client [SYN,ACK] >> > >> 1.2092 Client > Server [ACK] <-- TCP session >> established >> > >> 312.5276 Server > Client [FIN,ACK] >> > >> 313.1309 Client > Server [ACK] >> > >> 401.5089 Client > Server [HTTP POST /blah] >> > >> 403.2986 Server > Client [RST] >> > >> >> > >> In the above example, the server closes its side of the connection >> only >> > >> 300ms after establishment (by sending the FIN). (As an aside I'm >> curious >> > >> as >> > >> to why HttpClient is taking 400ms after the TCP connection has been >> > >> established to send the request - any suggestions are also much >> > >> appreciated, >> > >> but this doesn't happen often). >> > >> >> > >> >> > > This does not sound right. The stale connection check may cause a 20 >> to 30 >> > > millisecond delay (and generally should be avoided) but this is a bit >> too >> > > much. Can you produce a wire / context log of the session? >> > > >> > > >> > > But the above is an extreme example. We see other cases where the >> WebLogic >> > >> server is closing the connection of a keep-alive connection around >> 10-15 >> > >> seconds after the last request. >> > >> >> > > >> > > Does the server send a 'Keep-alive' header with the response? >> > > >> > > >> > > Our IdleConnectionEvictor doesn't run that >> > > >> > >> often, so we end up with unusable connections. We could just run >> > >> IdleConnectionEvictor more often, but that's not really desirable. >> > >> >> > >> I'm going to be digging into the WebLogic side of things this >> afternoon >> > >> (to >> > >> see if there's any limits we can modify there), but it does seem as >> though >> > >> there should be a nice way for HttpClient to detect such cases. I've >> got >> > >> stale connection checking enabled already by the way. >> > >> >> > >> >> > > Stale connection checking is (in most cases) evil and should be >> avoided. >> > > >> > > I'm interested in any feedback/ideas here! I can include a wire >> capture as >> > >> an example if it would be helpful. >> > >> >> > >> >> > > A wire / context log that correlates with the TCP dump would be great. >> > > >> > > Oleg >> > > >> > > Thanks again, >> > >> >> > >> Sam >> > >> >> > >> >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: [email protected] >> > > For additional commands, e-mail: [email protected] >> > > >> > > >> >> >> > >> > --------------------------------------------------------------------- >> > To unsubscribe, e-mail: [email protected] >> > For additional commands, e-mail: [email protected] >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >
