On Tue, 2009-10-13 at 23:17 +0100, Tony Poppleton wrote: > Hi Oleg, > > Thanks for the fast and excellent response, am implementing your > suggested changes now. > > Incidentally, the problem I observed where it sometimes didn't appear to > retry, was in fact due to problem 2 retrying, and then immediately being > followed by problem 1. The stack trace for problem 1 was on the console > stderr stream, so wasn't appearing the logs, which is why I thought it > was mysteriously exiting before retrying the full 5 times. So case > closed on that one, it was my misinterpretation. > > Regarding the stale connection check, I am sure you have already > considered this, but what is the problem with doing the following: > - removing the stale connection check (or at least disabling by default) > - making the default http client use a retry handler that silently > retries the IO exceptions
Tony, The trouble is HttpClient cannot simply silently retry _any_ I/O exception because the knowledge whether the failed method is safe to retry is application specific. > - making the default http client have an IdleConnectionMonitorThread as > per your reference [2], that cleans up the stale connections > This can be done, but is not enough. > Wouldn't that cover all the bases and be just as effective at managing > connections as with the stale connection check enabled, and no one would > report any problems, and get a free performance benefit? Or is there > something else that the stale connection check is doing that makes it a > better choice? > The stale connection check is a lesser evil compared to the disastrous effect an automatic retrial of non-idempotent methods can have on transactional applications. I think the best we can do is to add a section to the HttpClient tutorial describing in details what needs to be done in order to make the stale connection check unnecessary. Cheers Oleg > Thanks again, > Tony > > > Oleg Kalnichevski wrote: > > > > Tony Poppleton wrote: > >> Hi, > >> > >> I am eager to squeeze the most performance out of my usage of > >> HttpClient 4.0 and have been reading the archives where it was > >> suggested to disable the stale connection check. I have done this, > >> and it does indeed significantly improve performance, however I am > >> now occasionally getting the following two exceptions: > >> > >> 1. Exception in thread "main" java.net.SocketException: Connection > >> reset by peer: socket write error > >> at java.net.SocketOutputStream.socketWrite0(Native Method) > >> at > >> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92) > >> at java.net.SocketOutputStream.write(SocketOutputStream.java:136) > >> at > >> org.apache.http.impl.io.AbstractSessionOutputBuffer.flushBuffer(AbstractSessionOutputBuffer.java:106) > >> > >> > >> at > >> org.apache.http.impl.io.AbstractSessionOutputBuffer.flush(AbstractSessionOutputBuffer.java:113) > >> > >> > >> at > >> org.apache.http.impl.AbstractHttpClientConnection.doFlush(AbstractHttpClientConnection.java:260) > >> > >> > >> at > >> org.apache.http.impl.SocketHttpClientConnection.close(SocketHttpClientConnection.java:248) > >> > >> > >> at > >> org.apache.http.impl.conn.DefaultClientConnection.close(DefaultClientConnection.java:154) > >> > >> > >> at > >> org.apache.http.impl.conn.AbstractPooledConnAdapter.close(AbstractPooledConnAdapter.java:131) > >> > >> > >> at > >> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:130) > >> > >> > >> at > >> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:447) > >> > >> > >> at > >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:641) > >> > >> > >> at > >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:730) > >> > >> > >> at > >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:708) > >> > >> > >> at > >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:699) > >> > >> > >> ... > >> > >> 2. DefaultRequestDirector:455 - I/O exception > >> (org.apache.http.NoHttpResponseException) caught when processing > >> request: The target server failed to respond > >> DefaultRequestDirector:462 - Retrying request > >> > >> The first one is problematic because the program terminates (I don't > >> handle the exception anywhere). The second one usually retries as I > >> am using the DefaultHttpRequestRetryHandler, but on occasion the > >> program has just exited too (without doing the 5 retries that I have > >> requested - I am trying to capture this case in a debugger so I can > >> get a clue as to what is happening, will post an update if I find > >> anything). The second error is far more frequent than the first, for > >> example today I have seen the second error about 50 times, but the > >> 1st error only once. > >> > >> Anyway, I presume that both these errors are only now apparent > >> because I have disabled the stale connection check. > >> > > > > Tony, > > > > The best strategy to tackle the stale connection issue is very simple: > > do not let connections go stale in the first place by evicting > > connections from the connection pool that have been idle for too long. > > For details see [1] and [2]. > > > >> What advice can you give me on how to recover from the first > >> problem? It is important that I don't re-post my requests if they > >> did manage to get through before the error occurred. > >> > > > > As long as the method being executed can be considered idempotent [3] > > it is safe simply to retry the request using a custom retry handler > > [4] and be done with it. However, if your application uses HTTP > > transport to execute transactional business logic (like placing an > > order for an item) you may have a problem. Actually you do have a > > problem anyways because HTTP transport is not transaction safe and > > HTTP connections can and do fail in a middle of a transaction (the > > request gets executed, an order gets placed but the response is never > > delivered back to the client). To sum things up this is an application > > design issue, and not an issue of the HTTP transport. > > > >> For the second one, is it 100% safe to resume from that error? > > > > Yes, it is. > > > > > > Is it > >> fully equivalent in terms of network communications as having the > >> stale connection check enabled? > >> > > > > Pretty much. > > > >> Out of interest, what is the history of the stale connection check? > >> From reading the archives it appears it is a relic of an older > >> HttpClient, and its usage is not recommended anymore. Is this correct? > > > > Yes, it is > > > >> If so, why isn't it just removed from the latest version, and > >> substituted with a robust retry handler that can deal with all the > >> consequences? > >> > > > > Firstly, there is no such thing as a robust retry handler for HTTP > > transport for the reason given above. HTTP request retrial logic is > > always application specific. Secondly, there are tons of people out > > there who do not really understand what persistent connections are all > > about and start complaining about HttpClient nor working if they > > occasionally see a request failing with an I/O exception. It is easier > > for the sake of everyone's sanity to have the damn stale check on per > > default. > > > > Cheers > > > > Oleg > > > > [1] > > http://hc.apache.org/httpcomponents-client/tutorial/html/connmgmt.html#d4e638 > > > > > > [2] > > http://hc.apache.org/httpcomponents-client/tutorial/html/connmgmt.html#d4e645 > > > > > > [3] > > http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e255 > > > > > > [4] > > http://hc.apache.org/httpcomponents-client/tutorial/html/fundamentals.html#d4e280 > > > > > > > >> Many thanks, > >> Tony > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > > > --------------------------------------------------------------------------------------- > > > > > > Orange vous informe que cet e-mail a ete controle par l'anti-virus > > mail. Aucun virus connu a ce jour par nos services n'a ete detecte. > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
