> I cannot think of any good reason to continue using HttpCore older than > 4.0.1. Upgrade would be quite recommended. Oleg, thanks a lot for your quick response and suggestion. I think we will update our integration system to 4.0.1 first, and see how it runs there.
> The process of an I/O reactor shutdown is quite complex and therefore > lots of things can go sideways if a particular I/O worker terminates > abnormally due to a runtime or a fatal I/O error. To help deal with > postmortem analysis I/O reactors maintain so called exception log that > contains all exceptions thrown in the process of reactor shutdown > including the one that triggered it. > > http://hc.apache.org/httpcomponents-core/tutorial/html/nio.html#d0e1287 > Whenever an I/O reactor terminates it is advisable to examine the audit > log and if it contains any entries print them out to the application log. Maybe I have to have a closer look at the code level for both Synapse and http Core to get a better understanding... So please excuse, if my following thoughts are incorrect and rather correct them. Don't we generally have to differentiate between 1) situations the code using the http nio library initiates/requests a IOReactor shutdown and 2) those, where the http nio library does this on its own based on an IOReactorException? For the first case I now understand, that one should always check the audit log in order to also find out if there have been exceptions during the shutdown process. AFAIK Synapse currently does not follow this suggestion. For the second case I understood that the client code has a chance to intercept the exception handling by providing an IOReactorExceptionHandler which can attempt to decide which exceptions shall be considered recoverable and which not. This is the place where the client can/should also log the cause of both IOExceptions and RuntimeExceptions. For Synapse this is the case. Anyway it looks like there are situations where IOReactorExceptions are thrown at other places without a chance for the client to "intercept". In this cases the client code finds out that the IOReactor has been shutdown or is in the process of being shutdown (IllegalStateException while calling connect()) and can just attempt to restart it. Is it possible, that the actual cause of an IOReactor shutdown will be logged a couple of time after it actually occurred? Or to be more concrete, first error output: ERROR 2009-12-02 10:50:34,802 [HttpClientWorker-97][][] org.apache.synapse.core.axis2.Axis2Sender 'Unexpected error during sending message out' java.lang.IllegalStateException: I/O reactor has been shut down at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:176) at org.apache.synapse.transport.nhttp.HttpCoreNIOSender.sendAsyncRequest(HttpCoreNIOSender.java:392) [... a few 100 log lines later ...] FATAL 2009-12-02 10:50:35,031 [HttpCoreNIOSender][][] org.apache.synapse.transport.nhttp.HttpCoreNIOSender 'Encountered an I/O error: Failure registering channel with the selector' org.apache.http.nio.reactor.IOReactorException: Failure registering channel with the selector at org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:220) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:153) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:70) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:324) at java.lang.Thread.run(Thread.java:595) Caused by: java.nio.channels.ClosedChannelException at java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:252) at org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:217) ... 4 more Was in this case the java.nio.channels.ClosedChannelException the cause of the IOReactor shutdown? Was it correct to terminate the Reactor? If some code within the core nio library decides to throw an IOReactorException could it make sense to log the cause immediately or are there cases where the code at higher levels may still consider it otherwise. Anyway, if I'm not wrong and all of the above makes at least partially sense, than it looks to me that the fix in http://issues.apache.org/jira/browse/HTTPCORE-169 which went in beta-3 should have prevented the IOReactor restart. Oleg, please correct me if my understanding is wrong. I'm not very familiar with the code. Regards, Eric
