Hubert, Eric wrote:
I cannot think of any good reason to continue using HttpCore older than
4.0.1. Upgrade would be quite recommended.
Oleg, thanks a lot for your quick response and suggestion. I think we will
update our integration system to 4.0.1 first, and see how it runs there.
The process of an I/O reactor shutdown is quite complex and therefore
lots of things can go sideways if a particular I/O worker terminates
abnormally due to a runtime or a fatal I/O error. To help deal with
postmortem analysis I/O reactors maintain so called exception log that
contains all exceptions thrown in the process of reactor shutdown
including the one that triggered it.
http://hc.apache.org/httpcomponents-core/tutorial/html/nio.html#d0e1287
Whenever an I/O reactor terminates it is advisable to examine the audit
log and if it contains any entries print them out to the application log.
Maybe I have to have a closer look at the code level for both Synapse and http
Core to get a better understanding... So please excuse, if my following
thoughts are incorrect and rather correct them.
Don't we generally have to differentiate between
1) situations the code using the http nio library initiates/requests a IOReactor shutdown and
2) those, where the http nio library does this on its own based on an IOReactorException?
For the first case I now understand, that one should always check the audit log
in order to also find out if there have been exceptions during the shutdown
process. AFAIK Synapse currently does not follow this suggestion.
If the reactor shutdown was requested in an orderly fashion there is a
pretty good chance it will shut down cleanly and there will be nothing
to look at the audit log. The audit log usually comes handy when the
reactor shuts down due to an unexpected fatal error.
For the second case I understood that the client code has a chance to intercept the
exception handling by providing an IOReactorExceptionHandler which can attempt to decide
which exceptions shall be considered recoverable and which not. This is the place where
the client can/should also log the cause of both IOExceptions and RuntimeExceptions. For
Synapse this is the case. Anyway it looks like there are situations where
IOReactorExceptions are thrown at other places without a chance for the client to
"intercept". In this cases the client code finds out that the IOReactor has
been shutdown or is in the process of being shutdown (IllegalStateException while calling
connect()) and can just attempt to restart it.
This is one possibility. Alternatively the reactor thread can intercept
the fatal exception that caused the shutdown, optionally examine the
audit log for additional information, and based on the result of the
analysis choose to restart the reactor. HttpCoreNIOListener and
HttpCoreNIOSender classes would be the ones to look at
http://svn.apache.org/repos/asf/synapse/trunk/java/modules/transports/core/nhttp/src/main/java/org/apache/synapse/transport/nhttp/HttpCoreNIOListener.java
http://svn.apache.org/repos/asf/synapse/trunk/java/modules/transports/core/nhttp/src/main/java/org/apache/synapse/transport/nhttp/HttpCoreNIOSender.java
Is it possible, that the actual cause of an IOReactor shutdown will be logged a
couple of time after it actually occurred?
HttpCore does not do logging by itself. I do not know Synapse that well
to answer this question.
Or to be more concrete, first error output:
ERROR 2009-12-02 10:50:34,802 [HttpClientWorker-97][][]
org.apache.synapse.core.axis2.Axis2Sender 'Unexpected error during sending
message out'
java.lang.IllegalStateException: I/O reactor has been shut down
at
org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:176)
at
org.apache.synapse.transport.nhttp.HttpCoreNIOSender.sendAsyncRequest(HttpCoreNIOSender.java:392)
[... a few 100 log lines later ...]
FATAL 2009-12-02 10:50:35,031 [HttpCoreNIOSender][][]
org.apache.synapse.transport.nhttp.HttpCoreNIOSender 'Encountered an I/O error:
Failure registering channel with the selector'
org.apache.http.nio.reactor.IOReactorException: Failure registering channel
with the selector
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:220)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:153)
at
org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:70)
at
org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:324)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.nio.channels.ClosedChannelException
at
java.nio.channels.spi.AbstractSelectableChannel.configureBlocking(AbstractSelectableChannel.java:252)
at
org.apache.http.impl.nio.reactor.AbstractIOReactor.processNewChannels(AbstractIOReactor.java:217)
... 4 more
Was in this case the java.nio.channels.ClosedChannelException the cause of the
IOReactor shutdown?
Yes, it was.
Was it correct to terminate the Reactor?
No, it was not. ClosedChannelException should not be considered fatal.
It is a shame the exception is unchecked, though.
If some code within the core nio library decides to throw an IOReactorException
could it make sense to log the cause immediately or are there cases where the
code at higher levels may still consider it otherwise.
IOReactorException is considered fatal and should be propagated to the
I/O reactor thread unless discarded by the IOReactorExceptionHandler
Anyway, if I'm not wrong and all of the above makes at least partially sense,
than it looks to me that the fix in
http://issues.apache.org/jira/browse/HTTPCORE-169
which went in beta-3 should have prevented the IOReactor restart.
Correct. This is basically the same bug.
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
