Paul, the NIO sender (used by OperationClient objects on the server side) throws an ISE after a few minutes. I have a logger on the OUT_FLOW phase which captured the error. Here is the SOAP message:
ERROR 20:21:08,751 (LoggingHandler.java:161) - OUT_FAULT_FLOW: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope"; xmlns:wsa="http://www.w3.org/2005/08/addressing";><soapenv:Header><wsa:To>http://sealbook.ncl.ac.uk:11001/axis2/services/anonService16</wsa:To><wsa:ReplyTo><wsa:Address>http://www.w3.org/2005/08/addressing/none</wsa:Address></wsa:ReplyTo><wsa:MessageID>urn:uuid:50C54185E1E92FA9931201983671032</wsa:MessageID><wsa:Action>http://www.w3.org/2005/08/addressing/soap/fault</wsa:Action><wsa:RelatesTo>urn:uuid:FFB56CB70BDEA2D6F41201983736482</wsa:RelatesTo></soapenv:Header><soapenv:Body><soapenv:Fault><soapenv:Code><soapenv:Value>soapenv:Receiver</soapenv:Value></soapenv:Code><soapenv:Reason><soapenv:Text xml:lang="en-US">Error while trying to forward the request to [http://giga23.ncl.ac.uk:18080/axis2/services/Service2] </soapenv:Text></soapenv:Reason><soapenv:Detail><Exception>org.apache.axis2.AxisFault: Error while trying to forward the request to ..... hostname omitted .... at org.apache.axis2.AxisFault.makeFault(AxisFault.java:417) at org.apache.axis2.AxisFault.makeFault(AxisFault.java:406) at org.apache.axis2.receivers.RawXMLINOnlyMessageReceiver.invokeBusinessLogic(RawXMLINOnlyMessageReceiver.java:94) at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:96) at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:145) at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:275) at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:120) at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:263) at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:887) at org.apache.coyote.http11.Http11NioProtocol $Http11ConnectionHandler.process(Http11NioProtocol.java:696) at org.apache.tomcat.util.net.NioEndpoint $SocketProcessor.run(NioEndpoint.java:2009) at java.util.concurrent.ThreadPoolExecutor $Worker.runTask(ThreadPoolExecutor.java:885) at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) Caused by: java.lang.IllegalStateException: Error while trying to forward the request to [http://giga23.ncl.ac.uk:18080/axis2/services/Service2] at ncl.qosp.controller.service.streams.Sender.send(Sender.java:424) at ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.send(StreamServiceHandlerImpl.java:240) at ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.multipleForwards(StreamServiceHandlerImpl.java:226) at ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.releaseOrTryForward(StreamServiceHandlerImpl.java:505) at ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.response(StreamServiceHandlerImpl.java:581) at ncl.qosp.controller.service.streams.StreamServiceHandlerImpl.response(StreamServiceHandlerImpl.java:46) at ncl.qosp.controller.dispatcher.streams.StreamDispatcherImpl.response(StreamDispatcherImpl.java:572) at ncl.qosp.controller.streams.StreamRoutingService.result(StreamRoutingService.java:705) at sun.reflect.GeneratedMethodAccessor98.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.axis2.receivers.RawXMLINOnlyMessageReceiver.invokeBusinessLogic(RawXMLINOnlyMessageReceiver.java:90) ... 20 more Caused by: java.lang.IllegalStateException: I/O reactor has been shut down at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.connect(DefaultConnectingIOReactor.java:176) at org.apache.synapse.transport.nhttp.HttpCoreNIOSender.sendAsyncRequest(HttpCoreNIOSender.java:295) at org.apache.synapse.transport.nhttp.HttpCoreNIOSender.invoke(HttpCoreNIOSender.java:218) at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:396) at org.apache.axis2.description.OutOnlyAxisOperationClient.executeImpl(OutOnlyAxisOperation.java:265) at org.apache.axis2.client.OperationClient.execute(OperationClient.java:163) at ncl.qosp.controller.service.streams.Sender.send(Sender.java:399) ... 31 more The target node is up and it doesn't log any error, while Sender:399 is the statement OperationClient.execute(true); To summarize about the new NIO components from synapse 1.1.1 - the NIO listener looks ok - the NIO sender has still some problems Michele On Fri, 2008-02-01 at 19:59 +0000, Michele Mazzucco wrote: > Paul, > > I did the following: > - updated tomcat from 5.5 to 6.0.14 > - enabled and tuned the tomcat NIO connector > - upgraded the NIO listener and sender to synapse 1.1.1 (I had to > upgrade the http-core library from alpha5 to beta1) > - enabled the NIO sender on the server side (as I said I have some > ServiceClient instances running on the server too) > > but it didn't fix the problem. Indeed, about 50% of the connections > in TIME_WAIT state are due to the tomcat connector and the other 50% > to the axis2 sender. > > So I enabled the NIO sender and NIO receiver on the client side. > My scenario is composed by about 80% of asynchronous interactions and > 20% of very fast blocking in-out MEPs. Now the latter fail this way: > > ERROR 19:44:03,967 (QueueController.java:192) - Unable to add stream > for queue Service2 > [java] org.apache.axis2.AxisFault: The input stream for an > incoming message is null. > [java] at > org.apache.axis2.transport.TransportUtils.createSOAPMessage > (TransportUtils.java:71) > [java] at > org.apache.axis2.description.OutInAxisOperationClient.handleResponse > (OutInAxisOperation.java:326) > [java] at > org.apache.axis2.description.OutInAxisOperationClient.send > (OutInAxisOperation.java:389) > [java] at > org.apache.axis2.description.OutInAxisOperationClient.executeImpl > (OutInAxisOperation.java:211) > [java] at org.apache.axis2.client.OperationClient.execute > (OperationClient.java:163) > [java] at org.apache.axis2.client.ServiceClient.sendReceive > (ServiceClient.java:528) > [java] at org.apache.axis2.client.ServiceClient.sendReceive > (ServiceClient.java:508) > [java] at > ncl.qosp.test.client.streams.delayed.AddStream.send(AddStream.java:86) > [java] at > ncl.qosp.test.client.streams.delayed.QueueController.addStream > (QueueController.java:163) > [java] at > ncl.qosp.test.client.streams.delayed.QueueController.init > (QueueController.java:250) > [java] at > ncl.qosp.test.client.streams.delayed.QueueController.run > (QueueController.java:264) > [java] at java.util.concurrent.ThreadPoolExecutor > $Worker.runTask(ThreadPoolExecutor.java:650) > [java] at java.util.concurrent.ThreadPoolExecutor$Worker.run > (ThreadPoolExecutor.java:675) > [java] at java.lang.Thread.run(Thread.java:613) > > The executed statements (related to axis2) are: > final OMElement payload = createPayload(); > > ServiceClient sender = new ServiceClient > (AddStreamConfiguration.getConfigurationContext(), null); > sender.setOptions(getOptions()); > OMElement result = sender.sendReceive(payload); > > where the options are > options.setTransportInProtocol(Constants.TRANSPORT_HTTP); > options.setCallTransportCleanup(true); > options.setTo(StreamClient.getRouterEpr()); > options.setAction(TestConstants.ADD_STREAM_SOAP_ACTION); > options.setSoapVersionURI(SOAP12Constants.SOAP_ENVELOPE_NAMESPACE_URI); > options.setProperty(HTTPConstants.CHUNKED, Boolean.TRUE); > options.setProperty(HTTPConstants.AUTO_RELEASE_CONNECTION, > Boolean.TRUE); > > and the configuration context differs from the default one because of > the http connection manager with cached http client. > > > Michele > > On 1 Feb 2008, at 16:35, Paul Fremantle wrote: > > > Michele > > > > You shouldn't need to change anything apart from the axis2 xml sender > > and receiver sections. > > > > Paul > > > > On Feb 1, 2008 2:55 PM, Michele Mazzucco > > <[EMAIL PROTECTED]> wrote: > >> > >> On 1 Feb 2008, at 14:42, Paul Fremantle wrote: > >> > >>> Michele > >>> > >>> If you can post some test cases, we will do our best to improve the > >>> NIO transport. > >> > >> Paul, as I mentioned I'll do it as soon as I find some time. > >> > >>> You also should try the NIO transport code from the > >>> Synapse 1.1.1 release if you haven't already because that had a > >>> number > >>> of fixes in it. > >> > >> Thanks for the advice. Do I need to change anything apart from the > >> nio section in axis2.xml (the HttpCoreNIOListener and > >> HttpCoreNIOSender will be from package > >> org.apache.synapse.transport.nhttp and not > >> org.apache.axis2.transport.nhttp)? > >> > >> Thanks, > >> Michele > >> > >>> > >>> Thanks > >>> Paul > >>> > >>> On Feb 1, 2008 1:35 PM, Michele Mazzucco > >>> <[EMAIL PROTECTED]> wrote: > >>>> > >>>> On 1 Feb 2008, at 03:15, robert lazarski wrote: > >>>> > >>>>> Me == clueless . Just some general advice though. Have you thought > >>>>> about submitting patches to fix the NIO issues? I notice that > >>>>> someone > >>>>> from the synapse team (IIRC), three weeks ago, asked you to > >>>>> submit a > >>>>> test case for your failure in one of the jira's you posted. > >>>> > >>>> Robert, I know. I'll do that as soon as I find some time. > >>>>> > >>>>> As I've said before Michele, you easily have some of the most > >>>>> complex > >>>>> and long going use cases for axis2 as a non-committer. Your > >>>>> involvement via patches certainly have a better chance of getting > >>>>> developer attention. > >>>>> > >>>>> In respect to this issue, I can at least try to be a bit helpful: > >>>>> > >>>>> 1) Have you tried running your code on anything else besides > >>>>> OSX to > >>>>> see if these socket issues are OS related? I've run into several > >>>>> socket and nio issues specific to linux for example. > >>>> > >>>> Yes. Indeed my problem occurs on a 4 CPUs linux box > >>>> > >>>>> > >>>>> 2) Have you tried running http 1.0 instead of 1.1 ? > >>>> > >>>> Yes, it doesn't make any difference. > >>>>> > >>>>> 3) SimpleHTTPServer has never been meant for production use, so > >>>>> stop > >>>>> trying to use it like that. Fixing the nio issues seem to me to > >>>>> be the > >>>>> better path. > >>>>> > >>>> Yes, but the nio code comes from Synapse... and this would mean > >>>> digging into the Synapse code (which I don't know). > >>>> > >>>>> My understanding of TIME_WAIT, via an old usenet post of mine, was > >>>>> best explained to me this way: > >>>>> > >>>>> "After the connection is closed, there might still be some stray > >>>>> packets that were delayed and could still arrive. The TIME_WAIT > >>>>> status retains a record of a recent connection, so that the system > >>>>> can recognize these as delay packets." > >>>>> > >>>>> Are these connections going from CLOSE_WAIT to TIME_WAIT? Can > >>>>> you do > >>>>> a 'netstat -anp' and show the transition states? > >>>> > >>>> Well, netstat doesn't show me any transition, only the current > >>>> state > >>>> > >>>>> What issue do you > >>>>> have with TIME_WAIT exactly? > >>>> > >>>> The number of connections in TIME_WAIT state grows very fast to a > >>>> value > 500, and after that the my app. hangs (without crashing, it > >>>> just freezes) > >>>>> > >>>>> Anyways, since SimpleHTTPServer is deep involved into the several > >>>>> hundred unit / integration tests, I don't see major changes > >>>>> happening > >>>>> at this point. > >>>>> > >>>>> 4) Obviously try the latest axis2 nightlies and post questions to > >>>>> the > >>>>> http commons / reactor list. Its possible the latest snapshot of > >>>>> reactor fixes your issue and can be promoted to the upcomming > >>>>> axis2 > >>>>> 1.4 . > >>>> > >>>> I'll try. > >>>>> > >>>>> HTH, > >>>>> Robert > >>>>> > >>>> > >>>> Michele > >>>> > >>>> > >>>>> On Jan 31, 2008 7:48 PM, Michele Mazzucco > >>>>> <[EMAIL PROTECTED]> wrote: > >>>>>> Hello again, > >>>>>> > >>>>>> I've tried to write a TransportListener which uses Jetty as http > >>>>>> server > >>>>>> -- but I had no success: when requests come I get the following > >>>>>> error > >>>>>> > >>>>>> <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope > >>>>>> xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"; > >>>>>> xmlns:wsa="http://www.w3.org/2005/08/ > >>>>>> addressing"><soapenv:Header><wsa:Action>http://www.w3.org/ > >>>>>> 2005/08/ > >>>>>> addressing/soap/fault</wsa:Action></ > >>>>>> soapenv:Header><soapenv:Body><soapenv:Fault><faultcode>soapenv:Se > >>>>>> rv > >>>>>> er > >>>>>> </faultcode><faultstring>java.lang.NullPointerException</ > >>>>>> faultstring><detail><Exception>org.apache.axis2.AxisFault: > >>>>>> java.lang.NullPointerException > >>>>>> at > >>>>>> org.apache.axis2.transport.http.AxisServlet.doPost > >>>>>> (AxisServlet.java:182) > >>>>>> at javax.servlet.http.HttpServlet.service > >>>>>> (HttpServlet.java: > >>>>>> 709) > >>>>>> at javax.servlet.http.HttpServlet.service > >>>>>> (HttpServlet.java: > >>>>>> 802) > >>>>>> at > >>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter > >>>>>> (ApplicationFilterChain.java:252) > >>>>>> at > >>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter > >>>>>> (ApplicationFilterChain.java:173) > >>>>>> at > >>>>>> org.apache.catalina.core.StandardWrapperValve.invoke > >>>>>> (StandardWrapperValve.java:213) > >>>>>> at > >>>>>> org.apache.catalina.core.StandardContextValve.invoke > >>>>>> (StandardContextValve.java:178) > >>>>>> at > >>>>>> org.apache.catalina.core.StandardHostValve.invoke > >>>>>> (StandardHostValve.java:126) > >>>>>> at > >>>>>> org.apache.catalina.valves.ErrorReportValve.invoke > >>>>>> (ErrorReportValve.java:105) > >>>>>> at > >>>>>> org.apache.catalina.core.StandardEngineValve.invoke > >>>>>> (StandardEngineValve.java:107) > >>>>>> at > >>>>>> org.apache.catalina.connector.CoyoteAdapter.service > >>>>>> (CoyoteAdapter.java:148) > >>>>>> at > >>>>>> org.apache.coyote.http11.Http11Processor.process > >>>>>> (Http11Processor.java:869) > >>>>>> at org.apache.coyote.http11.Http11BaseProtocol > >>>>>> $Http11ConnectionHandler.processConnection > >>>>>> (Http11BaseProtocol.java: > >>>>>> 664) > >>>>>> at > >>>>>> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket > >>>>>> (PoolTcpEndpoint.java:527) > >>>>>> at > >>>>>> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt > >>>>>> (LeaderFollowerWorkerThread.java:80) > >>>>>> at org.apache.tomcat.util.threads.ThreadPool > >>>>>> $ControlRunnable.run(ThreadPool.java:684) > >>>>>> at java.lang.Thread.run(Thread.java:595) > >>>>>> Caused by: java.lang.NullPointerException > >>>>>> at > >>>>>> ncl.qosp.modules.manager.RouterDispatcher.isFault > >>>>>> (RouterDispatcher.java:230) > >>>>>> at > >>>>>> ncl.qosp.modules.manager.RouterDispatcher.invoke > >>>>>> (RouterDispatcher.java:267) > >>>>>> at org.apache.axis2.engine.Phase.invoke(Phase.java:292) > >>>>>> at > >>>>>> org.apache.axis2.engine.AxisEngine.invoke(AxisEngine.java:212) > >>>>>> at > >>>>>> org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:132) > >>>>>> at > >>>>>> org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPos > >>>>>> tR > >>>>>> eq > >>>>>> uest(HTTPTransportUtils.java:275) > >>>>>> at > >>>>>> org.apache.axis2.transport.http.AxisServlet.doPost > >>>>>> (AxisServlet.java:120) > >>>>>> ... 16 more > >>>>>> > >>>>>> I'm attaching my implementation. > >>>>>> > >>>>>> > >>>>>> Any help is appreciated, > >>>>>> Michele > >>>>>> > >>>>>> > >>>>>> On Thu, 2008-01-31 at 19:47 +0000, Michele Mazzucco wrote: > >>>>>>> Hello everybody, > >>>>>>> > >>>>>>> I'm facing a serious problem with too many connections left in > >>>>>>> TIME_WAIT > >>>>>>> state, causing my system to hang. > >>>>>>> Since on the server side I run some ServiceClient instances as > >>>>>>> well I > >>>>>>> wasn't sure about the root of my problem, but after some > >>>>>>> investigations > >>>>>>> I've found out that the responsible for this is the > >>>>>>> SimpleHTTPServer > >>>>>>> used as transport receiver for incoming http requests. > >>>>>>> > >>>>>>> Given that the NIO listener/sender are not stable (see the JIRAs > >>>>>>> below) > >>>>>>> I have to stick with these. Is there anything I can do? > >>>>>>> Is there any way to inject a custom http connection manager and > >>>>>>> manually > >>>>>>> clean up the unused connections? > >>>>>>> > >>>>>>> > >>>>>>> Thanks in advance, > >>>>>>> Michele > >>>>>>> > >>>>>>> > >>>>>>> [1] https://issues.apache.org/jira/browse/AXIS2-3473 > >>>>>>> [2] http://www.mail-archive.com/[email protected]/ > >>>>>>> msg37314.html > >>>>>>> [3] https://issues.apache.org/jira/browse/AXIS2-3428 > >>>>>>> > >>>>>>> > >>>>>>> ---------------------------------------------------------------- > >>>>>>> -- > >>>>>>> -- > >>>>>>> - > >>>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>>>>> For additional commands, e-mail: [EMAIL PROTECTED] > >>>>>>> > >>>>>> > >>>>>> ----------------------------------------------------------------- > >>>>>> -- > >>>>>> -- > >>>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>>>> For additional commands, e-mail: [EMAIL PROTECTED] > >>>>>> > >>>>> > >>>>> ------------------------------------------------------------------ > >>>>> -- > >>>>> - > >>>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>>> For additional commands, e-mail: [EMAIL PROTECTED] > >>>>> > >>>> > >>>> > >>>> ------------------------------------------------------------------- > >>>> -- > >>>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>>> For additional commands, e-mail: [EMAIL PROTECTED] > >>>> > >>>> > >>> > >>> > >>> > >>> -- > >>> Paul Fremantle > >>> Co-Founder and VP of Technical Sales, WSO2 > >>> OASIS WS-RX TC Co-chair > >>> > >>> blog: http://pzf.fremantle.org > >>> [EMAIL PROTECTED] > >>> > >>> "Oxygenating the Web Service Platform", www.wso2.com > >>> > >>> -------------------------------------------------------------------- > >>> - > >> > >>> To unsubscribe, e-mail: [EMAIL PROTECTED] > >>> For additional commands, e-mail: [EMAIL PROTECTED] > >>> > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [EMAIL PROTECTED] > >> For additional commands, e-mail: [EMAIL PROTECTED] > >> > >> > > > > > > > > -- > > Paul Fremantle > > Co-Founder and VP of Technical Sales, WSO2 > > OASIS WS-RX TC Co-chair > > > > blog: http://pzf.fremantle.org > > [EMAIL PROTECTED] > > > > "Oxygenating the Web Service Platform", www.wso2.com > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
