Hello All!
Here is the story: I have Synapse in its default configuration running
on my physical machine, three Apache ODE servers running in virtual
machines with separate IPs, and standard Axis2 ADB generated client
(actually it is placed in test case). Client is executed in separate
threads and each thread make call to synapse, which is configured to be
balancing proxy and Synapse should direct SOAP call to one of 3 ODE
servers. I should note that calling ODE server is quite time consuming,
each call takes from 20 to 40 seconds (maybe more), and quite resource
consuming (when running test processor is at 100% but there is steel 1,6
Gb of memory free). Below a list of problems I faced and steps I took,
maybe someone can help me to resolve this.
--- synapse configuration I used:
<definitions xmlns="http://ws.apache.org/ns/synapse";>
<log level="full"/>
<in>
<filter source="get-property('To')" regex=".*/dbService$">
<send>
<endpoint>
<loadbalance>
<endpoint>
<address
uri="http://192.168.38.128:8090/te/processes/dbService";>
<suspendDurationOnFailure>60</suspendDurationOnFailure>
</address>
</endpoint>
<endpoint>
<address
uri="http://192.168.38.131:8090/te/processes/dbService";>
<suspendDurationOnFailure>60</suspendDurationOnFailure>
</address>
</endpoint>
<endpoint>
<address
uri="http://192.168.38.132:8090/te/processes/dbService";>
<suspendDurationOnFailure>60</suspendDurationOnFailure>
</address>
</endpoint>
</loadbalance>
</endpoint>
</send>
<drop />
</filter>
<filter source="get-property('To')" regex=".*/registerUser$">
<send>
<endpoint>
<loadbalance>
<endpoint>
<address
uri="http://192.168.38.128:8090/te/processes/registerUser";>
<suspendDurationOnFailure>60</suspendDurationOnFailure>
</address>
</endpoint>
<endpoint>
<address
uri="http://192.168.38.131:8090/te/processes/registerUser";>
<suspendDurationOnFailure>60</suspendDurationOnFailure>
</address>
</endpoint>
<endpoint>
<address
uri="http://192.168.38.132:8090/te/processes/registerUser";>
<suspendDurationOnFailure>60</suspendDurationOnFailure>
</address>
</endpoint>
</loadbalance>
</endpoint>
</send>
<drop />
</filter>
..........
..........
</in>
<out>
<!-- Send the messages where they have been sent (i.e. implicit
To EPR) -->
<send />
</out>
<sequence name="fault">
<makefault>
<code value="tns:Receiver"
xmlns:tns="http://www.w3.org/2003/05/soap-envelope"; />
<reason value="COULDN'T SEND THE MESSAGE TO THE SERVER." />
</makefault>
<header name="To" action="remove" />
<property name="RESPONSE" value="true" />
<send />
</sequence>
</definitions>
--- 1 - Downloaded Synapse 1.1 release, 3 ODE servers, 4 threads, 5
iterations. From time to time the following exception occur in client:
18:12:09,218 DEBUG HttpMethodDirector.executeWithRetry(403) : Closing
the connection.
18:12:09,218 INFO HttpMethodDirector.executeWithRetry(438) : I/O
exception (org.apache.commons.httpclient.NoHttpResponseException) caught
when processing request: The server localhost failed to respond
18:12:09,234 DEBUG HttpMethodDirector.executeWithRetry(442) : The server
localhost failed to respond
org.apache.commons.httpclient.NoHttpResponseException: The server
localhost failed to respond
at
org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1835)
at
org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1590)
at
org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:995)
at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:397)
at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:170)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:396)
at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:346)
at
org.apache.axis2.transport.http.AbstractHTTPSender.executeMethod(AbstractHTTPSender.java:520)
at
org.apache.axis2.transport.http.HTTPSender.sendViaPost(HTTPSender.java:191)
at org.apache.axis2.transport.http.HTTPSender.send(HTTPSender.java:77)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.writeMessageWithCommons(CommonsHTTPTransportSender.java:327)
at
org.apache.axis2.transport.http.CommonsHTTPTransportSender.invoke(CommonsHTTPTransportSender.java:206)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:396)
at
org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:374)
at
org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:211)
at
org.apache.axis2.client.OperationClient.execute(OperationClient.java:163)
at
com.comped.TE.client.RegisterUserStub.registerUser(RegisterUserStub.java:185)
at
com.comped.TE.client.RegisterUserTest$1UserRegistrator.run(RegisterUserTest.java:173)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
18:12:09,421 INFO HttpMethodDirector.executeWithRetry(444) : Retrying
request
18:12:10,031 DEBUG HttpConnection.open(691) : Open connection to
localhost:8084
It is maybe not serious, because Axis2 client automatically generates
retry request, but main thing is the following: when Synapse closes
connection it is always sents request to one of the ODE servers, and ODE
server successfully registers user, and when retry happens user becomes
already registered and ODE server returns failure. It is not good, but
even if there would be no retry it will not be good -> "Synapse closes
connection, client gets an error but user is successfully registered".
--- 2 - Tried to deploy Synapse on Tomcat server, the same default
configuration, I cannot even call one of servers, the first call returns
exception:
2008-04-25 14:46:42,500 [172.29.140.179-ws00-aousov]
[http-8084-Processor25] INFO LogMediator To:
http://localhost:8084/synapse/soap/principalsService, From: 127.0.0.1,
WSAction: urn:addUser, SOAPAction: urn:addUser, Direction: request,
Envelope: <?xml version='1.0' encoding='utf-8'?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope";><soapenv:Header><wsse:Security
xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd";><wsse:UsernameToken>
<wsse:Username>root</wsse:Username>
<wsse:Password
Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText";>12qw</wsse:Password>
</wsse:UsernameToken></wsse:Security></soapenv:Header><soapenv:Body><ns1:addUser
xmlns:ns1="http://www.comped.it/te/principalsService";><ns1:name>user98f26728-74e7-4dd4-bcd0-4cedf1f3376d</ns1:name><ns1:authMethod>1</ns1:authMethod><ns1:password>E16Y3NQbIt</ns1:password></ns1:addUser></soapenv:Body></soapenv:Envelope>
2008-04-25 14:46:42,656 [172.29.140.179-ws00-aousov]
[http-8084-Processor25] INFO TimeoutHandler This engine will expire all
callbacks after : 86400 seconds, irrespective of the timeout action,
after the specified or optional timeout
2008-04-25 14:46:42,718 [172.29.140.179-ws00-aousov]
[http-8084-Processor25] INFO PipeImpl Using simulated buffered Pipes
for event-driven to stream IO bridging
2008-04-25 14:46:43,328 [172.29.140.179-ws00-aousov]
[HttpClientWorker-1] INFO LogMediator To:
http://www.w3.org/2005/08/addressing/anonymous, WSAction: , SOAPAction:
, MessageID: urn:uuid:3E27B7248E21F6AFA016069411008043211750373,
Direction: response, Envelope: <?xml version='1.0'
encoding='utf-8'?><soapenv:Envelope
xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope";><soapenv:Body><addUserResponse
xmlns="http://www.comped.it/te/principalsService";><return>68</return></addUserResponse></soapenv:Body></soapenv:Envelope>
Exception in thread "HttpClientWorker-1" java.lang.NullPointerException
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:747)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:432)
at
org.apache.coyote.http11.InternalOutputBuffer.flush(InternalOutputBuffer.java:305)
at
org.apache.coyote.http11.Http11Processor.action(Http11Processor.java:996)
at org.apache.coyote.Response.action(Response.java:183)
at
org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:322)
at
org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:293)
at
org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:124)
at com.ctc.wstx.io.UTF8Writer.flush(UTF8Writer.java:99)
at com.ctc.wstx.sw.BufferingXmlWriter.flush(BufferingXmlWriter.java:214)
at com.ctc.wstx.sw.BaseStreamWriter.flush(BaseStreamWriter.java:311)
at
org.apache.axiom.om.impl.MTOMXMLStreamWriter.flush(MTOMXMLStreamWriter.java:152)
at
org.apache.axiom.om.impl.llom.OMNodeImpl.serialize(OMNodeImpl.java:451)
at
org.apache.axis2.transport.http.SOAPMessageFormatter.writeTo(SOAPMessageFormatter.java:77)
at
org.apache.synapse.transport.nhttp.HttpCoreNIOSender.sendUsingOutputStream(HttpCoreNIOSender.java:449)
at
org.apache.synapse.transport.nhttp.HttpCoreNIOSender.invoke(HttpCoreNIOSender.java:266)
at org.apache.axis2.engine.AxisEngine.send(AxisEngine.java:448)
at
org.apache.synapse.core.axis2.Axis2Sender.sendBack(Axis2Sender.java:138)
at
org.apache.synapse.core.axis2.Axis2SynapseEnvironment.send(Axis2SynapseEnvironment.java:221)
at
org.apache.synapse.mediators.builtin.SendMediator.mediate(SendMediator.java:158)
at
org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:58)
at
org.apache.synapse.mediators.filters.OutMediator.mediate(OutMediator.java:60)
at
org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:58)
at
org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:125)
at
org.apache.synapse.core.axis2.Axis2SynapseEnvironment.injectMessage(Axis2SynapseEnvironment.java:178)
at
org.apache.synapse.core.axis2.SynapseCallbackReceiver.handleMessage(SynapseCallbackReceiver.java:298)
at
org.apache.synapse.core.axis2.SynapseCallbackReceiver.receive(SynapseCallbackReceiver.java:129)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:176)
at
org.apache.synapse.transport.nhttp.ClientWorker.run(ClientWorker.java:210)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
at java.lang.Thread.run(Thread.java:595)
--- 3 - Tried to call one of ODE servers directly from client, 3
threads, 5 iterations - all requests was successful, so it is not ODE issue.
--- 4 - downloaded and compiled trunk release, set axis2 configuration
option:
<transportReceiver name="http"
class="org.apache.synapse.transport.nhttp.HttpCoreNIOListener">
<parameter name="port">8084</parameter>
<parameter name="requestTimeout">120000</parameter>
<parameter name="non-blocking">true</parameter>
</transportReceiver>
--- 4a - 3 ODE servers, 5 threads, 5 iterations: number of failed
threads: 2 number of successive responses: 10 i.e. all threads which
sent first 3 requests for first 3 servers succeeded, and other two -
failed with error as described in --- 1 (user is already registered)
--- 4b - 2 ODE servers, 3 threads, 5 iterations: the same as --- 4a
--- 4c - 2 ODE servers, 2 threads, 5 iterations: everything SUCCESS
--- 5 - Tried to deploy Synapse 1.2-SNAPSHOT to Tomcat, got the same
error as in --- 2
Somebody can tell me if I misconfigured something or this is Synapse
specific issue?
